Multi-dialect speech recognition method, apparatus, device, medium and product

Abstract

The application provides a multi-dialect speech recognition method, device, equipment, medium and product, and relates to the field of audio and video. The method comprises: obtaining a dialect speech signal input by a user, performing feature extraction on the dialect speech signal, and obtaining a corresponding real-time speech feature set; inputting the real-time speech feature set into a type recognition branch of a dialect recognition model to obtain a dialect type of the dialect speech signal; based on the dialect type, matching target model parameters corresponding to a dialect recognition branch of the dialect recognition model, and calling the target model parameters to update the dialect recognition branch; inputting the real-time speech feature set into the updated dialect recognition branch to obtain a dialect recognition result of the dialect speech signal. The method effectively improves the accuracy of multi-dialect speech recognition, enhances the generalization ability of the dialect recognition model to different dialects, and improves the adaptation and running efficiency of the dialect recognition model for different dialects.

Description

Technical Field

[0001] This application relates to the field of audio and video, and in particular to a method, apparatus, device, medium and product for multi-dialect speech recognition. Background Technology

[0002] With the digital transformation of banking services and the deepening of inclusive finance, the geographical distribution of bank customers is becoming increasingly widespread, and the demand for dialect voice interaction among users in different regions is becoming increasingly prominent.

[0003] In existing voice service scenarios such as banking, a fixed, standardized voice recognition process is typically used to process multiple accents and languages.

[0004] However, due to the significant differences between dialects and standard Mandarin in terms of pronunciation rules, intonation patterns, and vocabulary structure, existing speech recognition solutions often face the problem of low recognition accuracy when processing speech from different dialects. Summary of the Invention

[0005] This application provides a multi-dialect speech recognition method, apparatus, device, medium, and product to solve the technical problem that existing speech recognition solutions often face low recognition accuracy when processing speech from different dialects.

[0006] In a first aspect, this application provides a multi-dialect speech recognition method, including:

[0007] The dialect speech signal input by the user is obtained, and the dialect speech signal is used to extract features to obtain the corresponding real-time speech feature set;

[0008] Input the real-time speech feature set into the type recognition branch of the dialect recognition model to obtain the dialect type of the dialect speech signal;

[0009] Based on dialect type, the target model parameters corresponding to the dialect recognition branch of the dialect recognition model are matched, and the target model parameters are called to update the dialect recognition branch;

[0010] The real-time speech feature set is input into the updated dialect recognition branch to obtain the dialect recognition result of the dialect speech signal.

[0011] Secondly, this application provides a multi-dialect speech recognition device, comprising:

[0012] The acquisition module is used to acquire historical multi-dialect speech signals.

[0013] The processing module is used to generate speech feature sets corresponding to different dialect types based on historical multi-dialect speech signals.

[0014] The processing module is also used to extract features from multiple speech feature sets to obtain general dialect features and dialect distinguishing features corresponding to different dialect types.

[0015] The processing module is also used to perform hierarchical training on the hybrid acoustic model based on the common features of dialects and the dialect distinguishing features corresponding to different dialect types, so as to obtain a dialect recognition model.

[0016] The acquisition module is also used to acquire dialect voice signals input by the user.

[0017] The processing module is also used to input dialect speech signals into the dialect recognition model to obtain dialect recognition results of the dialect speech signals.

[0018] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0019] The memory stores the instructions that the computer executes;

[0020] The processor executes computer execution instructions stored in memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0021] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0022] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0023] The multi-dialect speech recognition method provided in this application acquires historical multi-dialect speech signals and generates speech feature sets corresponding to different dialect types based on these signals. It then extracts features from these multiple speech feature sets to obtain general dialect features and dialect distinguishing features corresponding to different dialect types. Based on these general and distinguishing features, it performs hierarchical training on a hybrid acoustic model to obtain a dialect recognition model. Finally, it acquires user-input dialect speech signals and extracts features from them to obtain corresponding real-time speech feature sets. These real-time speech feature sets are then input into the dialect recognition model. The method employs a type recognition branch to identify the dialect type of the speech signal. Based on the dialect type, it matches the general model parameters and discrimination model parameters corresponding to the dialect recognition branch. It then determines the dialect complexity features of the speech signal and, based on these features, determines the parameter sharing ratio between the general and discrimination model parameters. Based on the parameter sharing ratio, the general and discrimination model parameters, it determines the target model parameters corresponding to the dialect recognition branch and updates the branch using these parameters. Finally, it inputs the real-time speech feature set into the updated dialect recognition branch to obtain the dialect recognition result. This method, through the synergistic optimization of hybrid acoustic models and transfer learning, enables the model to adapt to the acoustic feature differences of different dialects, effectively improving the accuracy of multi-dialect speech recognition. It not only significantly enhances the generalization ability and recognition stability of multi-dialect speech recognition in complex scenarios but also improves service efficiency and customer satisfaction. Attached Figure Description

[0024] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0025] Figure 1 This is a flowchart illustrating the multi-dialect speech recognition method provided in this application. Figure 1 .

[0026] Figure 2 This is a flowchart illustrating the multi-dialect speech recognition method provided in this application. Figure 2 .

[0027] Figure 3 This is a schematic diagram of the multi-dialect speech recognition device provided in this application.

[0028] Figure 4 This is a schematic diagram of the structure of the electronic device provided in this application.

[0029] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0030] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application.

[0031] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of the relevant data all comply with the relevant laws, regulations, and standards of the relevant countries and regions, have taken necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation access points for users to choose to authorize or refuse.

[0032] Furthermore, the technical solution involved in this application, which involves big data analysis of user information (including but not limited to personal biometrics, identity data, consumption data, asset data, electronic terminal operation data, etc.) and the use of artificial intelligence technology for automated decision-making, and makes decisions that have a significant impact on personal rights based on the results of automated decision-making, provides users with corresponding operation entry points for users to choose to agree to or reject the results of automated decision-making; if the user chooses to reject, the process will proceed to the expert decision-making process.

[0033] It should be noted that the multi-dialect speech recognition methods, devices, equipment, media, and products provided in this application can be used in the audio-visual field, as well as in any other field. The application fields of the multi-dialect speech recognition methods, devices, equipment, media, and products in this application are not limited.

[0034] First, the terms used in this application will be explained.

[0035] MFCC: Mel-Frequency Cepstral Coefficients.

[0036] MFCC matrix: Mel-frequency cepstral coefficient matrix. A continuous audio signal is divided into several short frames according to time. Mel-frequency cepstral coefficient feature vectors are extracted from each frame. All feature vectors are then concatenated in chronological order to form a two-dimensional matrix, which is the MFCC matrix.

[0037] With the digital transformation of banking services and the deepening of inclusive finance, the geographical distribution of bank customers is becoming increasingly widespread, and the demand for dialect voice interaction among users in different regions is becoming increasingly prominent.

[0038] Currently, multi-dialect speech recognition solutions typically employ a fixed, standardized speech recognition process to process speech from multiple accents and languages. This involves first preprocessing the acquired speech signal, then extracting spectral features, subsequently inputting them into a deep learning acoustic model for recognition, and finally correcting the results by combining language models or contextual rules.

[0039] This multi-dialect speech recognition solution is typically used in voice service scenarios such as banking, and is particularly suitable for business environments such as branch counter assistance, remote video customer service, telephone customer service centers, self-service terminal voice interaction, and mobile financial voice assistants.

[0040] In multi-dialect voice service scenarios, customers may use different regional dialects such as Cantonese, Sichuan-Chongqing dialect, and Wu dialect to express their business needs such as opening an account, transferring money, reporting lost items, financial advice, and applying for loans. Service personnel or existing intelligent voice systems need to accurately understand the customer's voice content in a short period of time to ensure business processing efficiency and service quality.

[0041] However, due to the significant differences between dialects and standard Mandarin in terms of pronunciation rules, intonation patterns, and vocabulary structure, existing speech recognition solutions often face problems such as low recognition accuracy and difficulty in semantic understanding when processing speech from different dialects.

[0042] On the one hand, most dialects lack sufficient, high-quality, and directly usable labeled speech data for training, resulting in insufficient training sample coverage for the model and making it difficult to fully learn the complex differences in pronunciation position, resonance characteristics, tone changes, speech rate and rhythm, and vocabulary expression among different dialects.

[0043] On the other hand, existing recognition models are often trained on Mandarin or a single dialect. Their parameter distribution and feature focus are more adapted to the mainstream speech patterns. When the input speech is switched to other dialects, problems such as inaccurate syllable segmentation, incorrect word boundary judgment, confusion of similar pronunciations, and semantic mapping deviation are likely to occur, resulting in deviations in the recognition results.

[0044] To address the aforementioned issues, this application provides a multi-dialect speech recognition method.

[0045] The multi-dialect speech recognition method provided in this application achieves dynamic adaptation of dialect recognition model parameters by training a hybrid acoustic model in a hierarchical manner and combining transfer learning strategies and dialect complexity features. After a user inputs a dialect speech signal, the dialect recognition model first identifies the dialect type, updates the corresponding branch parameters of the model, and then outputs the dialect recognition result using the updated model. This method, by first identifying the dialect type and then matching the corresponding target model parameters and updating the recognition branch, enables the recognition process to form a more targeted processing path for different dialects. This improves the adaptability and recognition effect of multi-dialect input, enhances the accuracy, generalization ability, and adaptation efficiency of multi-dialect speech recognition, reduces communication misunderstandings caused by dialect differences, and improves service efficiency and customer satisfaction.

[0046] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0047] Figure 1 This is a flowchart illustrating the multi-dialect speech recognition method provided in this application. Figure 1 The implementing entity in this embodiment can be, for example, a voice interaction service system. Figure 1 As shown, the multi-dialect speech recognition method provided in this embodiment includes:

[0048] S101: Acquire the dialect speech signal input by the user, and extract features from the dialect speech signal to obtain the corresponding real-time speech feature set.

[0049] In this embodiment, the voice interaction service system can be deployed in the bank's voice processing server, or it can be set in the edge nodes of the smart counter, self-service terminal, remote video customer service terminal, or telephone customer service system in the branch. The voice interaction service system works in conjunction with the microphone, smart counter, self-service terminal, or customer service seat system to provide more efficient multi-dialect voice recognition capabilities for counter services, remote customer service, and smart terminals without changing the bank's existing overall voice interaction application mode.

[0050] Understandably, dialect speech signals refer to continuous acoustic waveform data acquired by a voice interaction service system, used to carry the dialect speech content output by the user. These dialect speech signals can be, for example, continuous speech segments expressing customer needs regarding account opening, money transfer, or reporting lost items, or interactive speech streams composed of multiple short sentences. This application does not impose any special restrictions on dialect speech signals.

[0051] A real-time speech feature set refers to a set of feature data obtained after feature processing of the currently acquired dialect speech signal. It is used to quantitatively characterize the acoustic features of the currently acquired dialect speech signal in different acoustic dimensions (such as spectrum, tone, and prosody). This real-time speech feature set can be, for example, a multi-frame speech feature sequence arranged in chronological order, or a context feature matrix formed by further concatenating multi-frame features. This application does not impose any special restrictions on the real-time speech feature set.

[0052] The voice interaction service system acquires the dialect voice signal input by the user and extracts features from the dialect voice signal to obtain the corresponding real-time voice feature set.

[0053] Understandably, this step transforms the complex waveform-level input into a unified feature representation suitable for model processing by extracting a set of real-time speech features that can stably represent pronunciation differences from the original dialect speech signal. This provides a consistent data foundation for the subsequent dialect classification performed by the type recognition branch and the content recognition performed by the dialect recognition branch.

[0054] For example, dialect speech signals can be acquired through microphone arrays, single-microphone acquisition devices, customer service headsets, mobile terminal microphones, or voice acquisition units of self-service devices. During acquisition, analog speech can be converted to digital speech at a preset sampling rate to form a digital speech sequence that can be processed. If the dialect language signal acquired at this time is "check balance", the voice acquisition device acquires the raw dialect speech waveform signal in real time at a preset sampling frequency (e.g., 16kHz) and 16-bit quantization precision. The acquired dialect speech waveform signal is then preprocessed with pre-processing such as de-reverberation and silent segment trimming to effectively filter out environmental noise, current noise generated by the device itself, and invalid silent segments before and after the user's speech. This preserves the valid dialect speech signal and avoids invalid signals interfering with the accuracy of subsequent feature extraction. After the speech signal preprocessing is completed, multi-dimensional acoustic feature extraction is performed on the preprocessed dialect speech signal. The extracted single-dimensional acoustic features are then integrated, dimension-corresponding, and data-normalized to extract abnormal feature data, thereby obtaining a structured real-time speech feature set.

[0055] In one possible implementation, in order to ensure the stability of subsequent dialect recognition, basic speech front-end processing can be performed first after the dialect speech signal is acquired. This includes environmental noise suppression, echo cancellation, speech endpoint detection, silence removal, and amplitude normalization.

[0056] Among them, environmental noise suppression can be achieved through spectral subtraction, Wiener filtering or neural network noise reduction, speech endpoint detection can determine the effective speech start and end positions based on short-time energy, zero-crossing rate or model-based speech activity detection results, and amplitude normalization is used to reduce the impact of differences in speaking distance and speech intensity among different customers.

[0057] For multi-dialect banking scenarios, front-end feature processing separates non-target factors such as environmental noise, device differences, and volume fluctuations from the actual pronunciation features. This helps reduce the sensitivity of subsequent models to irrelevant factors, thereby reducing recognition errors caused by complex business environments, differences in customer age, and inconsistent terminal types.

[0058] In another possible implementation, for open environments in bank branches, beamforming processing can be combined to enhance the direction of the speaker's voice source, thereby reducing interference from other people around the counter to the voice acquisition.

[0059] In one possible implementation, the feature extraction process includes pre-emphasis, framing, windowing, time-frequency conversion, and spectrum mapping.

[0060] Among these features, pre-emphasis can enhance high-frequency components, framing can divide continuous speech into speech frames with a duration of 20ms to 40ms, and a frame shift of about 10ms is set between adjacent frames. Windowing can use Hamming windows to reduce spectral leakage caused by discontinuities at frame boundaries. Time-frequency conversion can obtain the linear spectrum of each speech frame through fast Fourier transform, and then the linear spectrum is mapped to logarithmic Mel spectral features, Mel frequency cepstral coefficient features, or filter bank energy features through Mel filter banks.

[0061] In another possible implementation, composite features containing fundamental frequency, formants, tone profiles, short-time energy, and duration information can be directly used to enhance the representation of tone changes, syllable lengthening, and local pronunciation differences in dialect speech.

[0062] Understandably, due to the differences in vowel resonance, tone fluctuation, aspiration and unaspiration distinction, and tone sandhi in different dialects, the design of real-time speech feature sets not only preserves spectral information but also introduces first-order and second-order difference features to express the dynamic trend of speech changes over time.

[0063] S102: Input the real-time speech feature set into the type recognition branch of the dialect recognition model to obtain the dialect type of the dialect speech signal.

[0064] In this embodiment of the application, the dialect recognition model is used to recognize and process dialect speech signals and output dialect type or recognition result; for example, the model can be deployed as a unified neural network framework, which includes at least a type recognition branch and a dialect recognition branch in the network structure, so as to output the dialect type recognition result or the dialect text recognition result for the dialect speech signal.

[0065] Understandably, the type recognition branch is used to identify the dialect category of the current dialect speech signal based on the real-time speech feature set. Its input is the real-time speech feature set generated in step S101, and its output is the dialect type of the dialect speech signal. In order to adapt to the requirements of response speed in the banking scenario, the type recognition branch can adopt a lightweight classification network structure, such as consisting of several temporal convolutional layers, bidirectional recurrent layers, attention layers and fully connected classification layers, or it can adopt a discriminant network based on the Transformer encoder.

[0066] Dialect type is used to characterize the dialect category to which the input dialect speech signal belongs, and serves as the basis for model parameter matching and updating. Dialect type can correspond to Cantonese, Sichuan-Chongqing dialect, Wu dialect, Minnan dialect, Gan dialect, Hakka dialect, or other regional dialect categories predefined by the business system. It can also be further subdivided into specific regional accent subcategories at a finer granular level.

[0067] The voice interaction service system inputs the real-time voice feature set into the type recognition branch of the dialect recognition model in frame sequence. The type recognition branch outputs classification scores or probability distributions for each candidate dialect type. Based on the maximum probability principle, threshold judgment principle, or combined with temporal smoothing rules, the dialect type that best matches the dialect language signal is selected.

[0068] In one possible implementation, the type recognition branch performs supervised learning based on historical multi-dialect speech samples during the training phase; wherein, the training samples may come from, for example, bank customer service voice recordings, counter business interaction voices, simulated collection data, and public speech databases, and a training set is established by combining manually labeled dialect categories.

[0069] To enhance the robustness of the dialect recognition model in practical applications, noise enhancement, speech rate perturbation, vocal tract perturbation, and volume perturbation can be introduced during the training process of the model, so that the type recognition branch can still stably distinguish dialect types under different terminals, different scenarios, and user input from different age groups.

[0070] In another possible implementation, the type recognition branch can not only make a single judgment based on the real-time speech feature set of the current input, but also combine the dialect prediction results of the previous few sentences of the current conversation to perform sequence consistency constraints.

[0071] For example, when multiple consecutive voice messages from the same customer are identified as the same dialect during a single business transaction, the voice interaction service system can increase the confidence level of that dialect type. When multiple candidate types have similar probabilities, a smooth selection can be made based on the historical conversation context to reduce misjudgments caused by short or unclear pronunciations in the user's voice.

[0072] Understandably, the core of this step is to first determine the dialect type of the currently input dialect speech signal through the type recognition branch, so that the subsequent recognition process will no longer use uniform fixed parameters to homogenize all dialects, but will create conditions for the dialect recognition branch to select more targeted parameter configurations.

[0073] Since there are significant differences in pronunciation position, syllable structure, tone pattern and lexical expression among different dialects, dialect type identification can establish a correlation between the input speech and the dialect model at the recognition front end. This avoids directly processing the speech features of different distributions such as Cantonese, Sichuan-Chongqing dialect, and Wu dialect with the same set of recognition parameters, thereby reducing the probability of syllable confusion, word boundary misjudgment and semantic mapping deviation.

[0074] For real-time banking service scenarios, the type recognition branch can output the dialect type with a single forward inference, without having to run a complete recognition model independently for each dialect. This allows for control of overall processing latency while taking into account the ability to adapt to multiple dialects.

[0075] S103: Based on dialect type, match the target model parameters corresponding to the dialect recognition branch of the dialect recognition model, and call the target model parameters to update the dialect recognition branch.

[0076] In this embodiment of the application, the dialect recognition branch is a sub-branch in the dialect recognition model that undertakes the function of dialect text recognition; the dialect recognition branch is used to further recognize the obtained real-time speech feature set and output the corresponding dialect recognition result after the model parameters of the corresponding branch are updated.

[0077] Understandably, model parameters are the basic variables that deep learning models can use for training and adjusting model operation logic. They include various trainable parameters such as weights, biases, normalized statistics, gating coefficients, and adaptation layer parameters, and are the core basis for the model to perform feature calculation and classification. Target model parameters are a set of parameters that are updated for the dialect recognition branch corresponding to a specific dialect type. In essence, they are combinations of model weights, biases, normalized statistics, gating coefficients, adaptation layer parameters, or other trainable parameters corresponding to different dialect types.

[0078] In one possible implementation, the voice interaction service system can pre-establish a dialect parameter database, which stores model parameters corresponding to different dialect types, and achieve retrieval through the mapping relationship between dialect type and parameter identifier.

[0079] Understandably, parameter libraries can be stored in server memory, local cache, distributed model repository, or solid-state storage on edge devices.

[0080] After determining the dialect type of the currently acquired dialect speech signal, the voice interaction service system generates a parameter retrieval request based on the dialect type, reads the corresponding model parameters from the parameter library, and determines the corresponding model parameters as the target model parameters for the dialect recognition branch update. The target model parameters are then loaded into the specified level of the dialect recognition branch to update the dialect recognition branch.

[0081] For example, the method of updating the dialect recognition branch based on the target model parameters may include: replacing the entire branch parameters, covering some layer parameters, inserting and updating the adaptation layer, or performing fusion calculation based on the basic shared parameters and dialect-specific parameters. If the fusion calculation method is adopted, the shared backbone network parameters can remain unchanged, and only the terminal decoding layer, attention layer, or normalization layer can be updated for dialect adaptation, so as to balance the model's generalization ability and response efficiency.

[0082] In one possible implementation, the target model parameters may include acoustic mapping weights corresponding to the dialect type; these acoustic mapping weights can make the dialect recognition branch more closely match the actual pronunciation rules of the corresponding dialect type during the phoneme-to-word conversion process.

[0083] For example, for dialects with significant tonal differences, the target model parameters can be enhanced with weights related to fundamental frequency changes and duration patterns; for dialects with features such as confusion between retroflex and alveolar consonants, changes in nasal consonant endings, or preservation of entering tones, the target model parameters can be enhanced with the ability to discriminate corresponding frequency bands or temporal features.

[0084] In another possible implementation, updating the dialect recognition branch by calling the target model parameters also includes executing a hierarchical update strategy based on the recognition confidence of the current dialect type.

[0085] Specifically, when the confidence level of the dialect type output by the type recognition branch is higher than the preset threshold, a complete parameter replacement scheme is adopted; when the recognition confidence level is in the transition range, a weighted fusion update scheme of basic shared parameters and target model parameters is adopted to reduce the impact of slight fluctuations in type recognition on the final recognition result.

[0086] For example, weighted fusion can be represented by the following formula 1:

[0087]

[0088] in, This indicates the parameters actually used by the dialect recognition branch after the update. This represents the target model parameters corresponding to the current dialect type. Indicates shared basic parameters. This represents the fusion coefficient determined based on the confidence level of the dialect type.

[0089] The fusion coefficient can increase numerically with increasing confidence, thereby enhancing dialect specificity when dialect type judgment is more certain, and preserving the robustness of the shared model to cross-dialect features when the judgment is not stable enough.

[0090] In another possible implementation, the dialect recognition branch update process is not limited to statically loading parameters, but can also combine the length, speech rate, signal-to-noise ratio and business scenario labels of the current speech segment to perform secondary adaptation of the target model parameters.

[0091] For example, when the voice interaction service system detects that the current dialect voice signal comes from the telephone customer service channel, it can simultaneously call compensation parameters suitable for narrowband telephone voice; when it detects that the current voice comes from the counter's near-field microphone and has a high signal-to-noise ratio, it can use more granular dialect-specific parameter configurations. By combining the current voice feature parameters (such as the length of the voice segment, speech rate, and signal-to-noise ratio) with business scenario information (such as business scenario labels) to perform secondary adaptation on the target model parameters, the dialect recognition branch can maintain dialect matching while also taking into account the differences in acquisition channels and environmental conditions.

[0092] Understandably, determining and updating the model parameters of the dialect recognition branch is a key step in solving the problems of multi-dialect recognition adaptation efficiency and recognition accuracy. Traditional solutions often require deploying independent recognition models for each dialect, resulting in an increase in the number of models, complex maintenance, and low switching efficiency. The embodiments of this application first identify the dialect type, then match the corresponding target model parameters from the parameter library and update the dialect recognition branch, so that the unified dialect recognition model can dynamically switch to the corresponding parameter state for different input speech. This not only retains the general learning ability of the shared model architecture on the basic speech mode, but also improves the adaptation to local pronunciation features, speech rhythm, and vocabulary habits through dialect-specific parameters.

[0093] When dialect training data is relatively limited, by combining shared parameters with dialect adaptation parameters, we can avoid training a complete large model for each dialect separately, achieve higher parameter utilization efficiency and lower system deployment costs, and reduce the continuous amplification of front-end misjudgments in subsequent recognition links.

[0094] S104: Input the real-time speech feature set into the updated dialect recognition branch to obtain a dialect recognition result of the dialect speech signal.

[0095] It can be understood that the dialect recognition result refers to the final recognition conclusion for the dialect speech signal output by the dialect recognition model. The dialect recognition result may be, for example, a recognition output in the form of text, or a structured recognition result including word segmentation results, timestamps, business slot information, keyword labels or confidence scores.

[0096] In the embodiments of the present application, the updated dialect recognition branch may adopt connectionist temporal classification decoding, attention decoding, Transducer (including Discriminator branch) decoding, or a decoding mechanism based on joint search of acoustic output and language constraints, so as to realize the mapping from acoustic features to text sequences. In the decoding process, the system may constrain candidate results in combination with the banking business vocabulary, account-related entity dictionary, amount expression rules, regional business terms and context phrase templates, so as to improve the recognition accuracy of proprietary vocabularies and key business statements in financial scenarios.

[0097] After the dialect recognition branch completes parameter update, the real-time speech feature set obtained in step S101 is re-input into the updated dialect recognition branch, and the dialect recognition branch executes recognition and inference matching the current dialect type to obtain a dialect recognition result corresponding to the current dialect speech signal.

[0098] For example, in a bank application scenario, the dialect recognition result may be directly presented as text content such as "I want to report the loss of my bank card" and "I want to transfer money to this account", and may be further transmitted to a counter auxiliary terminal, a customer service agent system, an intelligent quality inspection system or a background business processing platform.

[0099] In a possible implementation, the updated dialect recognition branch first outputs a frame-level or subword-level probability distribution according to the real-time speech feature set, and then a decoder generates a target text sequence according to the maximum a posteriori principle.

[0100] For example, for dialect speech with a large number of homophones and near-synophones, the system may perform post-processing in combination with the business scenario context after decoding, for example, correcting "guashi (false)" in the recognition result to "guashi (report loss)", and "zhuanzhang (open a sheet)" to "zhuanzhang (transfer account)", so as to improve the usability of the recognition result.

[0101] In another possible implementation, the dialect recognition result may further be accompanied by time start and end information corresponding to each word segment, so as to be directly used in scenarios such as customer service quality inspection, recording traceback and speech-text alignment.

[0102] If the overall confidence level of the recognition result is lower than the preset threshold, supplementary strategies can be triggered, such as requiring the user to restate, calling backup general recognition parameters for verification, or cross-validating the recognition result with the business context before outputting it to the business system.

[0103] Understandably, since the dialect recognition branch uses target model parameters corresponding to the current dialect type, it is more targeted in syllable boundary localization, local pronunciation difference identification, tone change modeling, and word mapping, thus significantly improving the recognition accuracy and stability in cross-dialect scenarios.

[0104] This application embodiment uses a processing path of first classifying, then updating, and then recognizing, which enables the voice interaction service system to quickly adapt to the pronunciation characteristics of customers in different regions and respond in a timely manner in a single conversation. This reduces the number of times customers repeat themselves, lowers the frequency of manual correction of recognition results by tellers or customer service personnel, and improves business processing efficiency and service consistency while ensuring controllable service response latency.

[0105] The multi-dialect speech recognition method provided in this embodiment acquires the dialect speech signal input by the user and extracts features from the dialect speech signal to obtain the corresponding real-time speech feature set. The real-time speech feature set is then input into the type recognition branch of the dialect recognition model to obtain the dialect type of the dialect speech signal. Based on the dialect type, the target model parameters corresponding to the dialect recognition branch of the dialect recognition model are matched, and the target model parameters are called to update the dialect recognition branch. The real-time speech feature set is then input into the updated dialect recognition branch to obtain the dialect recognition result of the dialect speech signal. This method determines the dialect type through the type recognition branch and dynamically loads the corresponding target model parameters, constructing targeted recognition paths for different types of dialects. This effectively improves the accuracy of multi-dialect speech recognition, while enhancing the generalization ability of the dialect recognition model to different dialects, and improving the adaptation and operating efficiency of the dialect recognition model for different dialects.

[0106] Figure 2 This is a flowchart illustrating the multi-dialect speech recognition method provided in this application. Figure 2 .like Figure 2 As shown, in this embodiment... Figure 1 Based on the embodiments, the multi-dialect speech recognition method is described in detail. The multi-dialect speech recognition method shown in this embodiment includes:

[0107] S201: Acquire historical multi-dialect speech signals and generate speech feature sets corresponding to different dialect types based on the historical multi-dialect speech signals.

[0108] S202: Extract features from multiple speech feature sets to obtain general dialect features and dialect distinguishing features corresponding to different dialect types.

[0109] S203: Based on the common features of dialects and the dialect distinguishing features corresponding to different dialect types, the hybrid acoustic model is trained in layers to obtain a dialect recognition model.

[0110] Among them, historical multi-dialect speech signals are used to represent multi-regional dialect speech data collected during the training phase, which can cover different dialect types such as Cantonese, Sichuan-Chongqing dialect, and Wu dialect.

[0111] The speech feature set is used to represent the sequence of feature vectors extracted from historical multi-dialect speech signals, which facilitates subsequent merging and comparison by dialect type.

[0112] Dialect common features are used to characterize the acoustic rules shared by multiple dialects, while dialect distinguishing features are used to characterize the differences between different dialects in articulation position, tone changes, speech rate and rhythm, and resonance characteristics.

[0113] Understandably, the hybrid acoustic model is the foundational model for dialect recognition. It uses the speech feature sets corresponding to different dialect types as training samples and learns both general dialect features and dialect distinguishing features corresponding to different dialect types through a hierarchical training mechanism. It integrates general modeling capabilities and differentiated modeling capabilities to provide core acoustic modeling capabilities for subsequent dialect recognition.

[0114] Historical multi-dialect speech signals are acquired and preprocessed to perform noise reduction, endpoint detection, and amplitude normalization. Then, speech feature sets corresponding to different dialect types are generated based on the dialect labels recorded during historical data acquisition. For each speech feature set, multiple acoustic feature parameters are jointly extracted, and common frequency nodes are obtained through cross-dialect alignment calculation, thereby obtaining dialect-wide features. At the same time, local spectral gain is calculated for the difference frequency bands between different dialect types, and dialect distinguishing features are extracted by combining local statistics.

[0115] In this embodiment, the dialect recognition model includes a type recognition branch and a dialect recognition branch, both of which are trained based on a hybrid acoustic model. The dialect recognition branch and the type recognition branch work together to complete the dialect recognition for that instance. The type recognition branch is used to learn the dialect category mapping relationship, and the dialect recognition branch is used to learn dialect difference expressions. After training, the dialect recognition model is obtained. Specifically, the dialect recognition branch includes a common feature layer and a unique feature layer. The common feature layer is used to learn general features (i.e., dialect general features), and the unique feature layer is used to learn differentiated features (i.e., dialect distinguishing features).

[0116] Understandably, hybrid acoustic models can learn stable general acoustic rules under the condition of limited historical multi-dialect speech samples, and effectively adapt to the differences of each dialect, thereby improving the accuracy of multi-dialect recognition and cross-dialect generalization ability. At the same time, the hierarchical training method reduces the need to train independent models for different dialects, which not only reduces the probability of misidentification between different dialects, but also reduces training costs and deployment complexity, making it suitable for real-time voice interaction scenarios such as bank counter assistance, remote customer service and self-service terminals.

[0117] In one possible implementation, historical multi-dialect speech signals are processed by framing and windowing, and then time-frequency conversion is performed using Fourier transform to obtain linear spectrograms corresponding to different dialect types. Mel filter banks are used to perform frequency band weighting on the linear spectrograms corresponding to different dialect types to obtain Mel spectrograms corresponding to different dialect types. Logarithmic transform and discrete cosine transform are then performed on the Mel spectrograms corresponding to different dialect types in sequence to obtain speech feature sets corresponding to different dialect types.

[0118] Understandably, frame-by-frame windowing is used to segment continuous speech into short time frames and reduce spectral leakage at frame boundaries; Fourier transform is used to map time-domain speech to frequency-domain amplitude distribution; and linear spectrograms are used to characterize the energy intensity of each frequency component.

[0119] The Mel filter bank is a set of bandpass filters arranged according to the Mel scale. Its function is to simulate the human ear's perception characteristics of different frequency bands and convert the linear spectrum into the Mel spectrum. The logarithmic transform is used to compress the amplitude dynamic range, and the discrete cosine transform is used to further decorrelate and extract low-dimensional representations, thereby forming a set of speech features suitable for subsequent dialect recognition model training.

[0120] After obtaining historical multi-dialect speech signals, endpoint detection and amplitude normalization are performed. The signals are then divided into multiple overlapping frames with a fixed frame length, and these overlapping frames are processed to obtain multiple short-time stationary frames. A Fast Fourier Transform (FFT) is performed on each frame to convert the time-domain speech signal corresponding to the current short-time stationary frame into a linear frequency domain spectrum, obtaining the original frequency energy distribution of the speech for different frames, i.e., the corresponding linear spectrogram. This linear spectrogram is used as the input of a Mel filter bank, which performs nonlinear band weighting on the spectrum to enhance the effective acoustic information in the low and mid-frequency ranges of the speech and weaken irrelevant high-frequency noise. The band-weighted Mel spectrum is then logarithmically processed and subjected to Discrete Cosine Transform (DCT) to extract a set of highly discriminative and stable acoustic features, which facilitates feature differentiation and model learning for different dialect types.

[0121] For example, the acquired historical multi-dialect speech signals are first pre-processed, sequentially performing endpoint detection and amplitude normalization to remove invalid silence segments at the beginning and end of the speech, and to unify the amplitude levels of the speech signals from different dialects, avoiding interference from signal strength differences in feature extraction. The pre-processed speech signals are then framed and windowed, with the frame length preset to 20ms to 30ms and the frame shift set to 10ms to 15ms to ensure reasonable overlap between adjacent speech frames and maintain the continuity of short-term speech information. During windowing, Hanning or Hamming windows can be used for smoothing to reduce spectral leakage. For each windowed speech frame, a Fast Fourier Transform is performed to convert the speech signal from the time domain to the frequency domain, obtaining a linear spectrum corresponding to each frame. This linear spectrum fully preserves the frequency energy distribution characteristics of the original speech. The Fourier Transform formula is shown in Formula 2 below.

[0122]

[0123] in, For Fourier functions, The original signal, This is a sampling rate adjustment factor used to adapt to different speech sampling specifications and ensure consistent frequency domain conversion accuracy.

[0124] A Mel filter bank containing 20 to 40 filter units is used to perform frequency band weighting mapping on the linear spectrum, and fine-grained energy convergence is performed on the core pronunciation range of low and mid frequencies of speech. This aligns with the human ear's auditory characteristics of being sensitive to low frequencies and insensitive to high frequencies, filtering and enhancing the core effective frequency band information of dialect pronunciation to obtain the Mel spectrum corresponding to each dialect. In other words, the nonlinear mapping from physical frequency (i.e., linear spectrum) to Mel frequency is completed through the Mel frequency conversion formula, as shown in Formula 3 below:

[0125]

[0126] in, For the first The frequency response function of a Mel-triangle filter For the first The center frequency of the Mel filter, subscript An independent filtering unit used to uniquely distinguish different frequency bands in a filter bank, thereby achieving fine segmentation of the entire audio segment; , These are the center frequencies of the preceding and following filters adjacent to the current filter, respectively.

[0127] Finally, logarithmic and discrete cosine transforms are performed sequentially on the Mel spectrum to obtain the MFCC speech feature set corresponding to a single speech sample. Each historical speech signal corresponds to a set of features, and multiple sets of speech features are contained within the same dialect type. That is, multiple speech samples correspond to the same category of dialects. The speech samples from multiple categories of dialects are combined to form a training dataset for training the subsequent dialect recognition model. In particular, the logarithmic transform compresses the dynamic range of spectral energy, weakening the interference caused by environmental noise and amplitude fluctuations. Then, the discrete cosine transform removes the correlation between feature dimensions and reduces the feature dimensionality.

[0128] Understandably, by progressively converting time-domain speech into frequency, me-domain, and cepstral domain representations, the differences in pronunciation, resonance, and acoustic distribution among different dialect types can be more fully preserved, while reducing the impact of original speech amplitude fluctuations on modeling. Since the speech feature set has good compressibility and discriminative power, the subsequent dialect recognition model can more stably acquire dialect-related information during type identification and parameter matching, thereby improving the accuracy, generalization ability, and real-time adaptation effect of multi-dialect speech recognition in banking voice service scenarios.

[0129] In this embodiment of the application, the historical multi-dialect speech signal includes multiple historical dialect speech samples, and each historical dialect speech sample can independently generate a set of speech features. Therefore, the same dialect type corresponds to multiple sets of speech features, which constitute multi-sample feature data under that dialect. All feature sets with dialect labels together form a training dataset, which is used for training the subsequent dialect recognition model.

[0130] In one possible implementation, multiple speech feature sets are cross-compared to determine the distinguishing frequency nodes and common frequency nodes among the multiple speech feature sets; local spectral gain is applied to the distinguishing frequency nodes of the speech feature sets corresponding to different dialect types to obtain gain spectrograms corresponding to multiple different dialect types; feature extraction is performed on the multiple gain spectrograms to obtain dialect distinguishing features corresponding to different dialect types; based on multiple common frequency nodes, the inherent acoustic parameters between different dialect types are determined, and feature transformation is performed on the inherent acoustic parameters to obtain dialect common features.

[0131] Understandably, the distinguishing frequency node is used to characterize the frequency positions that are significantly different in multiple speech feature sets, the general frequency node is used to characterize the frequency positions that are common in multiple speech feature sets, the local spectral gain is used to enhance the spectral energy corresponding to the distinguishing frequency node, and the gain spectrogram is used to characterize the spectral result after local spectral gain processing.

[0132] Intrinsic acoustic parameters are used to characterize the inherent acoustic commonalities among different dialect types, and feature transformation is used to convert intrinsic acoustic parameters into dialect-wide features that can be used for model training; wherein, intrinsic acoustic parameters include at least one of the following: fundamental formants, sound vibration frequencies, and human voice energy distribution density.

[0133] When cross-comparing multiple speech feature sets, the Mel spectrum or spectrum vector corresponding to different dialect types is first aligned, and then the amplitude difference, energy difference or correlation coefficient difference of each frequency position is statistically analyzed. Then, the frequency positions that meet the preset difference threshold are selected as distinguishing frequency nodes, and the frequency positions that meet the consistency threshold are selected as general frequency nodes.

[0134] When performing local spectral gain on the distinguishing frequency nodes, a gain coefficient is applied to the corresponding frequency band of the node to amplify the amplitude or energy of that frequency band. The gain coefficient can be a fixed value preset according to different dialect types, or it can be adaptively determined according to the intensity of the difference, thereby forming a gain spectrum map that can highlight the dialect differences. Then, convolutional coding, statistical pooling, or dimensionality reduction mapping are performed on the gain spectrum map to extract the corresponding dialect distinguishing features.

[0135] Based on common frequency nodes, their corresponding resonance positions, periodic vibration characteristics, and energy distribution patterns are extracted. After normalization, splicing, or vector mapping, common dialect features between different dialects are obtained for subsequent use in hybrid acoustic models.

[0136] Therefore, the system can simultaneously retain common and differential acoustic information in multi-dialect speech features, enabling dialect distinguishing features to enhance dialect discrimination ability and dialect general features to improve the shared expressive ability between different dialects, thereby improving the model's generalization ability and recognition stability when dialect data is limited.

[0137] Understandably, local spectral gain can amplify key difference frequency bands and reduce confusion between similar dialects, while the conversion results of inherent acoustic parameters corresponding to universal frequency nodes can provide a unified acoustic representation basis for different dialect types, thereby reducing the complexity of separate modeling for each dialect. At the same time, this method can extract discriminative dialect difference features without relying on a large amount of dialect annotation data, and simultaneously construct shareable universal acoustic features, thereby improving the accuracy, robustness and adaptation efficiency of multi-dialect speech recognition, which is especially suitable for application scenarios such as bank voice services that require rapid response and cover multiple regional accents.

[0138] In another possible implementation, the extraction of dialect general features and dialect distinguishing features can also be achieved by using a multi-layer convolutional neural network to perform local feature traversal and global temporal modeling on each speech feature set to extract and obtain dialect general features; and by using a multi-layer convolutional neural network to extract differentiated acoustic features on each speech feature set to obtain dialect distinguishing features corresponding to different dialect types.

[0139] In this embodiment, a multi-layer convolutional neural network is used to jointly model the local acoustic patterns and the overall temporal relationship in the speech feature set. Its convolutional layers are used to extract information such as spectral energy distribution, formant trajectory and fundamental frequency change, pooling layers are used to compress redundant features and retain the main acoustic representations, and temporal aggregation layers are used to form a correlation representation across time spans.

[0140] Local feature traversal refers to the layer-by-layer scanning extraction of fine-grained acoustic differences within each time segment, while global temporal modeling refers to encoding the continuity and dependency between adjacent speech frames, thereby enabling the stable representation of common pronunciation patterns in different dialects.

[0141] Differential acoustic feature extraction refers to the explicit separation of differences in pitch direction, aspiration features, vowel resonance, and rhythmic pauses among different dialect types, thereby forming dialect distinguishing features that can differentiate speech patterns such as Cantonese, Sichuan-Chongqing dialect, and Wu dialect.

[0142] For example, multiple speech feature sets, consisting of Mel spectrum, cepstral coefficients, or combinations thereof, are input into a multi-layer convolutional neural network configured in parallel. The network branch used to extract common dialect features can perform hierarchical convolution on the input features using continuous convolutional kernels. The kernel size can be set to alternate along the frequency and time dimensions to enhance the coverage of local common acoustic structures. Then, through max pooling or average pooling, the features of multiple frames are mapped into a stable temporal representation, and a common feature vector is output, which is the common dialect feature. The network branch used to extract dialect distinguishing features can be configured with different convolutional parameters or channel weights for different dialect types, making it more sensitive to the difference frequency bands, specific tone contours, and local energy abrupt changes in a specific dialect. The output corresponding to each dialect is mapped into a distinguishing feature vector, which is the dialect distinguishing feature of different dialect types.

[0143] The number of layers in the convolutional network can be set from three to eight layers depending on the size of the training samples, so as to balance feature representation ability and inference efficiency. In practical applications, the network can also choose other numbers of layers and convolutional structures, which are not limited in this embodiment.

[0144] Understandably, by obtaining a universal acoustic representation shared across dialects (i.e., dialect-wide features) and a specific representation for distinguishing different dialects (i.e., dialect-distinguishing features) within the same feature extraction framework, the universal acoustic representation is used to support the subsequent dialect recognition model in learning consistent pronunciation rules. The specific representations for different dialects are used to enhance the model's ability to distinguish dialect boundaries and easily confused syllables, thereby improving the model's generalization and recognition accuracy under the condition of limited dialect samples. It also reduces the model's training and maintenance costs and improves the recognition stability and response efficiency of multi-dialect inputs in banking voice service scenarios.

[0145] In one possible implementation, the dialect recognition model includes a type recognition branch and a dialect recognition branch. By determining the dialect types corresponding to historical multi-dialect speech signals, and training the hybrid acoustic model based on dialect types, common dialect features, and dialect distinguishing features of different dialects, the type recognition branch is obtained. Based on the common dialect features, a transfer learning strategy is used to optimize the parameters of the hybrid acoustic model to obtain an initial recognition branch. Based on the dialect distinguishing features corresponding to different dialect types, the initial recognition branch is adapted and optimized to obtain the dialect recognition branch.

[0146] Among them, the dialect general features are used to represent the basic acoustic rules shared between different dialects, the dialect distinguishing features are used to represent the differences between dialects in formant position, tone contour, speech rate rhythm and local phoneme pronunciation, the transfer learning strategy is used to transfer the general acoustic representation from the basic model to the recognition model and maintain parameter inheritance, and the adaptation optimization of the initial recognition branch is used to adjust the model parameters in a targeted manner according to the distinguishing features of specific dialects.

[0147] When training the dialect recognition model, specifically the type recognition branch and the dialect recognition branch, the dialect type is first used as a supervisory signal input to the hybrid acoustic model, enabling the model to learn the category discrimination boundaries of different dialects. Then, the shared layer and the classification layer are jointly optimized by combining the common dialect features and the distinctive dialect features, thus forming the type recognition branch. Next, the parameters obtained based on the common dialect features are transferred to the same model framework, and the intermediate layer or the shared layer is inherited and fine-tuned to retain the common acoustic expressive ability of each dialect and reduce the dependence on large-scale labeled data, thereby obtaining the initial recognition branch.

[0148] After obtaining the initial recognition branch, for any dialect type, the corresponding dialect distinguishing features are input, and the weight matrix, bias term or normalization parameter in the initial recognition branch is updated in a targeted manner. A dialect adaptation layer can also be introduced to correct the output distribution to enhance the ability to distinguish dialect-specific phoneme variants, and finally obtain the dialect recognition branch.

[0149] For example, historical multi-dialect speech signals can be from bank counter recordings, customer service call recordings, or self-service terminal interaction recordings, and carry manually labeled dialect type tags; the hybrid acoustic model is used to simultaneously carry dialect type recognition and dialect content recognition capabilities, and it can adopt a structure combining convolutional neural networks and recurrent neural networks, or a pure deep feedforward network structure, which is not limited in this application embodiment.

[0150] Understandably, this hierarchical training method allows type recognition capability, general recognition capability, and dialect-specific adaptation capability to be formed and progressively enhanced at different training stages. The type recognition branch can improve the accuracy of distinguishing historical multi-dialect speech types, the initial recognition branch can inherit general acoustic knowledge and reduce the cost of repeated training, and the dialect recognition branch can perform fine-grained adaptation for specific dialects, thereby improving the accuracy, generalization capability, and deployment efficiency of multi-dialect speech recognition, which is suitable for multi-dialect interaction scenarios such as banking.

[0151] In one possible implementation, after training the hybrid acoustic model based on dialect type, common dialect features, and dialect distinguishing features of different dialects to obtain a type recognition branch, the dialect complexity features of different dialect types are determined based on historical multi-dialect speech signals; for any dialect type, the parameter sharing ratio of the dialect recognition model is determined based on the dialect complexity features of the dialect type.

[0152] Among them, dialect complexity features are used to characterize the degree of complexity of different dialect types at the phonetic level and serve as the basis for setting the parameter sharing ratio; dialect complexity features include at least one of tone fluctuation degree, lexical structure difference degree and pronunciation regularity degree.

[0153] The parameter sharing ratio is the weight ratio of model parameters corresponding to the common features and distinguishing features of different dialects during the dialect recognition branch training process. In other words, the parameter sharing ratio is used to represent the weight ratio between the common parameters and distinguishing parameters during the dialect recognition branch training. This ratio can be used to control the update intensity of the shared layer and the exclusive layer, so that the model can retain the common representation across dialects while enhancing its ability to characterize the differences of specific dialects.

[0154] Understandably, historical multi-dialect speech signals can come from bank counter recordings, telephone customer service recordings, or self-service terminal interactive recordings. After being labeled by dialect, the differences in fundamental frequency fluctuation, pitch span, tone transition frequency, frequency of common word substitution, and syllable pronunciation stability of each dialect can be statistically analyzed to form a corresponding complexity representation.

[0155] For example, historical multi-dialect speech signals are first preprocessed and categorized into dialects. Then, based on statistical analysis or a learning-based evaluation model, the complexity scores of each dialect type are output, and the degree of tone fluctuation, lexical structure difference, and pronunciation regularity are mapped to corresponding quantitative indicators. Based on the quantitative results, the parameter ratios of common features and dialect distinguishing features for different dialects are set. For example, when the complexity of a dialect is high, the training weights of the parameters corresponding to the distinguishing features can be increased, so that the model pays more attention to the unique acoustic shifts of that dialect. When a dialect is close to the common speech pattern, the weights of the parameters corresponding to the common features can be increased to enhance the model's transferability. Therefore, this dialect recognition model can automatically adjust the parameter sharing relationship according to the dialect complexity, thereby adapting to the recognition needs of different dialects.

[0156] Understandably, by setting the parameter sharing ratio driven by complexity features, the hybrid acoustic model can establish a parameter allocation mechanism that matches the degree of dialect differences during the training phase, thereby forming a more stable type discrimination ability and a stronger dialect adaptability in subsequent recognition.

[0157] Since the parameter allocation is directly related to dialect complexity, the model can reduce its reliance on low-value sharing and overly specific modeling, improve training efficiency and recognition accuracy under imbalanced multi-dialect samples, and reduce the possibility of misrecognition caused by dialect switching in banking voice service scenarios.

[0158] S204: Acquire the dialect speech signal input by the user, and extract features from the dialect speech signal to obtain the corresponding real-time speech feature set.

[0159] S205: Input the real-time speech feature set into the type recognition branch of the dialect recognition model to obtain the dialect type of the dialect speech signal.

[0160] Steps S204-S205 are similar to steps S101-S102 above, and will not be described again here.

[0161] S206: Based on dialect type, match the general model parameters and difference model parameters corresponding to the dialect recognition branch.

[0162] S207: Determine the dialect complexity features of dialect speech signals, and based on the dialect complexity features of dialect speech signals, determine the parameter sharing ratio of general model parameters and distinguishing model parameters.

[0163] S208: Determine the target model parameters corresponding to the dialect recognition branch based on the parameter sharing ratio, general model parameters, and distinguishing model parameters.

[0164] S209: Update the dialect recognition branch by calling the target model parameters.

[0165] Understandably, dialect complexity features are quantitative features used to characterize the degree of difference between different dialects in terms of pronunciation, tone, syllables, vocabulary and grammar, and can reflect the difficulty of recognizing different dialects.

[0166] The general model parameters are the model parameters obtained by training based on the common features of dialects during the training phase. The general model parameters are used to characterize the set of parameters learned from the common acoustic laws of multiple dialects. They correspond to the common characteristics of each dialect in phoneme structure, basic formants and energy distribution.

[0167] The distinguishing model parameters are specific model parameters obtained by individual adaptation training based on the dialect distinguishing features of each type of dialect during the training phase. The distinguishing model parameters are used to represent a set of specialized parameters for a single dialect type, which correspond to the adaptation results of that dialect in terms of tone direction, pronunciation regularity and lexical expression differences.

[0168] The parameter sharing ratio is used to describe the fusion weight of the general model parameters and the difference model parameters in the dialect recognition branch. The weight can be used to control the contribution strength of the two to the target recognition result.

[0169] After identifying the dialect type, the general model parameters and the distinguishing model parameters corresponding to the dialect type are retrieved from the parameter library and mapped to the parameter interface of the same recognition branch. Then, the current dialect speech signal is analyzed and processed to extract the tone fluctuation degree, vocabulary structure difference degree and pronunciation regularity corresponding to the dialect speech signal, thereby obtaining the corresponding dialect complexity features.

[0170] The dialect complexity feature is input into a preset mapping relationship or a lightweight decision network, which outputs the sharing ratio of general model parameters and distinguishing model parameters. Based on this ratio, the two are weighted and combined to obtain the target model parameters.

[0171] Understandably, the way the target model parameters are determined enables the recognition branch to enhance the role of the distinguishing model parameters when facing dialect speech with high complexity, and to enhance the role of the general model parameters when the dialect features are relatively stable, so as to match the acoustic distribution of the current input.

[0172] For example, during the training phase, the general features of a dialect are learned from the acoustic patterns shared by historical multi-dialect speech, while the parameters of the distinguishing model are formed by adapting different dialect samples individually. The two can be stored in the model parameter file and the dialect adaptation parameter file, respectively. The pre-established mapping rule between dialect complexity scores and parameter sharing ratios can be as follows: the dialect complexity features are quantified into scores of 0-10, where scores of 0-3 correspond to low complexity, with the general model parameters and distinguishing model parameters accounting for 0.7 and 0.3, respectively; scores of 3-7 correspond to medium complexity, with both types of parameters accounting for 0.5; and scores of 7-10 correspond to high complexity, with the two types of parameters accounting for 0.3 and 0.7, respectively.

[0173] If the current dialect complexity feature score is 8.2, according to the mapping rule, the proportion of general parameters is determined to be 0.3 and the proportion of distinguishing parameters is 0.7. The two types of parameters are weighted and fused according to this ratio to obtain the target model parameters. The fusion method can be linear weighting, gated selection, or hierarchical residual superposition so that the output can be directly used for identifying branch updates.

[0174] Understandably, this parameter matching and sharing mechanism can reduce the computational burden of building a complete model for each dialect separately, while maintaining a unified model framework and improving the adaptability to complex dialect inputs, thereby improving recognition accuracy and reducing parameter maintenance costs.

[0175] S210: Input the real-time speech feature set into the updated dialect recognition branch to obtain the dialect recognition result of the dialect speech signal.

[0176] Step S210 is similar to step S104 above, and will not be described again here.

[0177] The multi-dialect speech recognition method provided in this embodiment acquires historical multi-dialect speech signals and generates speech feature sets corresponding to different dialect types based on these signals. It then extracts features from these multiple speech feature sets to obtain general dialect features and dialect distinguishing features corresponding to different dialect types. Based on these general and distinguishing features, it performs layered training on a hybrid acoustic model to obtain a dialect recognition model. Finally, it acquires user-input dialect speech signals and extracts features from them to obtain corresponding real-time speech feature sets. These real-time speech feature sets are then input into the dialect recognition model. The method employs a type recognition branch to identify the dialect type of the speech signal. Based on the dialect type, it matches the general model parameters and discrimination model parameters corresponding to the dialect recognition branch. It then determines the dialect complexity features of the speech signal and, based on these features, determines the parameter sharing ratio between the general and discrimination model parameters. Based on the parameter sharing ratio, the general and discrimination model parameters, it determines the target model parameters corresponding to the dialect recognition branch and updates the branch using these parameters. Finally, it inputs the real-time speech feature set into the updated dialect recognition branch to obtain the dialect recognition result. This method, through the synergistic optimization of hybrid acoustic models and transfer learning, enables the model to adapt to the acoustic feature differences of different dialects, effectively improving the accuracy of multi-dialect speech recognition. It not only significantly enhances the generalization ability and recognition stability of multi-dialect speech recognition in complex scenarios but also improves service efficiency and customer satisfaction.

[0178] Figure 3 This is a schematic diagram of the multi-dialect speech recognition device provided in this application. Figure 3 As shown, this application provides a multi-dialect speech recognition device, the multi-dialect speech recognition device 300 including:

[0179] The acquisition module 301 is used to acquire historical multi-dialect speech signals.

[0180] The processing module 302 is used to generate speech feature sets corresponding to different dialect types based on historical multi-dialect speech signals.

[0181] The processing module 302 is also used to extract features from multiple speech feature sets to obtain general dialect features and dialect distinguishing features corresponding to different dialect types.

[0182] The processing module 302 is also used to perform hierarchical training on the hybrid acoustic model based on the general features of dialects and the dialect distinguishing features corresponding to different dialect types, so as to obtain a dialect recognition model.

[0183] The acquisition module 301 is also used to acquire dialect voice signals input by the user.

[0184] The processing module 302 is also used to input the dialect speech signal into the dialect recognition model to obtain the dialect recognition result of the dialect speech signal.

[0185] In one possible embodiment, the acquisition module 301 is further configured to acquire historical multi-dialect speech signals and generate speech feature sets corresponding to different dialect types based on the historical multi-dialect speech signals.

[0186] The processing module 302 is also used to extract features from multiple speech feature sets to obtain general dialect features and dialect distinguishing features corresponding to different dialect types.

[0187] The processing module 302 is also used to perform hierarchical training on the hybrid acoustic model based on the general features of dialects and the dialect distinguishing features corresponding to different dialect types, so as to obtain a dialect recognition model.

[0188] In one possible embodiment, the processing module 302 is further configured to perform frame-by-frame windowing processing on the historical multi-dialect speech signal and perform time-frequency conversion through Fourier transform to obtain linear spectrum diagrams corresponding to different dialect types.

[0189] The processing module 302 is also used to perform frequency band weighting processing on the linear spectra corresponding to different dialect types using the Mel filter bank to obtain the Mel spectra corresponding to different dialect types.

[0190] The processing module 302 is also used to perform logarithmic transformation and discrete cosine transformation on the Mel spectrum corresponding to different dialect types in sequence to obtain the speech feature set corresponding to different dialect types.

[0191] In one possible embodiment, the processing module 302 is further configured to perform cross-comparison of multiple speech feature sets to determine the distinguishing frequency nodes and common frequency nodes among the multiple speech feature sets.

[0192] The processing module 302 is also used to perform local spectral gain on the distinguishing frequency nodes of the speech feature sets corresponding to different dialect types, so as to obtain gain spectrum diagrams corresponding to multiple different dialect types.

[0193] The processing module 302 is also used to extract features from multiple gain spectrograms to obtain dialect distinguishing features corresponding to different dialect types.

[0194] The processing module 302 is also used to determine the inherent acoustic parameters between different dialect types based on multiple common frequency nodes, and to perform feature transformation on the inherent acoustic parameters to obtain common dialect features;

[0195] The inherent acoustic parameters include at least one of the following:

[0196] Basic resonance peaks, sound vibration frequencies, and the energy distribution density of human voices.

[0197] In one possible embodiment, the processing module 302 is further configured to use a multi-layer convolutional neural network to perform local feature traversal and global temporal modeling on each speech feature set, and extract and obtain dialect-specific features.

[0198] The processing module 302 is also used to extract differential acoustic features from each speech feature set using a multi-layer convolutional neural network to obtain dialect distinguishing features corresponding to different dialect types.

[0199] In one possible embodiment, the processing module 302 is further configured to determine the dialect type corresponding to the historical multi-dialect speech signal, and to train the hybrid acoustic model based on the dialect type, common dialect features, and dialect distinguishing features of different dialects to obtain a type recognition branch.

[0200] The processing module 302 is also used to optimize the parameters of the hybrid acoustic model based on the common features of dialects and adopt a transfer learning strategy to obtain the initial recognition branch.

[0201] The processing module 302 is also used to adapt and optimize the initial recognition branch based on the dialect distinguishing features corresponding to different dialect types, so as to obtain the dialect recognition branch.

[0202] In one possible embodiment, the processing module 302 is further configured to determine the dialect complexity features of different dialect types based on historical multi-dialect speech signals.

[0203] The processing module 302 is also used to determine the parameter sharing ratio of the dialect recognition model based on the dialect complexity features of the dialect type for any dialect type;

[0204] Among them, dialect complexity features include at least one of tone fluctuation degree, lexical structure difference degree and pronunciation regularity degree; parameter sharing ratio is the weight ratio of model parameters corresponding to common features and dialect distinguishing features of different dialects during the dialect recognition branch training process.

[0205] In one possible embodiment, the processing module 302 is further configured to match the general model parameters and the difference model parameters corresponding to the dialect recognition branch based on the dialect type.

[0206] The processing module 302 is also used to determine the dialect complexity features of the dialect speech signal.

[0207] Processing module 302 is also used to determine the parameter sharing ratio of general model parameters and distinguishable model parameters based on the dialect complexity features of dialect speech signals;

[0208] Among them, the general model parameters are the model parameters obtained by training based on the general features of dialects during the training phase; the distinguishing model parameters are the specific model parameters obtained by training separately based on the distinguishing features of each type of dialect during the training phase.

[0209] The multi-dialect speech recognition device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0210] Figure 4 A schematic diagram of the structure of the electronic device provided in this application. Figure 4 As shown, the electronic device 400 provided in this embodiment includes at least one processor 401 and a memory 402. Optionally, the device 400 further includes a communication component 403. The processor 401, memory 402, and communication component 403 are connected via a bus 404.

[0211] In a specific implementation, at least one processor 401 executes computer execution instructions stored in memory 402, causing at least one processor 401 to perform the above-described method.

[0212] The specific implementation process of processor 401 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0213] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0214] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0215] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0216] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0217] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0218] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0219] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0220] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0221] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0222] When integrated units / modules are implemented in hardware, the hardware can be digital circuits, analog circuits, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the processor can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the storage unit can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc.

[0223] If the integrated unit / module is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0224] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0225] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0226] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A multi-dialect speech recognition method, characterized in that, include: Acquire the dialect speech signal input by the user, and extract features from the dialect speech signal to obtain the corresponding real-time speech feature set; The real-time speech feature set is input into the type recognition branch of the dialect recognition model to obtain the dialect type of the dialect speech signal; Based on the dialect type, the target model parameters corresponding to the dialect recognition branch of the dialect recognition model are matched, and the target model parameters are called to update the dialect recognition branch; The real-time speech feature set is input into the updated dialect recognition branch to obtain the dialect recognition result of the dialect speech signal.

2. The method according to claim 1, characterized in that, Before acquiring the dialect speech signal input by the user, the method further includes: Acquire historical multi-dialect speech signals, and based on the historical multi-dialect speech signals, generate speech feature sets corresponding to different dialect types; Feature extraction is performed on multiple sets of speech features to obtain general dialect features and dialect distinguishing features corresponding to different dialect types. Based on the common features of the dialects and the dialect distinguishing features corresponding to the different dialect types, the hybrid acoustic model is trained in layers to obtain a dialect recognition model.

3. The method according to claim 2, characterized in that, The step of generating speech feature sets corresponding to different dialect types based on the historical multi-dialect speech signals includes: The historical multi-dialect speech signals are processed by frame segmentation and windowing, and then time-frequency conversion is performed by Fourier transform to obtain linear spectrum diagrams corresponding to different dialect types. The linear spectra corresponding to the different dialect types are obtained by using a Mel filter bank to perform frequency band weighting on the linear spectra. Logarithmic transform and discrete cosine transform are performed sequentially on the Mel spectra corresponding to the different dialect types to obtain the speech feature sets corresponding to the different dialect types.

4. The method according to claim 3, characterized in that, The step of extracting features from multiple speech feature sets to obtain general dialect features and dialect distinguishing features corresponding to different dialect types includes: Cross-comparison is performed on multiple speech feature sets to determine the distinguishing frequency nodes and common frequency nodes among the multiple speech feature sets; Local spectral gain is applied to the distinguishing frequency nodes of the speech feature sets corresponding to the different dialect types to obtain multiple gain spectrum diagrams corresponding to the different dialect types; Feature extraction is performed on multiple gain spectrograms to obtain dialect distinguishing features corresponding to different dialect types; Based on multiple common frequency nodes, the inherent acoustic parameters between the different dialect types are determined, and feature transformation is performed on the inherent acoustic parameters to obtain the common dialect features. The inherent acoustic parameters include at least one of the following: Basic resonance peaks, sound vibration frequencies, and the energy distribution density of human voices.

5. The method according to claim 2, characterized in that, The step of extracting features from multiple speech feature sets to obtain general dialect features and dialect distinguishing features corresponding to different dialect types includes: A multi-layer convolutional neural network is used to perform local feature traversal and global temporal modeling on each of the aforementioned speech feature sets to extract and obtain the dialect-specific features. A multi-layer convolutional neural network is used to extract differential acoustic features from each of the aforementioned speech feature sets to obtain the dialect distinguishing features corresponding to the different dialect types.

6. The method according to claim 2, characterized in that, The method involves performing hierarchical training on the hybrid acoustic model based on the common dialect features and the dialect distinguishing features corresponding to different dialect types to obtain a dialect recognition model, including: The dialect type corresponding to the historical multi-dialect speech signal is determined, and the hybrid acoustic model is trained based on the dialect type, the common features of the dialect, and the dialect distinguishing features of different dialects to obtain the type recognition branch; Based on the common features of the dialect, a transfer learning strategy is used to optimize the parameters of the hybrid acoustic model to obtain an initial recognition branch; Based on the dialect distinguishing features corresponding to the different dialect types, the initial recognition branch is adapted and optimized to obtain the dialect recognition branch.

7. The method according to claim 6, characterized in that, After training the hybrid acoustic model based on the dialect type, the common features of the dialect, and the dialect distinguishing features of different dialects to obtain the type recognition branch, the method further includes: Based on the historical multi-dialect speech signals, the dialect complexity features of the different dialect types are determined; For any dialect type, the parameter sharing ratio of the dialect recognition model is determined based on the dialect complexity features of the dialect type; The dialect complexity features include at least one of tone fluctuation, lexical structure difference, and pronunciation regularity; the parameter sharing ratio is the weight ratio of the model parameters corresponding to the common features and the distinguishing features of different dialects during the training of the dialect recognition branch.

8. The method according to claim 1, characterized in that, The step of matching the target model parameters corresponding to the dialect recognition branch based on the dialect type includes: Based on the dialect type, match the general model parameters and the difference model parameters corresponding to the dialect recognition branch; Determine the dialect complexity features of the dialect speech signal, and based on the dialect complexity features of the dialect speech signal, determine the parameter sharing ratio of the general model parameters and the distinguishing model parameters; Based on the parameter sharing ratio, the general model parameters, and the distinguishing model parameters, the target model parameters corresponding to the dialect recognition branch are determined; The general model parameters are model parameters obtained by training based on the general features of dialects during the training phase; the distinguishing model parameters are specific model parameters obtained by training based on the distinguishing features of each type of dialect individually during the training phase.

9. A multi-dialect speech recognition device, comprising: The acquisition module is used to acquire historical multi-dialect speech signals; The processing module is used to generate speech feature sets corresponding to different dialect types based on the historical multi-dialect speech signals; The processing module is also used to extract features from multiple speech feature sets to obtain general dialect features and dialect distinguishing features corresponding to different dialect types. The processing module is also used to perform hierarchical training on the hybrid acoustic model based on the general dialect features and the dialect distinguishing features corresponding to the different dialect types, so as to obtain a dialect recognition model. The acquisition module is also used to acquire dialect voice signals input by the user; The processing module is further configured to input the dialect speech signal into the dialect recognition model to obtain the dialect recognition result of the dialect speech signal.

10. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-8.

12. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method according to any one of claims 1-8.

Images