Dialect recognition method, device and equipment fusing vocal tone features and medium
By introducing acoustic accent feature fusion module and hierarchical accent extraction network in speech recognition technology, combining pre-trained models and unique vocabulary lists, the problem of difficulty in identifying dialects with strong accent features in the existing technology is solved, and a few regions are realized High accuracy recognition of dialects.
Patent Information
- Application Number
- CN202411198396.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-29
- Publication Date
- 2025-01-07
AI Technical Summary
Existing speech recognition technology is difficult to effectively capture the accent features of dialect information, resulting in low accuracy in the recognition of dialects with strong accent features, especially in a few areas, with high error rates and lack of designs for dialects in a few areas, so it is impossible to cover the unique Expression of words.
The acoustic accent feature fusion module is adopted to integrate speech and tone features through multi-layer Conformer module, hierarchical accent extraction network and cross-attention mechanism, and combine speech and tone features with pre-trained speech models and unique vocabulary lists to perform speech recognition, improving dialects for a few regions identification accuracy.
The low error rate recognition of dialects in a few regions is achieved. By accurately capturing the tone characteristics and building special vocabulary lists, the word error rate and word error rate are significantly reduced, and the accuracy of dialect recognition is improved.
Smart Images
Figure CN119274538A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of speech recognition technology, and in particular to a dialect recognition method, device, equipment and medium integrating intonation features. Background Art
[0002] ASR (automatic speech recognition) technology can recognize speech input as corresponding text, which provides great convenience for people's lives. The current method is based on the encoder-decoder architecture design. By extracting speech features (such as Fbank, mfcc), it is sent to the encoding network with the conformer network as the backbone to deeply extract speech features and form a deep fusion vector. The vector will pass through the decoder and perform vocabulary mapping under the existing vocabulary, that is, mapping the vector to a single word or phrase to complete the prediction. This prediction process can also be called search. Common methods include beam search and greedy search. At the same time, in terms of training loss design, the current technology is mainly based on CTC loss. These technologies have achieved good results in Mandarin and regional dialects with a large number of annotated corpora.
[0003] However, current technologies lack the ability to capture dialect information and to deeply integrate dialect features with semantic information. In order to recognize dialects with natural dialect features, a large amount of labeled data is often required for training to counter the mapping uncertainty caused by the lack of dialect feature extraction, especially for local dialects with very strong dialect features. This makes it very difficult to directly apply existing technologies to Ningbo dialect recognition.
[0004] At the same time, these technologies are mainly designed for Mandarin and widely popular regional dialects (such as Cantonese and Minnan dialect), and lack design for special dialects in a few regions. The pre-established vocabulary of these technologies cannot cover some text expressions unique to the dialect, so when these existing technologies are directly used for dialect recognition in these minority regions, the error rate will still be very high. Summary of the invention
[0005] The purpose of the present invention is to provide a dialect recognition method, device, equipment and medium integrating intonation features, so as to improve the recognition accuracy of special dialects in a few regions.
[0006] The purpose of the present invention can be achieved by the following technical solutions:
[0007] A dialect recognition method integrating intonation features comprises the following steps:
[0008] Get the dialect voice input to be recognized;
[0009] Perform speech preprocessing on the input speech to obtain speech features;
[0010] Based on speech features, the acoustic and intonation feature fusion module is used to fuse acoustic and intonation features to obtain a fused feature vector.
[0011] The decoding module is used to decode the fused feature vector and output the speech recognition result of the dialect.
[0012] The speech preprocessing comprises the following steps:
[0013] Using the trained speech pre-training network, extract the speech pre-training features of the input speech;
[0014] Extract fbank features of input speech;
[0015] The speech pre-training features and fbank features are initially fused to form input features;
[0016] The input features are framed, windowed, and convolved to obtain speech features.
[0017] The acoustic intonation feature fusion module includes a primary acoustic encoder composed of multiple layers of Conformer modules, a hierarchical intonation extraction network and a fusion module, wherein the primary acoustic encoder uses the speech features obtained by preprocessing as input to extract acoustic features, the hierarchical intonation extraction network uses the speech features obtained by preprocessing and the acoustic features output by the primary acoustic encoder to extract intonation features, and the fusion module fuses the acoustic features output by the primary acoustic encoder and the intonation features output by the hierarchical intonation extraction network.
[0018] The primary acoustic encoder includes a first feature extraction network, an intermediate downsampling layer, and a second feature extraction network connected in sequence, wherein the first feature extraction network and the second feature extraction network are both composed of a multi-layer improved Conformer module, and the improved Conformer module sets a sliding convolution unit between the GLU activation layer and the BN layer of the basic Conformer network convolution module, so that the features focus on the features of other time steps through a dynamic window.
[0019] The hierarchical intonation extraction network extracts the intonation features from the input features in a hierarchical form, and performs micro-correction in combination with the acoustic features output by the primary acoustic encoder. Specifically, the hierarchical intonation extraction network is composed of a plurality of intonation extraction modules, and the intonation extraction modules have the same number of layers as the Conformer modules. The intonation extraction modules extract preliminary intonation features based on the input, and use the acoustic features extracted by the Conformer modules of the same layer for transformation, projection, and splicing correction, which are used as the input of the intonation extraction modules of the next layer. The intonation extraction modules of the first layer take the speech features as input, and the intonation extraction modules of the last layer output the final intonation features.
[0020] The fusion module is constructed based on the cross-attention mechanism, including two layers of cross-attention mechanisms. In the first layer of cross-attention, acoustic features are used as key values and accent features are used as queries to obtain intermediate results. In the second layer of cross-attention, the intermediate results are used as queries, and the acoustic features and accent features that have undergone shallow cross-attention are used as key values for fusion, and a fused feature vector is output.
[0021] The decoding module includes a decoder and a vocabulary. The decoder decodes the fused feature vector based on CTC loss and beam search mechanism, and constructs the relationship between phonemes and text through phoneme-level features. The vocabulary is established based on a dialect-related corpus to accurately include the unique text of the corresponding dialect and construct a vocabulary distribution adapted to the dialect.
[0022] A dialect recognition device integrating intonation features, comprising:
[0023] An input module, used to obtain the dialect speech input to be recognized;
[0024] A preprocessing module is used to perform speech preprocessing on the input speech to obtain speech features;
[0025] A feature extraction and fusion module is used to fuse acoustic and intonation features based on speech features using an acoustic intonation feature fusion module to obtain a fused feature vector;
[0026] The output module is used to decode the fused feature vector using the decoding module and output the speech recognition result of the dialect.
[0027] An electronic device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the method described above when executing the program.
[0028] A computer-readable storage medium stores a computer program, which implements the method described above when executed by a processor.
[0029] Compared with the prior art, the present invention has the following beneficial effects:
[0030] (1) The present invention utilizes a pre-trained speech model to extract pre-trained speech features, which adapts to the special distribution of dialect speech data. At the same time, the pre-trained features are fused with the fbank features of the mathematical method as model feature input, increasing the adaptability of the features to the special distribution of data and the robustness of the decoder.
[0031] (2) The hierarchical accent feature extraction network of the present invention extracts multi-granularity accent features through a multi-layer design to accurately capture the accent specificity of dialect speech data.
[0032] (3) The present invention designs an accent acoustic fusion module, which integrates the accent features into the acoustic features, so that the unique accent information of the dialect is captured by the network and integrated into the vector predicted by the encoder. Compared with the traditional method, this method integrates the accent features that are different from the dialect and the universal features of Mandarin into the network, so that the decoder can better establish the relationship between the phoneme features and the text.
[0033] (4) Taking into account the particularity of dialects in a few regions, the present invention constructs a special vocabulary, which is established by collecting a large amount of dialect-specific texts to conform to the distribution of the corresponding dialect, so as to better achieve vocabulary mapping.
[0034] (5) By comparing with other methods on dialect test data from a few regions, the present invention achieves the lowest character error rate and word error rate, thus realizing effective dialect recognition tasks. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1 is a flow chart of the method of the present invention;
[0036] Figure 2 It is a schematic diagram of the pretreatment process of the present invention;
[0037] Figure 3 Schematic diagram of the structure of the acoustic tone feature fusion module of the present invention;
[0038] Figure 4 It is a schematic diagram of the structure of a primary acoustic encoder of the present invention;
[0039] Figure 5 This is a schematic diagram of the structure of the improved Conformer module of the present invention;
[0040] Figure 6 Schematic diagram of the hierarchical tone extraction network structure of the present invention. DETAILED DESCRIPTION
[0041] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.
[0042] This embodiment provides a dialect recognition method integrating intonation features, such as Figure 1 As shown, the following steps are included:
[0043] S1, obtaining the dialect speech input to be recognized.
[0044] S2, performing speech preprocessing on the input speech to obtain speech features.
[0045] like Figure 2As shown, speech preprocessing includes the following steps:
[0046] S21, extracting speech pre-training features of the input speech using the trained speech pre-training network;
[0047] S22, extracting fbank features of input speech using traditional mathematical methods;
[0048] S23, preliminarily fusing the speech pre-training features with the fbank features to form input features;
[0049] S24, the input features are framed, windowed, and convoluted down-sampled to obtain speech features, which are sent to the subsequent acoustic intonation feature fusion module.
[0050] The advantage of using a speech pre-trained network is that it can be pre-trained on a large amount of unlabeled speech data to extract speech features. Compared with the fbank features extracted by traditional mathematical methods, this feature can better capture the special distribution of dialect data for better use by the decoder.
[0051] S3, based on the speech features, uses the acoustic and intonation feature fusion module to fuse the acoustic and intonation features to obtain a fused feature vector.
[0052] like Figure 3 As shown, the acoustic intonation feature fusion module includes a primary acoustic encoder composed of multiple layers of Conformer modules, a hierarchical intonation extraction network and a fusion module, wherein the primary acoustic encoder uses the speech features obtained by preprocessing as input to extract acoustic features, the hierarchical intonation extraction network uses the speech features obtained by preprocessing and the acoustic features output by the primary acoustic encoder to extract intonation features, and the fusion module fuses the acoustic features output by the primary acoustic encoder and the intonation features output by the hierarchical intonation extraction network.
[0053] like Figure 4 As shown, the primary acoustic encoder includes a first feature extraction network, an intermediate downsampling layer, and a second feature extraction network connected in sequence, and extracts acoustic features at multiple depths, wherein the first feature extraction network and the second feature extraction network are both composed of multiple layers of improved Conformer modules, and the number of layers of the improved Conformer modules in the first feature extraction network and the second feature extraction network can be the same or different. In this embodiment, the number of layers of the improved Conformer modules used by both is preferably selected to be N. Figure 5 As shown in the figure, the improved Conformer module sets a sliding convolution unit between the GLU activation layer and the BN layer of the basic Conformer network convolution module, and uses a dynamic window to make the features focus on the features of other time steps to better extract features.
[0054] The hierarchical tone extraction network extracts tone features from the input features in a hierarchical form and performs micro-correction based on the acoustic features output by the primary acoustic encoder. Figure 6 As shown, the hierarchical accent extraction network is composed of multiple accent extraction modules. The accent extraction module has the same number of layers as the Conformer module (set to 2N in this embodiment). The accent extraction module extracts preliminary accent features based on the input, and uses the acoustic features extracted by the Conformer module of the same layer for transformation, projection and splicing correction to serve as the input of the next layer of accent extraction module. The first layer of accent extraction module takes speech features as input, and the last layer of accent extraction module outputs the final accent features.
[0055] The fusion module is built based on the cross-attention mechanism, which includes two layers of cross-attention mechanisms. In the first layer of cross-attention, acoustic features are used as keys and accent features are used as queries to obtain intermediate results. In the second layer of cross-attention, the intermediate results are used as queries, and the acoustic features and accent features that have undergone shallow cross-attention are used as keys for fusion, and the fused feature vector is output.
[0056] S4, using the decoding module to decode the fused feature vector and output the speech recognition result of the dialect.
[0057] The decoding module includes a decoder and a vocabulary, wherein the decoder decodes the fused feature vector based on CTC loss and beam search mechanism, and constructs the relationship between phonemes and text through phoneme-level features; the vocabulary is established based on a dialect-related corpus to accurately include the unique text of the corresponding dialect and construct a vocabulary distribution adapted to the dialect.
[0058] This embodiment uses Ningbo dialect as an example to verify the accuracy of the above method in identifying dialects in a few regions. According to the characteristics of Ningbo dialect, some examples of vocabulary constructed in this embodiment are shown in Table 1 below:
[0059] Table 1 Some examples of vocabulary
[0060] <unk> < / unk> 0 Sun 4 little girl 19 <pad> < / pad> 1 moon 5 Old man 20 <s> < / s> 2 Thunder 6 Granny 21 3 Dragon Flash 7 Doctor 22 electronic 11 rainbow 12 Kitchen Master 23 Windy 13 people 14 Dad 24 slim Shady 25 Grandpa 26 Grandma 27 Grandpa 28 A Niang 29 Grandfather 30 grandmother 31 Brother 32 Sister 33 A-mei 34 uncle 35 aunt 36 Uncle 37 aunt 38 son 39 The Bride 40 Daughter 41 son in law 42 grandson 43 nephew 44 nephew 45
[0061] Based on the above vocabulary, the recognition accuracy of the method of the present invention and the existing methods (Conformer-ASR, zipconformer, etc.) are compared. The comparison results are shown in Table 2 below:
[0062] Table 2 Experimental results
[0063] Model Character Error Rate CER (%) Conformer-ASR 18.2 ZIPconformer 17.8 LAS-Conformer 19.5 CVAE-TT 16.2 The present invention 13.9
[0064] It can be seen that the present invention achieves the lowest character error rate (CER) on Ningbo dialect data, and effectively realizes the ASR task for Ningbo dialect.
[0065] This embodiment also provides a dialect recognition device integrating accent features, including:
[0066] An input module, used to obtain the dialect speech input to be recognized;
[0067] A preprocessing module is used to perform speech preprocessing on the input speech to obtain speech features;
[0068] A feature extraction and fusion module is used to fuse acoustic and intonation features based on speech features using an acoustic intonation feature fusion module to obtain a fused feature vector;
[0069] The output module is used to decode the fused feature vector using the decoding module and output the speech recognition result of the dialect.
[0070] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.
[0071] In one embodiment, the electronic device includes a computing unit that can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) or a computer program loaded from a storage unit into a random access memory (RAM). In the RAM, various programs and data required for the operation of the device can also be stored. The computing unit, ROM, and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.
[0072] Many components in an electronic device are connected to the I / O interface, including: input units, such as keyboards, mice, etc.; output units, such as various types of displays, speakers, etc.; storage units, such as disks, optical disks, etc.; and communication units, such as network cards, modems, wireless communication transceivers, etc. The communication unit allows the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunication networks.
[0073] The computing unit may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of computing units include, but are not limited to, central processing units (CPUs), graphics processing units (GPUs), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit performs the various methods and processes described above, such as methods S1 to S4. For example, in some embodiments, methods S1 to S4 may be implemented as computer software programs that are tangibly contained in machine-readable media, such as storage units. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via a ROM and / or a communication unit. When the computer program is loaded into RAM and executed by the computing unit, one or more steps of methods S1 to S4 described above may be performed. Alternatively, in other embodiments, the computing unit may be configured to perform methods S1 to S4 in any other appropriate manner (e.g., by means of firmware).
[0074] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0075] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0076] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.
[0077] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning, or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.
Claims
1. A dialect recognition method integrating intonation features, characterized in that: The following steps are involved: Get the dialect voice input to be recognized; Perform speech preprocessing on the input speech to obtain speech features; Based on speech features, the acoustic and intonation feature fusion module is used to fuse acoustic and intonation features to obtain a fused feature vector. The decoding module is used to decode the fused feature vector and output the speech recognition result of the dialect.
2. The dialect recognition method integrating intonation features according to claim 1, characterized in that: The speech preprocessing comprises the following steps: Using the trained speech pre-training network, extract the speech pre-training features of the input speech; Extract fbank features of input speech; The speech pre-training features and fbank features are initially fused to form input features; The input features are framed, windowed, and convolved to obtain speech features.
3. The dialect recognition method integrating intonation features according to claim 1, characterized in that: The acoustic intonation feature fusion module includes a primary acoustic encoder composed of multiple layers of Conformer modules, a hierarchical intonation extraction network and a fusion module, wherein the primary acoustic encoder uses the speech features obtained by preprocessing as input to extract acoustic features, the hierarchical intonation extraction network uses the speech features obtained by preprocessing and the acoustic features output by the primary acoustic encoder to extract intonation features, and the fusion module fuses the acoustic features output by the primary acoustic encoder and the intonation features output by the hierarchical intonation extraction network.
4. The dialect recognition method integrating intonation features according to claim 1, characterized in that: The primary acoustic encoder includes a first feature extraction network, an intermediate downsampling layer, and a second feature extraction network connected in sequence, wherein the first feature extraction network and the second feature extraction network are both composed of a multi-layer improved Conformer module, and the improved Conformer module sets a sliding convolution unit between the GLU activation layer and the BN layer of the basic Conformer network convolution module, so that the features focus on the features of other time steps through a dynamic window.
5. The dialect recognition method integrating intonation features according to claim 4, characterized in that: The hierarchical intonation extraction network extracts the intonation features from the input features in a hierarchical form, and performs micro-correction in combination with the acoustic features output by the primary acoustic encoder. Specifically, the hierarchical intonation extraction network is composed of a plurality of intonation extraction modules, and the intonation extraction modules have the same number of layers as the Conformer modules. The intonation extraction modules extract preliminary intonation features based on the input, and use the acoustic features extracted by the Conformer modules of the same layer for transformation, projection, and splicing correction, which are used as the input of the intonation extraction modules of the next layer. The intonation extraction modules of the first layer take the speech features as input, and the intonation extraction modules of the last layer output the final intonation features.
6. The dialect recognition method integrating intonation features according to claim 5, characterized in that: The fusion module is constructed based on the cross-attention mechanism, including two layers of cross-attention mechanisms. In the first layer of cross-attention, acoustic features are used as key values and accent features are used as queries to obtain intermediate results. In the second layer of cross-attention, the intermediate results are used as queries, and the acoustic features and accent features that have undergone shallow cross-attention are used as key values for fusion, and a fused feature vector is output.
7. The dialect recognition method integrating intonation features according to claim 1, characterized in that: The decoding module includes a decoder and a vocabulary. The decoder decodes the fused feature vector based on CTC loss and beam search mechanism, and constructs the relationship between phonemes and text through phoneme-level features. The vocabulary is established based on a dialect-related corpus to accurately include the unique text of the corresponding dialect and construct a vocabulary distribution adapted to the dialect.
8. A dialect recognition device integrating intonation features, characterized in that: include: An input module, used to obtain the dialect speech input to be recognized; A preprocessing module is used to perform speech preprocessing on the input speech to obtain speech features; A feature extraction and fusion module is used to fuse acoustic and intonation features based on speech features using an acoustic intonation feature fusion module to obtain a fused feature vector; The output module is used to decode the fused feature vector using the decoding module and output the speech recognition result of the dialect.
9. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.