Speech recognition method, device and equipment, storage medium and vehicle

Abstract

The invention provides a voice recognition method, device and equipment, a storage medium and a vehicle, and relates to the technical field of voice recognition. The method comprises the following steps: firstly, acquiring to-be-recognized voice information; performing dialect classification identification on the voice information to obtain dialect classification information corresponding to the voice information; performing voice feature recognition on the voice information to obtain voice feature information corresponding to the voice information; then, in a character set corresponding to the dialect classification information, mandarin information corresponding to the voice feature information is matched, the recognition result of the voice information is obtained, and the character set comprises mandarin information corresponding to dialects corresponding to the dialect classification information under different voice feature information. By adopting the technical scheme of the invention, accurate recognition of dialect voice can be realized.

Description

Technical Field

[0001] The present application relates to the field of speech recognition technology, and in particular to a speech recognition method, device, equipment, storage medium and vehicle. Background Art

[0002] The application of voice recognition technology in life is already very extensive, involving in the fields of car navigation, smart home, daily office and so on, bringing a lot of convenience to people's lives.

[0003] In order to improve the usability of human-computer interaction functions in various fields, when implementing speech recognition technology, considering that the user's speech signal may not be standard Mandarin, that is, the user speaks a dialect, a dialect recognition model is used to recognize the user's speech signal. However, due to the diversity of dialects, the large amount of corpus required for training the dialect recognition model is difficult to meet. Therefore, the final dialect recognition model is often unable to stably and accurately recognize the dialect in each sentence of the speech signal, and thus cannot meet the actual usage needs of users. Summary of the invention

[0004] In view of this, the present application provides a speech recognition method, device, equipment, storage medium and vehicle, the main purpose of which is to solve the technical problem in the prior art that user dialect voice information cannot be accurately recognized.

[0005] To achieve the above objectives, the first embodiment of the present application provides a speech recognition method, the method comprising:

[0006] Acquire voice information to be recognized;

[0007] Performing dialect classification and recognition on the voice information to obtain dialect classification information corresponding to the voice information; and,

[0008] Performing voice feature recognition on the voice information to obtain voice feature information corresponding to the voice information;

[0009] In the character set corresponding to the dialect classification information, the Mandarin information corresponding to the voice feature information is matched to obtain the recognition result of the voice information, wherein the character set includes the Mandarin information corresponding to the dialect corresponding to the dialect classification information under different voice feature information.

[0010] Optionally, performing dialect classification and recognition on the voice information to obtain dialect classification information corresponding to the voice information includes:

[0011] Performing pooling processing on the voice information to obtain aggregated information;

[0012] Performing a fully connected layer transformation on the aggregate information to obtain comprehensive features of the aggregate information;

[0013] Based on the comprehensive features, the dialect classification information corresponding to the voice information is identified.

[0014] Optionally, the performing pooling processing on the voice information to obtain aggregated information includes:

[0015] Performing preset information length processing on the voice information to obtain a plurality of block-based voice information;

[0016] Transforming the plurality of block-based voice information respectively using a parameter matrix to obtain aggregated information of each block-based voice information;

[0017] The step of performing a fully connected layer transformation on the aggregate information to obtain comprehensive features of the aggregate information includes:

[0018] The aggregate information of each block of voice information is respectively transformed through a fully connected layer to extract feature information, thereby extracting each segment of feature information corresponding to the aggregate information of each block of voice information;

[0019] Perform feature connection on each piece of feature information to determine the comprehensive feature.

[0020] Optionally, the identifying, based on the comprehensive features, dialect classification information corresponding to the voice information includes:

[0021] A weighted analysis is performed based on the dialect recognition results of the multiple block-based speech information contained in each piece of feature information and the corresponding excitation values, so as to determine that the dialect classification information is the target dialect.

[0022] Optionally, before matching Mandarin information corresponding to the voice feature information in a character set corresponding to the dialect classification information to obtain a recognition result of the voice information, the method further includes:

[0023] Acquire a dialect character set corresponding to the target dialect;

[0024] The step of matching the Mandarin information corresponding to the voice feature information in the character set corresponding to the dialect classification information to obtain the recognition result of the voice information includes:

[0025] In the dialect character set, Mandarin information corresponding to the voice feature information is matched to obtain a recognition result of the voice information.

[0026] Optionally, the transforming the plurality of block-based voice information respectively using a parameter matrix to obtain aggregate information of each block-based voice information includes:

[0027] Transforming the plurality of block-based speech information respectively using a target parameter matrix to generate a target result matrix;

[0028] Determine the attention weight matrix of the target result matrix at each preset time point when the target result matrix is ​​in the normalized index layer;

[0029] Obtaining a product value between the attention weight matrix and the matrix information of the block-based voice information;

[0030] The weighted sum of the product values ​​in the preset dimensions is calculated as the aggregation information of each block of voice information.

[0031] Optionally, before performing dialect classification and recognition on the voice information, the method further includes:

[0032] The speech information to be recognized is calculated through the conformer model to obtain speech signal data;

[0033] The performing dialect classification and recognition on the voice information to obtain dialect classification information corresponding to the voice information includes:

[0034] Performing dialect classification and recognition on the voice signal data to obtain dialect classification information corresponding to the voice information;

[0035] The performing voice feature recognition on the voice information to obtain voice feature information corresponding to the voice information includes:

[0036] Perform voice feature recognition on the voice signal data to obtain voice feature information corresponding to the voice information.

[0037] Optionally, before matching Mandarin information corresponding to the voice feature information in a character set corresponding to the dialect classification information to obtain a recognition result of the voice information, the method further includes:

[0038] Performing character recognition on the voice feature information according to pre-stored special characters;

[0039] If the voice feature information includes the special character, confirming a special character set corresponding to the special character;

[0040] The step of matching the Mandarin information corresponding to the voice feature information in the character set corresponding to the dialect classification information to obtain the recognition result of the voice information includes:

[0041] In the special character set corresponding to the dialect classification information, Mandarin information corresponding to the voice feature information is matched to obtain a recognition result of the voice information.

[0042] In a second aspect of the present application, an embodiment provides a speech recognition device, the device comprising:

[0043] An acquisition module, used for acquiring voice information to be recognized;

[0044] A recognition module, configured to perform dialect classification recognition on the voice information to obtain dialect classification information corresponding to the voice information; and perform voice feature recognition on the voice information to obtain voice feature information corresponding to the voice information;

[0045] A matching module is used to match the Mandarin information corresponding to the voice feature information in the character set corresponding to the dialect classification information to obtain the recognition result of the voice information, wherein the character set includes the Mandarin information corresponding to the dialect corresponding to the dialect classification information under different voice feature information.

[0046] In a third aspect of the present application, an electronic device is provided, including:

[0047] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any one of the methods disclosed in the first aspect.

[0048] In a fourth aspect of the present application, an embodiment provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0049] In a fifth aspect embodiment of the present application, a vehicle is provided, in which the speech recognition device as described in the second aspect or the electronic device as described in the third aspect is mounted.

[0050] In summary, according to the technical solution disclosed in the present application, by performing dialect classification and speech feature recognition on the speech information to be recognized, the dialect classification information and speech feature information corresponding to the speech information are obtained, and the relevant information of the speech information is collected from two perspectives at the same time. After confirming the dialect classification corresponding to the speech information, the corresponding relationship between the speech feature information and the Mandarin information is established through the character set corresponding to the dialect classification, that is, in the character set corresponding to the dialect classification information, the Mandarin information corresponding to the speech feature information is matched to obtain the recognition result of the speech information, wherein the character set includes the Mandarin information corresponding to the dialect classification information under different speech feature information. Compared with the current method of performing dialect speech recognition based on a dialect recognition model, the use of the technical solution of the present application can realize accurate recognition of dialect speech through two aspects, realize accurate recognition of the dialect in the user's speech information, and meet the actual use needs of the user's dialect speech recognition.

[0051] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0053] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0054] Figure 1 A flow chart of a method for speech recognition provided by an embodiment of the present application is shown;

[0055] Figure 2 A model structure diagram of speech recognition provided by an embodiment of the present application is shown;

[0056] Figure 3 A structural diagram of a speech recognition device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0057] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the scheme of the present application will be further described below. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0058] In order to solve the technical problem that the user's dialect voice information cannot be accurately recognized in the prior art, this application provides the following embodiments:

[0059] The first aspect of this embodiment provides a speech recognition method, such as Figure 1 FIG. 1 is a flowchart of the method of this embodiment, and the method of this embodiment may specifically include the following steps:

[0060] Step 101: Acquire voice information to be recognized.

[0061] For example, a microphone device can be used to collect voice information sent by a user for recognition, or voice information that needs to be recognized is collected from other systems.

[0062] The execution subject of this embodiment may be a client device or a server device, etc. The server device may be a local server device or a cloud server device, etc. There is no limitation on the execution subject of this embodiment.

[0063] Step 102, performing dialect classification and recognition on the voice information to obtain dialect classification information corresponding to the voice information; and performing voice feature recognition on the voice information to obtain voice feature information corresponding to the voice information.

[0064] In the content of this embodiment, after acquiring the voice information, the voice information is respectively subjected to dialect classification recognition and voice feature recognition, so as to obtain the dialect classification information and voice feature information corresponding to the voice information. The content of this embodiment is to simultaneously perform dialect classification recognition and voice feature recognition on the voice information to confirm the dialect type to which the voice information belongs, such as Shandong dialect, Henan dialect, Sichuan dialect, etc.

[0065] It should be noted that, when performing dialect classification and recognition, this embodiment does not specifically recognize the Mandarin content corresponding to the dialect, but rather recognizes which dialect type the voice information belongs to. The corresponding recognition model can be obtained in advance using corpora of different dialects for training, and then the recognition model is used to perform dialect classification recognition on the voice information to be recognized.

[0066] In addition to identifying the dialect type of the voice information, this embodiment also needs to identify the voice features of the voice information, which may include voice, intonation, speaking speed, pronunciation, etc. From the perspective of voice features, each dialect has its own special voice features.

[0067] Step 103, matching the Mandarin information corresponding to the voice feature information in the character set corresponding to the dialect classification information to obtain a recognition result of the voice information, wherein the character set includes Mandarin information corresponding to the dialect corresponding to the dialect classification information under different voice feature information.

[0068] After confirming the dialect classification corresponding to the voice information, this embodiment confirms the character set corresponding to the dialect classification. The character set includes Mandarin information corresponding to the dialect corresponding to the dialect classification information under different voice feature information. Therefore, through the character set, a correspondence is established between the voice feature information and the Mandarin information to achieve the acquisition of voice information.

[0069] For example, the voice information A spoken by the user is identified as the Henan dialect through voice recognition classified by dialect, and the voice feature information B of the voice information A is also identified; and for the Henan dialect, there will be a corresponding character set, which includes the Mandarin information corresponding to the Henan dialect under different voice feature information. By searching in the character set, the Mandarin information C corresponding to the voice feature information B is found, and then the voice recognition result of the voice information A can be determined based on the content of the Mandarin information C, which can be subsequently converted into corresponding text information and displayed on a display device (such as a display screen, etc.).

[0070] In the speech recognition method disclosed in the above embodiment, dialect classification and speech feature recognition are performed on the speech information to be recognized, so as to obtain dialect classification information and speech feature information corresponding to the speech information, and at the same time, relevant information of the speech information is collected from two perspectives. After confirming the dialect classification contained in the speech information, a correspondence between the speech feature information and the Mandarin information is established through the character set corresponding to the dialect classification, thereby realizing accurate recognition of dialect speech and meeting the actual usage needs of users for dialect speech recognition.

[0071] In a possible embodiment, performing dialect classification and recognition on voice information to obtain dialect classification information corresponding to the voice information includes:

[0072] The speech information is pooled to obtain aggregated information; the aggregated information is transformed by a fully connected layer to obtain comprehensive features of the aggregated information; based on the comprehensive features, the dialect classification information corresponding to the speech information is identified.

[0073] In the content of this embodiment, the process of dialect classification and recognition is further explained. Specifically, the process of dialect recognition is mainly divided into: pooling processing and fully connected layer transformation. The content of this embodiment describes the above process in detail. First, the voice information is pooled, and then the aggregated information generated by the pooling process is transformed by the fully connected layer to obtain the comprehensive features of the aggregated information, so as to obtain the dialect classification information based on the comprehensive features. Among them, the pooling operation divides the feature map into several sub-regions (generally rectangular), and statistically summarizes each sub-region to obtain aggregated information. The fully connected layer can complete the further fusion of features, so that the global features of the voice information finally obtained can realize the judgment of the dialect category.

[0074] In the content of this embodiment, the specific steps for obtaining dialect classification information are explained in detail, that is, through pooling and fully connected layer changes, the features are recognized and aggregated in sequence to generate comprehensive features, and further based on the judgment of the obtained comprehensive features, the dialect classification information corresponding to the speech information is judged.

[0075] In a possible embodiment, the voice information is pooled to obtain aggregate information, including: performing preset information length processing on the voice information to obtain multiple block voice information; transforming the multiple block voice information using a parameter matrix to obtain aggregate information of each block voice information; correspondingly, performing a full connection layer transformation on the aggregate information to obtain comprehensive features of the aggregate information, including: performing a full connection layer transformation on the aggregate information of each block voice information to extract feature information, extracting each segment of feature information corresponding to the aggregate information of each block voice information; and performing feature connection on each segment of feature information to determine the comprehensive features.

[0076] In the content of this embodiment, specific steps for pooling voice information are proposed. First, it is proposed to process the voice information with a preset information length, divide the voice information into multiple chunked voice information, and make the voice information subsequently processed with the preset information length. At the same time, further, in order to realize the streaming processing of voice information, the voice information with a preset information length is used for dialect classification processing, and the voice information of dialect classification processing and voice feature processing can be synchronized by controlling the length value of the voice information, so as to realize the orderliness of streaming processing. Exemplarily, the information with a length of T is divided into information with a preset length of 1 to obtain chunked voice information with a length of 1. After obtaining multiple chunked voice information corresponding to the voice information, the chunked voice information is transformed by a parameter matrix to obtain aggregated information. Exemplarily, for the chunked voice information, its three-dimensional matrix A[N,T,H] is obtained, where N is the voice information identifier, T is the voice information length, and H is the feature dimension value of the voice information.

[0077] After the block-based speech information is pooled to obtain aggregate information, the aggregate information of each block is further transformed through a fully connected layer to aggregate the feature information contained in the aggregate information of each block, and further generate aggregate information containing feature information. The dialect classification information is determined by identifying the dialect feature information corresponding to the aggregate information of each block.

[0078] By adopting the technical solution of this embodiment, it is possible to generate block-based voice information and standardize the length value of voice information by processing the voice information at a preset information length, and it is also possible to transform the block-based voice information through a learnable parameter matrix, perform feature statistics on each voice information, and realize the recognition of each voice information, so that the dialect classification result can be used together with the voice feature information to realize the recognition of voice information in a streaming manner, wherein streaming processing refers to a processing mode that can support real-time return of results during the processing of voice information, and can realize the processing of voice information in real time. Ensure the accuracy of subsequent recognition of voice information. At the same time, after each block-based voice information is pooled and transformed into a fully connected layer, the dialect recognition of voice information can be realized through the comprehensive features of each block.

[0079] In a possible embodiment, based on comprehensive features, dialect classification information corresponding to speech information is identified, including: performing weighted analysis based on dialect recognition results of multiple block-based speech information contained in each segment of feature information and their corresponding excitation values, and determining that the dialect classification information is the target dialect.

[0080] In this embodiment, after the voice information is divided into preset length values, the voice information is divided into block voice information, and feature information is extracted for each block of voice information. By identifying each piece of feature information corresponding to the aggregate information of each block of voice information, the overall feature recognition of the voice information is determined, thereby realizing the dialect type recognition of the entire voice information. Specifically, by identifying each piece of feature information corresponding to the voice information of each group of blocks, the voice types of each group of blocks are weighted together to confirm the dialect type of the overall voice information. Exemplarily, a voice message is divided into 10 blocks. In the voice information in the same block, it may be impossible to determine the specific type of dialect due to the similarity of the dialects. However, through the joint recognition and weighted calculation of multiple blocks, the dialect type that each block may belong to and the excitation value k corresponding to the dialect type are respectively confirmed. 1 , k 2 ,...k 10 , weighted calculation of dialect types, and weighted result values, so that the target dialect to which the entire voice information belongs can be finally determined.

[0081] By adopting the technical solution of this embodiment, the dialect recognition results of the block-based speech information are weighted respectively, and the feature information of all the block-based speech information that constitute the speech information is weighted, so as to finally confirm the dialect classification information of the speech information and finally realize the confirmation of the target dialect.

[0082] In a possible embodiment, before matching the Mandarin information corresponding to the voice feature information in the character set corresponding to the dialect classification information to obtain the recognition result of the voice information, the method also includes: acquiring a dialect character set corresponding to the target dialect; matching the Mandarin information corresponding to the voice feature information in the character set corresponding to the dialect classification information to obtain the recognition result of the voice information, including: matching the Mandarin information corresponding to the voice feature information in the dialect character set to obtain the recognition result of the voice information.

[0083] In the content of this embodiment, after determining the target dialect corresponding to the voice information, before matching the Mandarin information, it is also necessary to confirm the dialect character set corresponding to the Mandarin information, by obtaining the dialect character set corresponding to the target dialect, and matching with the Mandarin information in the selected dialect character set, thereby further achieving matching of the voice information.

[0084] The technical solution of this embodiment is adopted to disclose the dialect character set corresponding to the preset dialect to achieve matching with the Mandarin information. After confirming the target dialect, the dialect data of the dialect character set is used to achieve the corresponding relationship with the Mandarin information, thereby realizing the recognition of the voice information.

[0085] In a possible embodiment, a plurality of block-based speech information are transformed respectively using a parameter matrix to obtain aggregate information of each block-based speech information, including: a plurality of block-based speech information are transformed respectively using a target parameter matrix to generate a target result matrix; determining an attention weight matrix of the target result matrix at each preset time point when in a normalized index layer; obtaining a product value between the attention weight matrix and the matrix information of the block-based speech information; and calculating a weighted sum of the product values ​​in a preset dimension as aggregate information of each block-based speech information.

[0086] In the content of this embodiment, a method of performing parameter matrix transformation on the voice information of each block in the process of dialect classification of voice information is proposed, specifically: after the voice information is divided into blocks, the dialect category can be judged by analyzing the dialect features contained in each block. Therefore, in the process of pooling the voice information, the three-dimensional matrix A corresponding to the block-based voice information is transformed into the target parameter matrix B to generate the target result matrix, and the target result matrix is ​​normalized index calculated to obtain the attention weight matrix at each preset time point, and further calculate the corresponding results of the attention weight matrix and the original block-based voice information, so as to obtain the weighted sum value in the preset dimension value as the aggregation information of each block-based voice information, so as to realize the extraction of the dialect feature information corresponding to the voice information of the block.

[0087] For example, the specific content may be: the three-dimensional matrix A is transformed using the target transformation matrix B [H 1 ,A 1 ] is transformed, where the target transformation matrix is ​​a parameter learnable matrix, A 1 is the first parameter value, H 1 is the first bias value, generating a result matrix C[N 2 ,T 2 ], where N 2 is the second dimension value of the aggregated information, T 2 is the second length value corresponding to the second dimension value. After the result matrix C is input into the Softmax layer (normalized exponential function layer) for transformation, the attention weight represented by the transformed result matrix C at each preset time point is calculated, and the attention weight is converted through the weight matrix D:[N 3 ,T 3 ] to represent, where N 3 is the third dimension value of the probability distribution result, T 3 is the third length value corresponding to the third dimension value. At the same time, the probability distribution result is multiplied by the three-dimensional matrix A to obtain the product value E[N 4 ,H 4 ], where N 4is the fourth length value of the weighted sum, H 4 The fourth length value corresponding to the fourth length value is obtained, and the weighted sum value is obtained in the preset dimension, and the weighted sum value is used as the aggregation information of each block of voice information, and based on the aggregation information of each block of voice information, the dialect classification information corresponding to the voice information is identified.

[0088] By adopting the technical solution of this embodiment, in the specific step of further implementing pooling processing of voice information, and based on the subsequent fully connected layer transformation calculation, the feature information corresponding to the dialect voice information is extracted, and dialect classification can be further realized based on the extracted feature information.

[0089] In a possible embodiment, before performing dialect classification and recognition on speech information, the method also includes: calculating the speech information to be recognized through a conformer model to obtain speech signal data; performing dialect classification and recognition on the speech information to obtain dialect classification information corresponding to the speech information, including: performing dialect classification and recognition on the speech signal data to obtain dialect classification information corresponding to the speech information; performing speech feature recognition on the speech information to obtain speech feature information corresponding to the speech information, including: performing speech feature recognition on the speech signal data to obtain speech feature information corresponding to the speech information.

[0090] The content of this embodiment emphasizes that before implementing dialect classification and speech feature recognition of speech information, the speech information to be recognized is calculated through a conformer model to obtain speech signal data. The conformer model is a deep neural network architecture built using a self-attention mechanism that can capture dependencies in information. Using the conformer model to process speech information can better extract local features in speech information and better achieve the accuracy of dialect classification and speech feature recognition results.

[0091] In a possible embodiment, the dialect character set is generated by modeling Chinese characters.

[0092] The dialect character set in the embodiment of the present application generates a corresponding character set by modeling Chinese characters, so that after confirming the corresponding dialect type and accurately matching the voice feature information in the character set, the voice information can be directly represented in text form, which is convenient for users to confirm that the voice recognition is correct, and can also form a text representation that is easy for users to read in appropriate situations.

[0093] In a possible embodiment, before matching the Mandarin information corresponding to the voice feature information in the character set corresponding to the dialect classification information to obtain the recognition result of the voice information, the method also includes: performing character recognition on the voice feature information based on pre-stored special characters; if the voice feature information contains special characters, confirming the special character set corresponding to the special characters; matching the Mandarin information corresponding to the voice feature information in the character set corresponding to the dialect classification information to obtain the recognition result of the voice information, including: matching the Mandarin information corresponding to the voice feature information in the special character set corresponding to the dialect classification information to obtain the recognition result of the voice information.

[0094] In order to reduce the mutual interference between the modeling units used to store the character set, the content of this embodiment proposes to store the characters corresponding to some easy-to-error speech information separately as a special character set. In the process of establishing the special character set, the corpus information is decoded by the Mandarin model to obtain the decoding result, and the error probability of speech recognition that may occur shown by the decoding result is calculated, and the speech information with an error probability higher than the threshold is regarded as easy-to-error speech information, wherein the error probability can be the error probability of at least one character when recognizing all corpus, or it can be the error probability of at least one character when repeatedly recognizing the same corpus, and the characters with higher error probability are regarded as easy-to-error speech information, and the easy-to-error speech information is regarded as a special character. For example, the easy-to-error speech information may contain multiple characters with one pronunciation, multiple pronunciations of one character, and multiple meanings of one pronunciation. During the speech recognition process, the speech feature information is identified based on the special characters contained in the existing special character set to determine whether the current speech information contains special characters. If it is confirmed that special characters are contained, in the process of matching the speech feature information with the Mandarin information, it is necessary to further call the special character set corresponding to the special characters, match the Mandarin information corresponding to the speech feature information, and obtain the recognition result of the speech information.

[0095] Exemplarily, among different dialects, there may be characters with the same pronunciation that express different meanings. For example, the Chinese word "掉了" means "lost" in some dialects, while it means "fall from a height" in other dialects. Due to the diversity of its meanings, there is a relatively high probability of error in the process of Mandarin recognition. Another example is that for the sentence "爸爸你关起音乐" expressed in Sichuan dialect, its meaning in Mandarin is "爸爸你关上音乐". Because it is quite different from Mandarin, the recognition of the character "起" is prone to error. During the test phase of generating the special character set, when the error probability of recognizing the character "起" in all corpora exceeds 70%, for example, the character "起" can be regarded as a special character at this time. While the error probability of recognizing the character "你" in all corpora is 5%, which can be considered as a relatively low error probability and it is not regarded as a special character. Therefore, this embodiment proposes to establish a dedicated special character set. When matching Mandarin information, the speech feature information is compared with the special characters, and it is confirmed that the special characters in the speech information are directly matched with Mandarin through the special character set, thereby reducing the error rate of speech recognition.

[0096] This embodiment takes into account that the error-prone speech within the same dialect or between different dialects may interfere with the final recognition result during the character recognition process. Therefore, it is proposed that during the character set modeling process, that is, the characters corresponding to this kind of speech are modeled separately. When performing speech matching, the error-prone speech information can be directly matched through the special character set. On the basis of establishing the special character set, the number of output nodes of the multi-character set changes from the original K to K + M, where M refers to the number of special characters additionally selected from different dialects.

[0097] On the basis of implementing the method of the above embodiment, this application embodiment additionally discloses a model structure diagram for speech recognition, as Figure 2As shown, the module includes: a conformer model module, a dialect classification module, and a speech recognition module, wherein the conformer model module is used to implement the process of calculating the speech information to be recognized and generating speech signal data in the above embodiment, the dialect classification module is used to implement the dialect classification and recognition process in the above embodiment, and the speech recognition module is used to implement the speech feature recognition process in the above embodiment. The conformer model module processing process in the figure is represented by the conformer block module of the conformer model, and the conformer block module includes two feed forward modules, and a multi-headed self-attention module and a convolution module are sandwiched between them. The dialect classification module in the example diagram is generated by training with the cross entropy objective function, and the main layers include SAP layer (database layer), Full layer (fully connected layer) and CE loss (cross entropy function layer), wherein CE loss is used to describe the distance between two distributions. The speech recognition module is generated by training with the CTC objective function, and the main layers include Full layer (fully connected layer) and CTC loss (loss function for time series data classification problems). Among them, in the continuous training process of the dialect classification module and the speech recognition module, the accuracy of the speech recognition model is verified by judging the output results of CE loss and CTC loss. The speech signal to be recognized is input into the speech recognition model through Input Embedding. After being processed by the conformer model, the generated speech data signal is input into the dialect classification module and the speech recognition module respectively. The corresponding dialect character set matches the speech features with the Mandarin information, thereby realizing the generation of speech signal data.

[0098] According to the technical solution disclosed in this embodiment, dialect classification and speech feature recognition are performed on the speech information to be recognized, so as to obtain dialect classification information and speech feature information corresponding to the speech information, and at the same time, relevant information of the speech information is collected from two perspectives. After confirming the dialect classification contained in the speech information, a corresponding relationship between the speech feature information and the Mandarin information is established through the character set corresponding to the dialect classification, so as to realize accurate recognition of dialect speech.

[0099] Furthermore, this embodiment provides a speech recognition device, such as Figure 3 As shown, Figure 3 A structure diagram of a speech recognition device provided in an embodiment of the present application includes:

[0100] An acquisition module 31 is used to acquire voice information to be recognized;

[0101] The recognition module 32 is used to perform dialect classification recognition on the voice information to obtain dialect classification information corresponding to the voice information; and perform voice feature recognition on the voice information to obtain voice feature information corresponding to the voice information;

[0102] The matching module 33 is used to match the Mandarin information corresponding to the voice feature information in the character set corresponding to the dialect classification information to obtain the recognition result of the voice information, wherein the character set includes the Mandarin information corresponding to the dialect corresponding to the dialect classification information under different voice feature information.

[0103] In a possible embodiment, the recognition module 32 is specifically used to perform pooling processing on the voice information to obtain aggregate information; perform a fully connected layer transformation on the aggregate information to obtain comprehensive features of the aggregate information; and identify the dialect classification information corresponding to the voice information based on the comprehensive features.

[0104] In a possible embodiment, the recognition module 32 is specifically used to perform preset information length processing on the voice information to obtain multiple block voice information; respectively transform the multiple block voice information using a parameter matrix to obtain aggregate information of each block voice information;

[0105] The recognition module 32 is specifically used to perform feature information extraction on the aggregate information of each block of voice information through a fully connected layer transformation, and extract each segment of feature information corresponding to the aggregate information of each block of voice information;

[0106] Perform feature connection on each piece of feature information to determine the comprehensive feature.

[0107] In a possible embodiment, the recognition module 32 is specifically configured to perform a weighted analysis based on the dialect recognition results of the multiple block-based speech information contained in each segment of feature information and the corresponding excitation values, and determine that the dialect classification information is the target dialect.

[0108] In a possible embodiment, the matching module 33 is further configured to obtain a dialect character set corresponding to the target dialect;

[0109] The matching module 33 is specifically used to match the Mandarin information corresponding to the voice feature information in the dialect character set to obtain the recognition result of the voice information.

[0110] In a possible embodiment, the recognition module 32 is specifically used to transform the multiple block-based voice information using the target parameter matrix respectively to generate a target result matrix; determine the attention weight matrix of the target result matrix at each preset time point when it is in the normalized index layer; obtain the product value of the attention weight matrix and the block-based voice information; calculate the weighted sum of the product value in the preset dimension as the aggregation information of each block-based voice information.

[0111] In a possible embodiment, the recognition module 32 is further configured to calculate the speech information to be recognized through a conformer model to obtain speech signal data.

[0112] The recognition module 32 is used to perform dialect classification recognition on the voice signal data to obtain dialect classification information corresponding to the voice information; and perform voice feature recognition on the voice signal data to obtain voice feature information corresponding to the voice information.

[0113] In a possible embodiment, the dialect character set is generated by modeling Chinese characters.

[0114] In a possible embodiment, the matching module 33 is further configured to perform character recognition on the voice feature information according to pre-stored special characters; if the voice feature information includes the special character, confirm the special character set corresponding to the special character;

[0115] The matching module 33 is used to match the Mandarin information corresponding to the voice feature information in the special character set corresponding to the dialect classification information to obtain the recognition result of the voice information.

[0116] Based on this understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, USB flash drive, mobile hard disk, etc.), including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods of various implementation scenarios of the present application.

[0117] Based on the above Figure 1 The method shown, and Figure 3 In order to achieve the above-mentioned purpose, the embodiment of the present application further provides an electronic device that can be configured on the vehicle (such as a new energy vehicle) side, and the device includes at least one processor and a memory that is communicatively connected to the at least one processor; the memory is used to store instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor, and the processor is used to execute a computer program to implement the above-mentioned Figure 1 The method shown.

[0118] Optionally, the above-mentioned physical device may also include a user interface, a network interface, a camera, a radio frequency (RF) circuit, a sensor, an audio circuit, a WI-FI module, etc. The user interface may include a display, an input unit such as a keyboard, etc., and the optional user interface may also include a USB interface, a card reader interface, etc. The network interface may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface), etc.

[0119] Those skilled in the art will appreciate that the above-mentioned physical device structure provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or a combination of certain components, or different arrangements of components.

[0120] Based on the above Figure 1 The method shown, the embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method corresponding to any embodiment is implemented. The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the above-mentioned entity device, and supports the operation of information processing programs and other software and / or programs. The network communication module is used to realize communication between the components inside the storage medium, and communication with other hardware and software in the information processing entity device.

[0121] Based on the above electronic device, the embodiment of the present application further provides a vehicle, which may specifically include: Figure 3 The device shown in the figure or the electronic device as described above. The vehicle can be a new energy vehicle or a traditional vehicle.

[0122] Through the description of the above implementation methods, those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform, or by hardware. By applying the solution of this embodiment, compared with the current prior art, this embodiment performs dialect classification and speech feature recognition on the speech information to be recognized, thereby obtaining the dialect classification information and speech feature information corresponding to the speech information, and at the same time collects relevant information of the speech information from two perspectives, confirms the dialect classification contained in the speech information, and establishes the corresponding relationship between the speech feature information and the Mandarin information through the character set corresponding to the dialect classification, so as to realize accurate recognition of the dialect speech.

[0123] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprises" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0124] The above is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features applied for herein.

Claims

1. A speech recognition method, characterized in that: include: Acquire voice information to be recognized; Performing dialect classification and recognition on the voice information to obtain dialect classification information corresponding to the voice information; and, Performing voice feature recognition on the voice information to obtain voice feature information corresponding to the voice information; In the character set corresponding to the dialect classification information, the Mandarin information corresponding to the voice feature information is matched to obtain the recognition result of the voice information, wherein the character set includes the Mandarin information corresponding to the dialect corresponding to the dialect classification information under different voice feature information.

2. The method according to claim 1, characterized in that The performing dialect classification and recognition on the voice information to obtain dialect classification information corresponding to the voice information includes: Performing pooling processing on the voice information to obtain aggregated information; Performing a fully connected layer transformation on the aggregate information to obtain comprehensive features of the aggregate information; Based on the comprehensive features, the dialect classification information corresponding to the voice information is identified.

3. The method according to claim 2, characterized in that The performing pooling processing on the voice information to obtain aggregated information includes: Performing preset information length processing on the voice information to obtain a plurality of block-based voice information; Transforming the plurality of block-based voice information respectively using a parameter matrix to obtain aggregated information of each block-based voice information; The step of performing a fully connected layer transformation on the aggregate information to obtain comprehensive features of the aggregate information includes: The aggregate information of each block of voice information is respectively transformed through a fully connected layer to extract feature information, thereby extracting each segment of feature information corresponding to the aggregate information of each block of voice information; Perform feature connection on each piece of feature information to determine the comprehensive feature.

4. The method according to claim 3, characterized in that The identifying, based on the comprehensive features, the dialect classification information corresponding to the voice information includes: A weighted analysis is performed based on the dialect recognition results of the multiple block-based speech information contained in each piece of feature information and the corresponding excitation values, so as to determine that the dialect classification information is the target dialect.

5. The method according to claim 4, characterized in that Before matching the Mandarin information corresponding to the voice feature information in the character set corresponding to the dialect classification information and obtaining the recognition result of the voice information, the method further includes: Acquire a dialect character set corresponding to the target dialect as a character set corresponding to the dialect classification information; The step of matching the Mandarin information corresponding to the voice feature information in the character set corresponding to the dialect classification information to obtain the recognition result of the voice information includes: In the dialect character set, Mandarin information corresponding to the voice feature information is matched to obtain a recognition result of the voice information.

6. The method according to claim 3, characterized in that The step of transforming the plurality of block-based voice information using a parameter matrix to obtain aggregated information of each block-based voice information includes: Transforming the plurality of block-based speech information respectively using a target parameter matrix to generate a target result matrix; Determine the attention weight matrix of the target result matrix at each preset time point when the target result matrix is ​​in the normalized index layer; Obtaining a product value between the attention weight matrix and the matrix information of the block-based voice information; The weighted sum of the product values ​​in the preset dimensions is calculated as the aggregation information of each block of voice information.

7. The method according to claim 1, characterized in that Before performing dialect classification and recognition on the voice information, the method further includes: The speech information to be recognized is calculated through the conformer model to obtain speech signal data; The performing dialect classification and recognition on the voice information to obtain dialect classification information corresponding to the voice information includes: Performing dialect classification and recognition on the voice signal data to obtain dialect classification information corresponding to the voice information; The performing voice feature recognition on the voice information to obtain voice feature information corresponding to the voice information includes: Perform voice feature recognition on the voice signal data to obtain voice feature information corresponding to the voice information.

8. The method according to claim 1, characterized in that Before matching the Mandarin information corresponding to the voice feature information in the character set corresponding to the dialect classification information and obtaining the recognition result of the voice information, the method further includes: Performing character recognition on the voice feature information according to pre-stored special characters; If the voice feature information includes the special character, confirming a special character set corresponding to the special character; The step of matching the Mandarin information corresponding to the voice feature information in the character set corresponding to the dialect classification information to obtain the recognition result of the voice information includes: In the special character set corresponding to the dialect classification information, Mandarin information corresponding to the voice feature information is matched to obtain a recognition result of the voice information.

9. A speech recognition device, characterized in that: include: An acquisition module, used for acquiring voice information to be recognized; A recognition module, used to perform dialect classification recognition on the voice information to obtain dialect classification information corresponding to the voice information; and, performing voice feature recognition on the voice information to obtain voice feature information corresponding to the voice information; A matching module is used to match the Mandarin information corresponding to the voice feature information in the character set corresponding to the dialect classification information to obtain the recognition result of the voice information, wherein the character set includes the Mandarin information corresponding to the dialect corresponding to the dialect classification information under different voice feature information.

10. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

12. A vehicle, characterized in that: The vehicle is equipped with the device according to claim 9 or the electronic device according to claim 10.

Images