Artificial intelligence-based speech recognition method and device, computer device and medium

Abstract

The application is suitable for the medical technology field, and particularly relates to a speech recognition method and device based on artificial intelligence, computer equipment and medium. The application obtains a first speech enhancement matrix and a second speech enhancement matrix missing different semantic information by randomly shielding rows and columns of a mel spectrum matrix; calculates a metric sub-loss to perform self-supervised training on an encoder according to the first speech frame features and the second speech frame features extracted by the encoder; obtains speech fusion features and inputs the speech fusion features into a decoder to obtain mapped characters, combines preset characters to calculate a prediction loss to perform supervised training on a speech recognition model, weights and adds the prediction loss and the metric loss according to the number of zero characters and non-zero characters to obtain a target loss to train the encoder and the decoder, and combines the self-supervised training mode and the supervised training mode to improve the recognition accuracy of the speech recognition model, greatly improving the instantaneity, convenience and accuracy of information input in the medical technology field.

Description

Artificial intelligence-based speech recognition methods, devices, computer equipment, and media Technical Field

[0001] This invention relates to the field of medical technology, and in particular to a speech recognition method, device, computer equipment, and medium based on artificial intelligence. Background Technology

[0002] With the continuous development of medical informatization and the constant improvement of hospital information systems, the construction of integrated platforms has further promoted data sharing and data mining. Faced with numerous subsystems, medical staff need to complete a large amount of text input work. The immediacy, convenience, and accuracy of text input have a certain impact on the overall work efficiency, medical costs, and medical quality of the hospital. Text input methods based on speech recognition have greatly improved the immediacy and convenience of information input.

[0003] Among them, text input methods based on speech recognition rely on speech recognition models. Existing end-to-end speech recognition models require a large amount of supervised data during training. However, in actual medical applications, due to the difficulty in labeling speech data, it is difficult to obtain enough labeled samples for supervised training, resulting in low recognition accuracy of the trained speech recognition model and easy misrecognition.

[0004] Therefore, in the field of medical technology, improving the recognition accuracy of speech recognition models has become an urgent problem to be solved. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a speech recognition method, apparatus, computer device and medium based on artificial intelligence to solve the problem of low recognition accuracy of existing speech recognition models.

[0006] In a first aspect, embodiments of the present invention provide an artificial intelligence-based speech recognition method, the speech recognition method comprising:

[0007] Obtain N speech frame signals of the speech to be recognized, and the preset characters corresponding to each speech frame signal, and obtain the Mel spectrum matrix of each speech frame signal, where N is a positive integer;

[0008] Randomly occlude the rows of each of the Mel spectrum matrices to obtain a first speech enhancement matrix, and randomly occlude the columns of each of the Mel spectrum matrices to obtain a second speech enhancement matrix;

[0009] Each of the first speech enhancement matrices is input into the encoder for feature extraction to obtain the first speech frame features. Each of the second speech enhancement matrices is input into the encoder for feature extraction to obtain the second speech frame features. Based on the N first speech frame features, the N second speech frame features, and the preset metric loss function, N metric sub-losses are calculated.

[0010] The features of each first speech frame and the corresponding second speech frame are fused to obtain N speech fusion features. Each speech fusion feature is then input into the decoder for feature mapping to obtain N mapped characters corresponding to the N speech frame signals.

[0011] The prediction loss is calculated based on the N mapping characters, the N preset characters, and the preset prediction loss function. The metric loss is calculated by summing the N metric losses.

[0012] The predicted loss and the metric loss are weighted and summed to determine the target loss. Based on the target loss, the encoder and the decoder are trained to obtain a trained encoder and a trained decoder, which are then used as a trained speech recognition model for speech recognition.

[0013] Secondly, embodiments of the present invention provide an artificial intelligence-based speech recognition device, the speech recognition device comprising:

[0014] The voice information acquisition module is used to acquire N voice frame signals of the voice to be recognized, as well as the preset characters corresponding to each voice frame signal, and to acquire the Mel spectrum matrix of each voice frame signal, where N is a positive integer;

[0015] The speech enhancement module is used to randomly occlude the rows of each of the Mel spectrum matrices to obtain a first speech enhancement matrix, and to randomly occlude the columns of each of the Mel spectrum matrices to obtain a second speech enhancement matrix;

[0016] The feature extraction module is used to input each of the first speech enhancement matrices into the encoder for feature extraction to obtain the first speech frame features, input each of the second speech enhancement matrices into the encoder for feature extraction to obtain the second speech frame features, and calculate N metric sub-losses based on N first speech frame features, N second speech frame features and a preset metric loss function.

[0017] The feature fusion mapping module is used to fuse the features of each first speech frame and the corresponding features of the second speech frame to obtain N speech fusion features, and input each speech fusion feature into the decoder for feature mapping to obtain N mapped characters corresponding to the N speech frame signals;

[0018] The loss calculation module is used to calculate the prediction loss based on the N mapping characters, the N preset characters and the preset prediction loss function, and to add the N metric losses to calculate the metric loss.

[0019] The model training module is used to perform a weighted summation of the prediction loss and the metric loss, determine the weighted summation result as the target loss, and train the encoder and the decoder based on the target loss to obtain the trained encoder and the trained decoder, which are then used as the trained speech recognition model for speech recognition.

[0020] Thirdly, embodiments of the present invention provide a computer device, the computer device including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the speech recognition method as described in the first aspect.

[0021] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the speech recognition method as described in the first aspect.

[0022] The beneficial effects of this invention compared to existing technologies are as follows: By acquiring N speech frame signals of the speech to be recognized, and the preset characters corresponding to each speech frame signal, the Mel spectrum matrix of each speech frame signal is obtained. Random row and column occlusion is performed on each Mel spectrum matrix to obtain a first speech enhancement matrix and a second speech enhancement matrix that lack different semantic information. This effectively characterizes the intelligibility of each speech frame in the speech to be recognized, as well as the feature extraction accuracy of the speech to be recognized. Each first speech enhancement matrix and the second speech enhancement matrix are input into the encoder for feature extraction to obtain N first speech frame features and N second speech frame features. N metric losses are calculated by combining the preset metric loss function. The encoder is then trained under self-supervised conditions using the metric losses, which effectively improves the feature extraction accuracy of the encoder. Feature fusion is performed on each first speech frame feature and the corresponding second speech frame feature to obtain N speech fusion features. Speech fusion features are input into the decoder for feature mapping, resulting in N mapped characters corresponding to N speech frame signals. The prediction loss is calculated by combining N preset characters and a preset prediction loss function. The metric loss is calculated by summing the N metric losses. The prediction loss and metric loss are then weighted and summed to determine the target loss. Based on the target loss, the encoder and decoder are trained, resulting in a trained encoder and decoder, which serve as the trained speech recognition model. The predicted loss is used for supervised training of the speech recognition model. The prediction loss and metric loss are weighted based on the number of zero and non-zero characters. This combination of self-supervised and supervised training methods ensures model training stability while improving model fitting performance, thereby increasing the recognition accuracy of the speech recognition model and significantly improving the immediacy, convenience, and accuracy of information entry in the medical field. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 is a schematic diagram of an application environment for an artificial intelligence-based speech recognition method provided in Embodiment 1 of the present invention;

[0025] Figure 2 is a flowchart illustrating an artificial intelligence-based speech recognition method provided in Embodiment 1 of the present invention;

[0026] Figure 3 is a schematic diagram of the structure of an artificial intelligence-based speech recognition device provided in Embodiment 2 of the present invention;

[0027] Figure 4 is a schematic diagram of the structure of a computer device provided in Embodiment 3 of the present invention. Detailed Implementation

[0028] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the invention. However, those skilled in the art will understand that the invention can be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of the invention with unnecessary detail.

[0029] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0030] It should also be understood that the term “and / or” as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0031] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0032] Furthermore, in the description of this invention and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0033] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of the invention include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0034] The embodiments of this invention can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that utilize digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0035] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0036] It should be understood that the sequence number of each step in the following embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0037] To illustrate the technical solution of the present invention, specific embodiments are described below.

[0038] The first embodiment of this invention provides an artificial intelligence-based speech recognition method, which can be applied in the application environment shown in Figure 1, wherein the client and the server communicate with each other. The client includes, but is not limited to, PDAs, desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, cloud computing devices, personal digital assistants (PDAs), and other computer devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0039] Referring to Figure 2, which is a flowchart illustrating an artificial intelligence-based speech recognition method according to Embodiment 1 of the present invention, the speech recognition method can be applied to the client shown in Figure 1. The speech recognition method may include the following steps:

[0040] Step S201: Obtain N speech frame signals of the speech to be recognized, as well as the preset characters corresponding to each speech frame signal, and obtain the Mel spectrum matrix of each speech frame signal.

[0041] In speech recognition tasks, the speech to be recognized can be audio containing several frames. If feature extraction and analysis are directly performed on the speech to be recognized, and the speech recognition result is obtained based on the global speech information while ignoring the local audio information, the semantic representation ability of the speech to be recognized will be reduced, resulting in lower accuracy. In information entry scenarios in the medical field, the speech to be recognized can be obtained by medical staff reading the text to be entered into an integrated platform. This speech can then be stored in a corresponding database for centralized speech recognition. The speech recognition result is the corresponding text content to be entered, thus transforming the text entry task into a speech recognition task. This improves the immediacy and convenience of information entry, enhances the overall work efficiency and quality of medical care in hospitals, and reduces medical costs.

[0042] Therefore, in order to improve the semantic representation ability of the speech to be recognized, the speech to be recognized is first divided into N speech frame signals. By performing feature extraction and feature analysis on each speech frame signal, the local audio information of the speech to be recognized is obtained. Then, the speech information of the N speech frame signals is combined to obtain the global speech information of the speech to be recognized. Thus, the local audio information and global speech information of the speech to be recognized are combined to improve the semantic representation ability of the speech to be recognized and improve the accuracy of the speech recognition results. Here, N is a positive integer.

[0043] In this embodiment, the speech to be recognized can serve as the basis for optimizing the various model parameters in the speech recognition model. In order to improve the recognition accuracy of the speech to be recognized, while acquiring the N speech frame signals of the speech to be recognized, the preset characters corresponding to each speech frame signal are also acquired. These can serve as the basis for calculating the recognition accuracy of the speech recognition model, thereby improving the recognition accuracy of the speech recognition model.

[0044] In the field of speech processing, we need to convert speech signals into corresponding spectrograms and use the data on the spectrograms as features of the speech signal. Typically, spectrogram frequencies are linearly distributed, but the human ear's perception of frequency is logarithmic—that is, it is sensitive to changes in the low-frequency range but insensitive to changes in the high-frequency range. Therefore, linearly distributed spectrograms are not suitable for speech feature extraction tasks, while non-linearly distributed Mel spectrograms can effectively match the human ear's frequency perception and are widely used in speech feature extraction tasks.

[0045] Therefore, in this embodiment, the Mel spectrum matrix of each speech frame signal is obtained, and the Mel spectrum matrix is ​​used as the spectrogram of the speech frame signal to improve the reliability of the spectrogram.

[0046] The steps described above—obtaining N speech frame signals of the speech to be recognized, the preset characters corresponding to each speech frame signal, and obtaining the Mel spectrum matrix of each speech frame signal—divide the speech to be recognized into N speech frame signals to facilitate the extraction of local audio information of the speech to be recognized and to obtain a Mel spectrum matrix that effectively matches the human ear's perception of frequency, which can effectively improve the semantic representation ability of the speech to be recognized.

[0047] Step S202: Randomly occlude the rows of each Mel spectrum matrix to obtain the first speech enhancement matrix, and randomly occlude the columns of each Mel spectrum matrix to obtain the second speech enhancement matrix.

[0048] The primary goal of speech enhancement is to extract the purest possible original speech from noisy speech signals. However, since interference is usually random, extracting completely pure speech from noisy speech is nearly impossible. In this case, speech enhancement has two main objectives: first, to improve speech quality and eliminate background noise; and second, to increase speech intelligibility, which is a measure of the ability to understand speech under given conditions. These two objectives are often mutually exclusive.

[0049] Since it is difficult to extract the speech to be recognized as pure original speech, and it is impossible to obtain high-quality enhanced speech, this embodiment improves the intelligibility of the speech to be recognized through speech enhancement, thereby improving the semantic representation ability of the speech to be recognized.

[0050] Specifically, in this embodiment, the rows of each Mel spectrum matrix are randomly occluded to obtain a first speech enhancement matrix, and the columns of each Mel spectrum matrix are randomly occluded to obtain a second speech enhancement matrix. Compared with the Mel spectrum matrix, the first and second speech enhancement matrices obtained after partially occluding the Mel spectrum matrix lack some semantic information, and the missing semantic information in the first and second speech enhancement matrices is different.

[0051] Therefore, the first and second speech enhancement matrices, which lack different semantic information, can be used as the basis for self-supervised training of the speech recognition model. At the same time, by randomly occluding the row and column of the Mel spectrum matrix, a large number of self-supervised training samples can be obtained.

[0052] Specifically, after feature extraction is performed on the first speech enhancement matrix and the second speech enhancement matrix respectively, the similarity of the extracted features is calculated. Since the first speech enhancement matrix and the second speech enhancement matrix originate from the same Mel spectrum matrix, the first speech enhancement matrix and the second speech enhancement matrix, which lack different semantic information, can still extract features with high similarity. Therefore, the intelligibility of the speech to be recognized and the accuracy of feature extraction by the speech recognition model can be characterized by the similarity of the extracted features, thereby improving the semantic representation ability of each speech frame in the speech to be recognized.

[0053] Optionally, the rows of each Mel spectrum matrix are randomly occluded to obtain the first speech enhancement matrix, which includes:

[0054] The number of rows in each Mel spectrum matrix is ​​counted, and the ratio of the first preset value to the number of rows is used as the first sampling frequency of each Mel spectrum matrix;

[0055] Sampling is performed on all rows of each Mel spectrum matrix according to the first sampling frequency, and the sampled rows are occluded in the Mel spectrum matrix;

[0056] Repeat the process of sampling all rows of each Mel spectrum matrix at the first sampling frequency and occluding the sampled rows in the Mel spectrum matrix until the number of occluded rows in the Mel spectrum matrix is ​​consistent with the preset first occlusion number. Then, determine the occluded Mel spectrum matrix as the first speech enhancement matrix.

[0057] In this process, the number of rows to be occluded in the Mel spectrum matrix needs to be set according to the actual situation. This ensures that the first speech enhancement matrix, which is missing some semantic information after occlusion, can still effectively represent the semantic information in the Mel spectrum matrix after feature extraction. At the same time, it ensures that the Mel spectrum matrix can be occluded as much as possible to improve the accuracy of feature extraction in the speech recognition model. Therefore, a first preset value is set according to the actual situation to determine the first sampling frequency for row sampling of the Mel spectrum matrix, and a preset first occlusion number is set.

[0058] Therefore, the number of rows in each Mel spectrum matrix is ​​counted, and the ratio of the first preset value to the number of rows is used as the first sampling frequency of each Mel spectrum matrix. Then, all rows of the Mel spectrum matrix are sampled according to the first sampling frequency. The sampled rows in the Mel spectrum matrix are occluded. The above process of row sampling and row occlusion is repeated until the number of occluded rows in the Mel spectrum matrix is ​​consistent with the preset first occlusion number. The Mel spectrum matrix with occlusion completed is then determined as the first speech enhancement matrix.

[0059] Optionally, the columns of each Mel spectrum matrix are randomly occluded to obtain the second speech enhancement matrix, which includes:

[0060] The number of columns in each Mel spectrum matrix is ​​counted, and the ratio of the second preset value to the number of columns is used as the second sampling frequency of each Mel spectrum matrix;

[0061] Sampling is performed on all columns of each Mel spectrum matrix according to the second sampling frequency, and the sampled columns are masked in the Mel spectrum matrix;

[0062] Repeat the process of sampling all columns of each Mel spectrum matrix at the first sampling frequency and occluding the sampled columns in the Mel spectrum matrix until the number of occluded columns in the Mel spectrum matrix is ​​consistent with the preset second occlusion number. Then, determine the occluded Mel spectrum matrix as the second speech enhancement matrix.

[0063] Specifically, the number of columns to be occluded in the Mel spectrum matrix needs to be set according to the actual situation to ensure that the second speech enhancement matrix, which is missing some semantic information after occlusion, can still effectively represent the semantic information in the Mel spectrum matrix after feature extraction. At the same time, it is necessary to ensure that the Mel spectrum matrix can be occluded as much as possible to improve the accuracy of feature extraction in the speech recognition model. Therefore, a second preset value is set according to the actual situation to determine the second sampling frequency for column sampling of the Mel spectrum matrix, and a preset second occlusion number is set.

[0064] Therefore, the number of columns in each Mel spectrum matrix is ​​counted, and the ratio of the second preset value to the number of columns is used as the second sampling frequency of each Mel spectrum matrix. Then, all columns of the Mel spectrum matrix are sampled according to the second sampling frequency. The sampled columns in the Mel spectrum matrix are then occluded. The above column sampling and column occlusion process is repeated until the number of occluded columns in the Mel spectrum matrix is ​​consistent with the preset second occlusion number. The Mel spectrum matrix with completed occlusion is then determined as the second speech enhancement matrix.

[0065] The steps described above—randomly occluding the rows of each Mel spectrum matrix to obtain the first speech enhancement matrix, and randomly occluding the columns of each Mel spectrum matrix to obtain the second speech enhancement matrix—result in first and second speech enhancement matrices that are missing different semantic information. These matrices can be used to characterize the intelligibility of each speech frame in the speech to be recognized, as well as the accuracy of feature extraction by the speech recognition model. As a basis for self-supervised training of the speech recognition model, this effectively improves the semantic representation ability of each speech frame in the speech to be recognized.

[0066] Step S203: Input each first speech enhancement matrix into the encoder for feature extraction to obtain the first speech frame features. Input each second speech enhancement matrix into the encoder for feature extraction to obtain the second speech frame features. Calculate N sub-losses based on N first speech frame features, N second speech frame features, and a preset metric loss function.

[0067] The first speech enhancement matrix and the second speech enhancement matrix are based on the same Mel spectrum matrix but lack different semantic information. First, each of the first speech enhancement matrices is input into the encoder for feature extraction to obtain the first speech frame features. Then, each of the second speech enhancement matrices is input into the encoder for feature extraction to obtain the second speech frame features.

[0068] Since the first and second speech enhancement matrices originate from the same Mel spectrum matrix, when the encoder's feature extraction accuracy is high, the similarity between the first and second speech frame features is also high. Therefore, the first and second speech enhancement matrices can be used to characterize the accuracy of the encoder's feature extraction. The encoder can be self-supervised based on the similarity between the first and second speech frame features. Furthermore, by randomly occluding the Mel spectrum matrix in rows and columns, a large number of self-supervised training samples can be obtained, thereby effectively improving the encoder's feature extraction accuracy.

[0069] Specifically, for N first speech frame features and N second speech frame features corresponding to N Mel spectrum matrices, each first speech frame feature and the corresponding second speech frame feature are substituted into a preset metric loss function to calculate the corresponding N metric sub-losses.

[0070] For example, let X be the feature of the i-th (i = 1, 2, ..., N) first speech frame. 1i Let X be the feature of the i-th second speech frame. 2i The preset loss function is:

[0071]

[0072] In the formula, l 1i For the loss of the i-th metric, X 1i X is the feature of the i-th first speech frame. 2i This represents the feature of the i-th second speech frame.

[0073] The above steps involve inputting each first speech enhancement matrix into the encoder for feature extraction to obtain first speech frame features, inputting each second speech enhancement matrix into the encoder for feature extraction to obtain second speech frame features, and calculating N metric sub-losses based on N first speech frame features, N second speech frame features, and a preset metric loss function. By extracting features from the first and second speech frame features using the encoder, and calculating the similarity between the first and second speech frame features to perform self-supervised training of the encoder, the feature extraction accuracy of the encoder is effectively improved.

[0074] Step S204: Perform feature fusion on each first speech frame feature and the corresponding second speech frame feature to obtain N speech fusion features. Input each speech fusion feature into the decoder for feature mapping to obtain N mapped characters corresponding to the N speech frame signals.

[0075] The first speech frame features and the corresponding second speech frame features originate from the same Mel spectrum matrix. When performing a speech recognition task, the first speech frame features and the corresponding second speech frame features are fused to obtain speech fusion features. These speech fusion features can effectively represent the semantic information in each speech frame of the speech to be recognized.

[0076] The various speech fusion features are then input into the decoder for feature mapping, which yields the mapped characters corresponding to each speech frame signal. These mapped characters are the speech recognition results for the corresponding speech frame. Thus, the N mapped characters corresponding to N speech frame signals are the speech recognition results for the speech to be recognized.

[0077] Optionally, the first speech frame feature corresponds to the number of rows that are occluded, and the second speech frame feature corresponds to the number of columns that are occluded.

[0078] The features of each first speech frame and the corresponding second speech frame are fused to obtain speech fusion features, including:

[0079] Calculate the sum of the number of rows and columns that are obscured;

[0080] The number of occluded rows, the number of occluded columns, and the sum of their values ​​are normalized to obtain the first fusion weight of the first speech frame features and the second fusion weight of the second speech frame features.

[0081] The speech fusion features are obtained by weighted fusion of the first speech frame features, the first fusion weight, the corresponding second speech frame features, and the second fusion weight.

[0082] The first speech frame features and the second speech frame features are obtained by extracting features from the first speech enhancement matrix and the second speech enhancement matrix, respectively. The first speech enhancement matrix is ​​obtained by occluding several rows of the Mel spectrum matrix, and the second speech enhancement matrix is ​​obtained by occluding several columns of the Mel spectrum matrix. The more rows or columns of the Mel spectrum matrix are occluded, the more semantic information is missing from the corresponding speech enhancement matrix, and the smaller the corresponding fusion weight is when performing feature fusion.

[0083] Therefore, the first speech frame feature corresponds to the number of occluded rows, and the second speech frame feature corresponds to the number of occluded columns. First, the sum of the number of occluded rows and columns is calculated. Then, the maximum value of the number of occluded rows, columns, and the sum is normalized to obtain the normalized values ​​of the number of rows and columns. The difference between the normalized value of the number of rows and 1 is used as the first fusion weight of the first speech frame feature, and the difference between the normalized value of the number of columns and 1 is used as the second fusion weight of the second speech frame feature. The speech fusion feature is obtained by weighted fusion based on the first speech frame feature, the first fusion weight, the corresponding second speech frame feature, and the second fusion weight.

[0084] For example, let S1 be the number of rows that are obscured, S2 be the number of columns that are obscured, and S be the sum of the number of rows and columns that are obscured, i.e., S = S1 + S2. Then, we perform maximum value normalization on the number of rows S1, the number of columns S2, and the sum S to obtain the normalized values ​​for the number of rows and columns:

[0085]

[0086] In the formula, S′1 is the normalized value of the number of rows, S′2 is the normalized value of the number of columns, S1 is the number of rows that are obscured, S2 is the number of columns that are obscured, and S is the sum of the number of rows and columns that are obscured.

[0087] Then, the difference between the normalized value of the number of rows and 1 is calculated and used as the first fusion weight of the first speech frame feature, and the difference between the normalized value of the number of columns and 1 is calculated and used as the second fusion weight of the second speech frame feature. The first fusion weight and the second fusion weight are then:

[0088] α1 = 1 - S′1; α2 = 1 - S′2

[0089] In the formula, α1 is the first fusion weight, α2 is the second fusion weight, S′1 is the normalized value of the number of rows, and S′2 is the normalized value of the number of columns.

[0090] Finally, based on the feature X of the i-th first speech frame 1i The first fusion weight α1 and the corresponding second speech frame feature X 2i The second fusion weight α2 is used for weighted fusion to obtain the i-th speech fusion feature:

[0091] X i =α1X 1i +α2X 2i

[0092] In the formula, X i For the i-th speech fusion feature, X 1i X is the feature of the i-th first speech frame. 2i Let α1 be the first fusion weight and α2 be the second fusion weight for the i-th second speech frame feature.

[0093] The above steps involve fusing features of each first speech frame and the corresponding second speech frame to obtain N speech fusion features, inputting each speech fusion feature into the decoder for feature mapping to obtain N mapped characters corresponding to the N speech frame signals. By fusing features of multiple speech frame features extracted from multiple speech enhancement matrices, the obtained speech fusion features effectively characterize the semantic information in each speech frame of the speech to be recognized, thus improving the accuracy of speech recognition results.

[0094] Step S205: Calculate the prediction loss based on N mapped characters, N preset characters, and a preset prediction loss function; sum the N metric losses to calculate the metric loss.

[0095] In the process of model training, it is difficult to label speech data and obtain enough labeled samples for supervised training. Therefore, this embodiment performs speech enhancement on the Mel spectrum matrix of each speech frame signal, calculates the metric loss based on each first speech enhancement matrix and second speech enhancement matrix, and calculates the metric loss to perform self-supervised training on the speech recognition model. However, when only self-supervised training is used, the stability of the model is difficult to guarantee.

[0096] Therefore, this embodiment calculates the prediction loss based on N mapped characters, N preset characters, and a preset prediction loss function to perform supervised training on the speech recognition model. This combines self-supervised and supervised training methods, ensuring the stability of model training while improving the model's fitting effect, thereby improving the recognition accuracy of the speech recognition model.

[0097] Optionally, based on N mapped characters, N preset characters, and a preset prediction loss function, the prediction loss is calculated, including:

[0098] Pair N mapped characters with N preset characters to obtain N matching character pairs;

[0099] Calculate the prediction sub-loss for each matching character pair based on the mapped character, preset character, and preset prediction loss function in each matching character pair.

[0100] The prediction sub-losses of N matching character pairs are summed, and the sum is determined as the prediction loss.

[0101] The closer the mapped character is to the corresponding preset character, the higher the feature extraction accuracy of the encoder and the feature mapping accuracy of the decoder. Therefore, the N mapped characters and N preset characters are first paired to obtain N matching character pairs. Each matching character pair includes a mapped character and a corresponding preset character.

[0102] Then, based on the mapped character, preset character, and preset prediction loss function in each matched character pair, the prediction sub-loss of each matched character pair is calculated. Specifically, the similarity between the mapped character and the preset character in each matched character pair is calculated as the corresponding prediction sub-loss. Then, the prediction sub-losses of N matched character pairs are added together, and the sum is determined as the prediction loss, which is used to characterize the feature extraction accuracy of the encoder and the feature mapping accuracy of the decoder.

[0103] For example, to facilitate similarity calculation, the i-th mapped character is first converted into the i-th mapped character vector Y using word vector techniques. i The i-th preset character is converted into the i-th preset character vector Z using word vector technology. i Then, the i-th mapped character vector Y i and the i-th preset character vector Z i Substituting the values ​​into the preset prediction loss function to calculate the similarity, we obtain the loss of the i-th prediction sub-value:

[0104]

[0105] In the formula, l 2i For the loss of the i-th predictor, Y i Let Z be the i-th mapped character vector. i Let i be the i-th preset character vector.

[0106] Then, by summing the losses of the N predictors, we obtain the prediction loss as follows:

[0107]

[0108] In the formula, L2 is the prediction loss, N is the number of prediction sub-losses, and l 2i Let be the loss of the i-th predictor.

[0109] Simultaneously, by summing the losses of the N metrics, the metric loss is calculated as follows:

[0110]

[0111] In the formula, L1 is the metric loss, N is the number of metric sub-losses, and l 1i Let be the loss of the i-th metric.

[0112] The above steps, which calculate the prediction loss based on N mapped characters, N preset characters, and a preset prediction loss function, and sum the N metric losses to calculate the metric loss, are used to perform self-supervised training of the speech recognition model. At the same time, the prediction loss is calculated to perform supervised training of the speech recognition model. By combining self-supervised and supervised training methods, the model's fitting effect is improved while ensuring the stability of the model training, thereby improving the recognition accuracy of the speech recognition model.

[0113] Step S206: The prediction loss and the measurement loss are weighted and summed to determine the target loss. Based on the target loss, the encoder and decoder are trained to obtain the trained encoder and the trained decoder, which are then used as the trained speech recognition model for speech recognition.

[0114] Among them, the prediction loss is used to measure the supervised loss of the speech recognition model, and the metric loss is used to measure the unsupervised loss of the speech recognition model. Therefore, in order to improve the recognition accuracy of the speech recognition model, the prediction loss and the metric loss are first weighted and added together to obtain the target loss. Then, based on the target loss, the encoder and decoder are trained to obtain the trained encoder and the trained decoder, which are used as the trained speech recognition model for speech recognition.

[0115] Optionally, the preset characters include zero characters and non-zero characters; the predicted loss and the measurement loss are weighted and summed to determine the target loss, including:

[0116] Count the number of zero characters as the first character and the number of non-zero characters as the second character among N preset characters;

[0117] The number of the first character, the number of the second character, and the number of preset characters N are subjected to maximum normalization to obtain the normalized value of non-zero characters, which is used as the first weighted weight for measuring the loss, and the normalized value of zero characters is used as the second weighted weight for predicting the loss.

[0118] Based on the predicted loss, the second weighted weight, the measurement loss, and the first weighted weight, the predicted loss and the measurement loss are weighted and summed to determine the target loss.

[0119] Since the N speech frame signals are obtained by segmenting the speech to be recognized, and there are pauses between the speech and text in the speech to be recognized, each speech frame signal may correspond to a zero character or a non-zero character. Therefore, the preset characters include zero characters and non-zero characters.

[0120] When the preset character corresponding to the speech frame signal is zero, the difference between the first and second speech frame features obtained after speech enhancement and feature extraction is very small, resulting in a small metric loss. Therefore, the metric loss can only characterize the feature extraction accuracy of the encoder to a certain extent. Furthermore, the more zero characters there are among the N preset characters, the less the metric loss characterizes the encoder's feature extraction accuracy. Since the prediction loss measures the supervised loss of the speech recognition model, and the metric loss measures the unsupervised loss, when weighting the prediction and metric losses, the more zero characters there are among the N preset characters, the smaller the weight of the metric loss and the larger the weight of the prediction loss.

[0121] Therefore, we count the number of zero characters in the first character and the number of non-zero characters in the second character among N preset characters. Then, we perform maximum normalization on the number of first characters, the number of second characters, and the number of preset characters N. The first weight of the measurement loss is proportional to the number of non-zero characters in the second character, and the second weight of the prediction loss is proportional to the number of zero characters in the first character. Therefore, we use the normalized value of non-zero characters as the first weight of the measurement loss and the normalized value of zero characters as the second weight of the prediction loss. Then, we perform a weighted sum of the prediction loss, the second weight, the measurement loss, and the first weight, and determine the weighted sum as the target loss. This target loss can be used to characterize the feature extraction accuracy of the encoder and the feature mapping accuracy of the decoder.

[0122] Optionally, the encoder and decoder can be trained based on the target loss to obtain a trained encoder and a trained decoder, including:

[0123] Based on the target loss, the encoder and decoder are trained using gradient descent until the target loss converges, resulting in a trained encoder and decoder.

[0124] Since a larger target loss indicates lower feature extraction accuracy of the encoder and lower feature mapping accuracy of the decoder, and a smaller target loss indicates higher feature extraction accuracy of the encoder and higher feature mapping accuracy of the decoder, in order to improve the feature extraction accuracy of the encoder and the feature mapping accuracy of the decoder, the encoder and decoder are trained according to the gradient descent method based on the target loss until the target loss converges, thus obtaining a trained encoder and a trained decoder.

[0125] The above steps involve weighted summation of the prediction loss and metric loss to determine the target loss. Based on this target loss, the encoder and decoder are trained to obtain a trained encoder and decoder, which serve as the trained speech recognition model for speech recognition. Since the more zero characters among the N preset characters, the less the metric loss represents the accuracy of the encoder's feature extraction, the normalized values ​​of non-zero characters are used as the first weighting weight of the metric loss, and the normalized values ​​of zero characters are used as the second weighting weight of the prediction loss. The prediction loss and metric loss are weighted and summed to obtain the target loss, which is then used to train the encoder and decoder. This improves the accuracy of the target loss, thereby improving the training quality of the encoder and decoder.

[0126] This invention acquires N speech frame signals of the speech to be recognized, along with preset characters corresponding to each speech frame signal. It then obtains the Mel-spectrum matrix of each speech frame signal, performs random row and column occlusion on each Mel-spectrum matrix, and obtains a first speech enhancement matrix and a second speech enhancement matrix that lack different semantic information. This effectively characterizes the intelligibility of each speech frame in the speech to be recognized, as well as the accuracy of feature extraction. The first and second speech enhancement matrices are then input into an encoder for feature extraction, yielding N first speech frame features and N second speech frame features. These are combined with a preset metric loss function to calculate N metric sub-losses. The encoder is then trained using these metric sub-losses under self-supervised conditions, effectively improving the encoder's feature extraction accuracy. Finally, the first speech frame features and the corresponding second speech frame features are fused to obtain N fused speech features. These fused speech features are then input into... The signal is mapped to the decoder to obtain N mapped characters corresponding to N speech frames. The prediction loss is calculated by combining N preset characters and a preset prediction loss function. The metric loss is calculated by adding N metric losses. The prediction loss and metric loss are weighted and summed to determine the target loss. The encoder and decoder are trained based on the target loss to obtain the trained encoder and decoder. This trained speech recognition model is then used for speech recognition. The prediction loss is calculated to perform supervised training on the speech recognition model. The prediction loss and metric loss are weighted and summed according to the number of zero characters and non-zero characters. This combination of self-supervised and supervised training methods ensures the stability of model training while improving the model's fitting effect, thereby improving the recognition accuracy of the speech recognition model and greatly enhancing the immediacy, convenience, and accuracy of information entry in the medical field.

[0127] Corresponding to the speech recognition method in the above embodiments, Figure 3 shows a structural block diagram of the artificial intelligence-based speech recognition device provided in Embodiment 2 of the present invention. For ease of explanation, only the parts related to the embodiments of the present invention are shown.

[0128] Referring to Figure 3, the voice recognition device includes:

[0129] The voice information acquisition module 31 is used to acquire N voice frame signals of the voice to be recognized, as well as the preset characters corresponding to each voice frame signal, and to acquire the Mel spectrum matrix of each voice frame signal, where N is a positive integer.

[0130] The speech enhancement module 32 is used to randomly occlude the rows of each Mel spectrum matrix to obtain a first speech enhancement matrix, and to randomly occlude the columns of each Mel spectrum matrix to obtain a second speech enhancement matrix.

[0131] The feature extraction module 33 is used to input each first speech enhancement matrix into the encoder for feature extraction to obtain the first speech frame features, input each second speech enhancement matrix into the encoder for feature extraction to obtain the second speech frame features, and calculate N metric sub-losses based on N first speech frame features, N second speech frame features and a preset metric loss function.

[0132] The feature fusion mapping module 34 is used to fuse the features of each first speech frame and the corresponding second speech frame features to obtain N speech fusion features, and input each speech fusion feature into the decoder for feature mapping to obtain N mapped characters corresponding to the N speech frame signals.

[0133] The loss calculation module 35 is used to calculate the prediction loss based on N mapped characters, N preset characters and a preset prediction loss function, and to add the N metric losses to calculate the metric loss.

[0134] The model training module 36 is used to perform a weighted summation of the prediction loss and the measurement loss, and determine the weighted summation result as the target loss. Based on the target loss, the encoder and decoder are trained to obtain the trained encoder and the trained decoder, which are then used as the trained speech recognition model for speech recognition.

[0135] Optionally, the aforementioned voice enhancement module 32 includes:

[0136] The first sampling frequency calculation submodule is used to count the number of rows in each Mel spectrum matrix and use the ratio of the first preset value to the number of rows as the first sampling frequency of each Mel spectrum matrix.

[0137] The first occlusion submodule is used to sample all rows of each Mel spectrum matrix according to the first sampling frequency, and to occlude the sampled rows in the Mel spectrum matrix.

[0138] The first speech enhancement submodule is used to repeatedly sample all rows of each Mel spectrum matrix according to the first sampling frequency, and to mask the sampled rows in the Mel spectrum matrix until the number of masked rows in the Mel spectrum matrix is ​​consistent with the preset first masking number, and to determine the masked Mel spectrum matrix as the first speech enhancement matrix.

[0139] Optionally, the aforementioned voice enhancement module 32 includes:

[0140] The second sampling frequency calculation submodule is used to count the number of columns in each Mel spectrum matrix and use the ratio of the second preset value to the number of columns as the second sampling frequency of each Mel spectrum matrix.

[0141] The second occlusion submodule is used to sample all columns of each Mel spectrum matrix according to the second sampling frequency, and to occlude the sampled columns in the Mel spectrum matrix.

[0142] The second speech enhancement submodule is used to repeatedly sample all columns of each Mel spectrum matrix according to the first sampling frequency, and mask the sampled columns in the Mel spectrum matrix until the number of masked columns in the Mel spectrum matrix is ​​consistent with the preset second masking number, and then determine the masked Mel spectrum matrix as the second speech enhancement matrix.

[0143] Optionally, the first speech frame feature corresponds to the number of occluded rows, and the second speech frame feature corresponds to the number of occluded columns. The feature fusion mapping module 34 includes:

[0144] The occlusion count and calculation submodule is used to calculate the sum of the number of occluded rows and the number of occluded columns;

[0145] The fusion weight calculation submodule is used to normalize the maximum value of the number of occluded rows, the number of occluded columns and their sum, to obtain the first fusion weight of the first speech frame features and the second fusion weight of the second speech frame features.

[0146] The feature fusion submodule is used to perform weighted fusion based on the features of the first speech frame, the first fusion weight, the corresponding features of the second speech frame, and the second fusion weight to obtain speech fusion features.

[0147] Optionally, the loss calculation module 35 mentioned above includes:

[0148] The character matching submodule is used to perform pairing operations on N mapped characters and N preset characters to obtain N matching character pairs;

[0149] The prediction sub-loss calculation submodule is used to calculate the prediction sub-loss of each matching character pair based on the mapped character, preset character, and preset prediction loss function in each matching character pair.

[0150] The prediction loss calculation submodule is used to add the prediction sub-losses of N matching character pairs and determine the sum as the prediction loss.

[0151] Optionally, the preset characters include zero characters and non-zero characters, and the above model training module 36 includes:

[0152] The character count submodule is used to count the number of zero-character first characters and the number of non-zero-character second characters among N preset characters;

[0153] The weighted weight calculation submodule is used to perform maximum normalization on the number of first characters, the number of second characters, and the number of preset characters N to obtain the normalized value of non-zero characters, which is used as the first weighted weight for measuring loss, and the normalized value of zero characters, which is used as the second weighted weight for predicting loss.

[0154] The target loss calculation submodule is used to perform a weighted sum of the predicted loss and the measured loss based on the predicted loss, the second weighted weight, the measured loss, and the first weighted weight, and determine the weighted sum as the target loss.

[0155] Optionally, the above model training module 36 includes:

[0156] The model training submodule is used to train the encoder and decoder using gradient descent based on the target loss until the target loss converges, resulting in a trained encoder and decoder.

[0157] It should be noted that the information interaction and execution process between the above modules are based on the same concept as the method embodiments of the present invention. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0158] Figure 4 is a schematic diagram of a computer device provided in Embodiment 3 of the present invention. As shown in Figure 4, the computer device of this embodiment includes: at least one processor (only one is shown in Figure 4), a memory, and a computer program stored in the memory and executable on at least one processor. When the processor executes the computer program, it implements the steps in any of the above-described speech recognition method embodiments.

[0159] The computer device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that Figure 4 is merely an example of a computer device and does not constitute a limitation thereof. The computer device may include more or fewer components than illustrated, or a combination of certain components, or different components, such as a network interface, a display screen, and input devices.

[0160] The processor referred to can be a CPU, but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0161] Memory includes readable storage media, internal memory, etc., wherein internal memory can be the RAM of a computer device, providing an environment for the operation of the operating system and computer-readable instructions stored in the readable storage media. The readable storage media can be the hard drive of a computer device, or in other embodiments, it can be an external storage device of the computer device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, memory can include both internal storage units and external storage devices of a computer device. Memory is used to store the operating system, applications, bootloader, data, and other programs, such as program code for computer programs. Memory can also be used to temporarily store data that has been output or will be output.

[0162] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the functions described above can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this invention. The specific working process of the units and modules in the above device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention can implement all or part of the processes in the methods of the above embodiments by instructing related hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the above method embodiments. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium can include at least: any entity or device capable of carrying computer program code, a recording medium, a computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0163] The present invention can implement all or part of the processes in the methods of the above embodiments, or it can be accomplished by a computer program product. When the computer program product is run on a computer device, the computer device executes the steps in the above method embodiments.

[0164] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0165] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for various specific applications, but such implementations should not be considered beyond the scope of this invention.

[0166] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / computer devices and methods can be implemented in other ways. For example, the apparatus / computer device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0167] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0168] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A speech recognition method based on artificial intelligence, characterized in that, The speech recognition method includes: acquiring N speech frame signals of the speech to be recognized, and preset characters corresponding to each speech frame signal; acquiring the Mel-spectrum matrix of each speech frame signal, where N is a positive integer; randomly occluding the rows of each Mel-spectrum matrix to obtain a first speech enhancement matrix; randomly occluding the columns of each Mel-spectrum matrix to obtain a second speech enhancement matrix; inputting each of the first speech enhancement matrices into an encoder for feature extraction to obtain first speech frame features; inputting each of the second speech enhancement matrices into an encoder for feature extraction to obtain second speech frame features; and calculating N metrics based on the N first speech frame features, the N second speech frame features, and a preset metric loss function. The loss is calculated as follows: Features of each first speech frame and corresponding features of the second speech frame are fused to obtain N fused speech features. These features are then input into the decoder for feature mapping to obtain N mapped characters corresponding to the N speech frame signals. A prediction loss is calculated based on the N mapped characters, N preset characters, and a preset prediction loss function. The N metric losses are summed to calculate the metric loss. The prediction loss and the metric loss are weighted and summed to determine the target loss. The encoder and decoder are trained based on the target loss to obtain a trained encoder and a trained decoder, which are then used as a trained speech recognition model for speech recognition.

2. The speech recognition method according to claim 1, characterized in that, The step of randomly occluding the rows of each of the Mel-spectrum matrices to obtain the first speech enhancement matrix includes: counting the number of rows in each of the Mel-spectrum matrices, and using the ratio of a first preset value to the number of rows as the first sampling frequency of each of the Mel-spectrum matrices; sampling all rows of each of the Mel-spectrum matrices according to the first sampling frequency, and occluding the sampled rows in the Mel-spectrum matrices; repeating the steps of sampling all rows of each of the Mel-spectrum matrices according to the first sampling frequency and occluding the sampled rows in the Mel-spectrum matrices until the number of occluded rows in the Mel-spectrum matrices is consistent with the preset first occlusion number, and determining the occluded Mel-spectrum matrix as the first speech enhancement matrix.

3. The speech recognition method according to claim 2, characterized in that, The step of randomly occluding the columns of each of the Mel-spectrum matrices to obtain the second speech enhancement matrix includes: counting the number of columns in each of the Mel-spectrum matrices, using the ratio of a second preset value to the number of columns as the second sampling frequency of each of the Mel-spectrum matrices; sampling all columns of each of the Mel-spectrum matrices according to the second sampling frequency, and occluding the sampled columns in the Mel-spectrum matrices; repeating the steps of sampling all columns of each of the Mel-spectrum matrices according to the first sampling frequency and occluding the sampled columns in the Mel-spectrum matrices until the number of occluded columns in the Mel-spectrum matrices is consistent with the preset second occlusion number, and determining the occluded Mel-spectrum matrix as the second speech enhancement matrix.

4. The speech recognition method according to claim 3, characterized in that, The first speech frame feature corresponds to the number of occluded rows, and the second speech frame feature corresponds to the number of occluded columns. The step of fusing the first speech frame features and the corresponding second speech frame features to obtain speech fusion features includes: calculating the sum of the number of occluded rows and the number of occluded columns; performing maximum value normalization on the number of occluded rows, the number of occluded columns, and the sum to obtain a first fusion weight for the first speech frame feature and a second fusion weight for the second speech frame feature; and performing weighted fusion based on the first speech frame feature, the first fusion weight, the corresponding second speech frame feature, and the second fusion weight to obtain the speech fusion features.

5. The speech recognition method according to claim 1, characterized in that, The step of calculating the prediction loss based on the N mapped characters, the N preset characters, and the preset prediction loss function includes: performing a pairing operation on the N mapped characters and the N preset characters to obtain N matching character pairs; calculating the prediction sub-loss of each matching character pair based on the mapped character, the preset character, and the preset prediction loss function in each matching character pair; and adding the prediction sub-losses of the N matching character pairs to determine the sum as the prediction loss.

6. The speech recognition method according to claim 5, characterized in that, The preset characters include zero characters and non-zero characters; the weighted summation of the prediction loss and the measurement loss to determine the target loss includes: counting the number of zero characters and the number of non-zero characters among the N preset characters; performing maximum normalization on the number of the first characters, the number of the second characters, and the number N of preset characters to obtain the normalized value of the non-zero characters, which is used as the first weight of the measurement loss, and the normalized value of the zero characters, which is used as the second weight of the prediction loss; and performing a weighted summation of the prediction loss and the measurement loss based on the prediction loss, the second weight, the measurement loss, and the first weight, to determine the target loss.

7. The speech recognition method according to claim 1, characterized in that, Training the encoder and decoder based on the target loss to obtain a trained encoder and a trained decoder includes: training the encoder and decoder according to gradient descent based on the target loss until the target loss converges, thereby obtaining a trained encoder and a trained decoder.

8. A speech recognition device based on artificial intelligence, characterized in that, The speech recognition device includes: a speech information acquisition module, used to acquire N speech frame signals of the speech to be recognized, and preset characters corresponding to each speech frame signal; for each speech frame signal, it acquires the Mel-spectrum matrix of the speech frame signal, where N is a positive integer; a speech enhancement module, used to randomly occlude the rows of each Mel-spectrum matrix to obtain a first speech enhancement matrix, and randomly occlude the columns of each Mel-spectrum matrix to obtain a second speech enhancement matrix; and a feature extraction module, used to input each of the first speech enhancement matrices into an encoder for feature extraction to obtain first speech frame features, input each of the second speech enhancement matrices into an encoder for feature extraction to obtain second speech frame features, and calculate N based on the N first speech frame features, N second speech frame features, and a preset metric loss function. The system includes a metric loss module; a feature fusion mapping module, used to fuse the features of each first speech frame and the corresponding features of the second speech frame to obtain N speech fusion features, and input each speech fusion feature into the decoder for feature mapping to obtain N mapped characters corresponding to the N speech frame signals; a loss calculation module, used to calculate the prediction loss based on the N mapped characters, the N preset characters and the preset prediction loss function, and add the N metric losses to calculate the metric loss; and a model training module, used to perform a weighted summation of the prediction loss and the metric loss, determine the weighted summation result as the target loss, and train the encoder and the decoder based on the target loss to obtain a trained encoder and a trained decoder, which are used as a trained speech recognition model for speech recognition.

9. A computer device, characterized in that, The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the speech recognition method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the speech recognition method as described in any one of claims 1 to 7.

Images