Speech recognition method and device, electronic equipment and storage medium

Abstract

The application relates to the technical field of speech recognition, and provides a speech recognition method and device, electronic equipment and a storage medium, which can greatly improve the accuracy of a speech recognition result, improve the efficiency of speech recognition and reduce the cost of speech recognition through extraction of target entity features in audio data. An audio entity feature extraction unit in a speech recognition model is trained based on a non-text-labeled audio and video training sample, so that not only is manual labeling reduced, labeling cost is reduced, but also the internal relationship between audio data and video data can be fully mined, the pre-training process pays more attention to entities in the audio, the speech recognition model can be applied to a speech recognition downstream task, the speech recognition model has the ability to improve hot word recognition effect. Moreover, the speech recognition model can complete different types of speech recognition tasks, can improve the generalization of the speech recognition model, and expands the application scenarios of the speech recognition model.

Description

Technical Field

[0001] This invention relates to the field of speech recognition technology, and in particular to a speech recognition method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the popularization of speech recognition, speech recognition technology has been applied in various fields. Users can use speech recognition technology to achieve intelligent input. They can complete text input, command control, etc., simply by speaking, which greatly facilitates people's production and life.

[0003] However, single-modal systems still have some inherent problems. For example, Automatic Speech Recognition (ASR) has insufficient tolerance to noise, its performance drops significantly when noise pollution is severe, and it cannot compensate for missing information when sound is missing. Video speech recognition, or Visual Speech Recognition (VSR), suffers from homonyms; the same lip shape may represent different word pronunciations, and the same word pronunciation may have different lip shape sequences. To address the shortcomings of single-modal systems, Audio-Visual Speech Recognition (AVSR) was proposed.

[0004] In existing technologies, audio and video speech recognition requires a large amount of manually labeled data, which is both time-consuming and expensive. Furthermore, most existing unsupervised audio and video representation learning models are pre-trained on video data used for action recognition and event detection, and are applied to downstream tasks such as video action recognition and sound event detection. These models focus on events occurring in the video and have limited application in downstream speech recognition tasks. Moreover, existing unsupervised audio and video representation learning models mostly learn global features, such as short-term instance-level representations in sound event classification scenarios. Such learning objectives may not be suitable for speech recognition, as speech recognition requires continuously changing sequence representations with long-term contextual dependencies. Summary of the Invention

[0005] This invention provides a speech recognition method, device, electronic device, and storage medium to address the deficiencies in the prior art.

[0006] This invention provides a speech recognition method, comprising:

[0007] Acquire the audio data to be processed;

[0008] The audio data is input into the acoustic feature extraction unit and the audio entity feature extraction unit of the speech recognition model, respectively, to obtain the target acoustic features output by the acoustic feature extraction unit and the target entity features output by the audio entity feature extraction unit.

[0009] The target acoustic features and the target entity features are input into the splicing unit of the speech recognition model to obtain the splicing result output by the splicing unit;

[0010] The splicing result is input into the speech recognition unit of the speech recognition model to obtain the speech recognition result output by the speech recognition unit;

[0011] The audio entity feature extraction unit is trained based on audio and video training samples without text annotations, and the speech recognition model is trained based on audio training samples with text annotations, on the basis of the audio entity feature extraction unit.

[0012] According to the speech recognition method provided by the present invention, the audio and video training samples include pairs of audio data samples and video data samples;

[0013] The audio entity feature extraction unit is trained based on the following steps:

[0014] Based on the initial audio feature extraction unit, feature extraction is performed on the audio data sample to obtain audio sample features;

[0015] Based on the initial video feature extraction unit, feature extraction is performed on the video data samples to obtain video sample features;

[0016] Based on the audio sample features and the video sample features, a first loss function is calculated, and based on the first loss function, the structural parameters of the initial audio feature extraction unit and the initial video feature extraction unit are simultaneously iterated.

[0017] The target audio feature extraction unit obtained by iterating the structural parameters is used as the audio entity feature extraction unit.

[0018] According to the speech recognition method provided by the present invention, the positive sample pairs in the audio and video training samples are determined based on the following steps:

[0019] Collect video segments from audio and video data within a preset time period, where the preset time period has a preset duration.

[0020] The audio and video data collected includes a preset number of audio segments of the preset duration, including the preset time period.

[0021] Each video segment and each audio segment are identified as a positive sample pair.

[0022] According to the speech recognition method provided by the present invention, the step of collecting the audio and video data, which includes a preset number of audio segments of the preset duration, including the preset time period, includes:

[0023] Determine the midpoint of the video segment;

[0024] Centered on the midpoint, and with a specified duration as the interval, select the preset number of audio segments;

[0025] Wherein, the specified duration is less than or equal to the preset duration.

[0026] According to the speech recognition method provided by the present invention, the first loss function includes a multi-instance learning noise contrast estimation loss function.

[0027] According to the speech recognition method provided by the present invention, the step of extracting features from the video data samples based on the initial video feature extraction unit to obtain video sample features includes:

[0028] Extract the Fbank features from the video data samples;

[0029] The Fbank features are input into the initial video feature extraction unit to obtain the video sample features output by the initial video feature extraction unit.

[0030] According to the speech recognition method provided by the present invention, the speech recognition model is trained based on the following steps:

[0031] The audio training samples are input into the initial acoustic feature extraction unit and the audio entity feature extraction unit respectively to obtain the sample acoustic features output by the initial acoustic feature extraction unit and the sample entity features output by the audio entity feature extraction unit.

[0032] The acoustic features and entity features of the sample are input into the initial stitching unit to obtain the sample stitching result output by the initial stitching unit;

[0033] The sample splicing result is input into the initial speech recognition unit to obtain the sample recognition result output by the initial speech recognition unit;

[0034] Based on the sample recognition results and the text annotations, a second loss function is calculated, and based on the second loss function, the structural parameters of the initial acoustic feature extraction unit, the initial splicing unit, and the initial speech recognition unit are simultaneously iterated to obtain the speech recognition model.

[0035] The present invention also provides a voice recognition device, comprising:

[0036] The data acquisition module is used to acquire the audio data to be processed;

[0037] The feature extraction module is used to input the audio data into the acoustic feature extraction unit and the audio entity feature extraction unit of the speech recognition model, respectively, to obtain the target acoustic features output by the acoustic feature extraction unit and the target entity features output by the audio entity feature extraction unit.

[0038] The feature splicing module is used to input the target acoustic features and the target entity features into the splicing unit of the speech recognition model to obtain the splicing result output by the splicing unit;

[0039] A speech recognition module is used to input the splicing result into the speech recognition unit of the speech recognition model to obtain the speech recognition result output by the speech recognition unit;

[0040] The audio entity feature extraction unit is trained based on audio and video training samples without text annotations, and the speech recognition model is trained based on audio training samples with text annotations, on the basis of the audio entity feature extraction unit.

[0041] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the speech recognition method as described above.

[0042] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the speech recognition method as described above.

[0043] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the speech recognition method as described above.

[0044] The speech recognition method, apparatus, electronic device, and storage medium provided by this invention first acquire the audio data to be processed; then, the audio data is input into the acoustic feature extraction unit and the audio entity feature extraction unit of the speech recognition model, respectively, to obtain the target acoustic features output by the acoustic feature extraction unit and the target entity features output by the audio entity feature extraction unit; subsequently, the target acoustic features and target entity features are input into the concatenation unit of the speech recognition model to obtain the concatenation result output by the concatenation unit; finally, the concatenation result is input into the speech recognition unit of the speech recognition model to obtain the speech recognition result output by the speech recognition unit. This method, by extracting target entity features from the audio data, can greatly improve the accuracy of speech recognition results, increase the efficiency of speech recognition, and reduce the cost of speech recognition. The audio entity feature extraction unit in the speech recognition model is trained based on unannotated audio and video training samples, which not only eliminates the need for manual annotation, reducing annotation costs, but also fully explores the inherent relationship between audio and video data, making the pre-training process more focused on entities in the audio. This can be applied to downstream tasks in speech recognition, enabling the speech recognition model to improve the performance of hot word recognition. Moreover, this speech recognition model can complete different types of speech recognition tasks, which can improve the generalization of the speech recognition model and expand the application scenarios of the speech recognition model. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on the drawings described below without creative effort.

[0046] Figure 1 This is one of the flowcharts of the speech recognition method provided by the present invention;

[0047] Figure 2 This is a schematic diagram of positive sample pairs in the speech recognition method provided by the present invention;

[0048] Figure 3 This is the second flowchart of the speech recognition method provided by the present invention;

[0049] Figure 4 This is a schematic diagram of the training process of the audio entity feature extraction unit in the speech recognition method provided by the present invention;

[0050] Figure 5 This is a schematic diagram of the structure of the speech recognition device provided by the present invention;

[0051] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0053] Existing audio and video pre-training schemes typically include Advanced Video Coding (AVC), Audio Visual Technology Services (AVTS), and Cross-Modal Deep Clustering (XDC), all based on the idea of ​​audio-video synchronization. These schemes primarily leverage the observation that visual and audio events often occur simultaneously, constructing a pre-training task: cutting unsupervised videos into 1-3 second clips. The video features of each clip should be correlated with its corresponding audio features, while the audio features of other clips should be uncorrelated. This allows the construction of a network structure to extract audio and video features separately, and model learning is performed by determining the correlation between the input audio and video. However, these pre-training schemes primarily use action recognition and event detection video data for pre-training, applying to downstream tasks such as video action recognition and sound event detection. They focus on events occurring in the video and have limited application in downstream tasks such as speech recognition, and they lack the ability to improve hot word recognition performance.

[0054] Furthermore, most existing models that employ unsupervised audio and video representation learning learn global features, such as short-term instance-level representations in sound event classification scenarios. Such learning objectives may not be suitable for speech recognition, because speech recognition requires continuously changing sequence representations that contain long-term contextual dependencies.

[0055] Based on this, an embodiment of the present invention provides a speech recognition method.

[0056] Figure 1 This is a flowchart illustrating a speech recognition method provided in an embodiment of the present invention, as shown below. Figure 1 As shown, the method includes:

[0057] S1, acquire the audio data to be processed;

[0058] S2, the audio data is input to the acoustic feature extraction unit and the audio entity feature extraction unit of the speech recognition model respectively, to obtain the target acoustic features output by the acoustic feature extraction unit and the target entity features output by the audio entity feature extraction unit;

[0059] S3, input the target acoustic features and the target entity features into the splicing unit of the speech recognition model to obtain the splicing result output by the splicing unit;

[0060] S4, input the splicing result into the speech recognition unit of the speech recognition model to obtain the speech recognition result output by the speech recognition unit;

[0061] The audio entity feature extraction unit is trained based on audio and video training samples without text annotations, and the speech recognition model is trained based on audio training samples with text annotations, on the basis of the audio entity feature extraction unit.

[0062] Specifically, the speech recognition method provided in this embodiment of the invention is executed by a speech recognition device, which can be configured in a computer. The computer can be a local computer or a cloud computer. The local computer can be a computer, tablet, etc., and no specific limitation is made here.

[0063] First, step S1 is executed to obtain the audio data to be processed, which refers to the speech that needs to be converted into text. The domain of this audio data can be set according to the actual situation, and is not specifically limited here.

[0064] Then, step S2 is executed, introducing a speech recognition model. This model may include an acoustic feature extraction unit, an audio entity feature extraction unit, a concatenation unit, and a speech recognition unit. Here, audio data can be input into the acoustic feature extraction unit and the audio entity feature extraction unit of the speech recognition model, respectively, to obtain the target acoustic features output by the acoustic feature extraction unit and the target entity features output by the audio entity feature extraction unit. The acoustic feature extraction unit can be an encoder structure, and the audio entity feature extraction unit can be an LSTM model. The target acoustic features can be used to represent the acoustic information in the audio data, and the target entity features can be used to represent the entity information in the audio data.

[0065] Next, step S3 is executed, whereby the target acoustic features and target entity features are input into the concatenation unit of the speech recognition model to obtain the concatenation result output by the concatenation unit. This concatenation unit can concatenate the target acoustic features and target entity features along the channel dimension. It is understandable that if the frame length dimensions of the target acoustic features and target entity features are different, they need to be adjusted to be the same through duplication.

[0066] Finally, step S4 is executed, inputting the concatenation result into the speech recognition unit of the speech recognition model to obtain the speech recognition result output by the speech recognition unit. This speech recognition result is the text converted from the audio data to be processed.

[0067] In this speech recognition model, the audio entity feature extraction unit can be obtained through unsupervised training based on unannotated audio and video training samples. These training samples can be synchronized audio and video data. Here, the initial audio feature extraction unit and the initial video feature extraction unit can be simultaneously trained in a self-supervised manner using the audio and video training samples, yielding the target audio feature extraction unit and the target video feature extraction unit, respectively. Subsequently, the trained target audio feature extraction unit can be used as the audio entity feature extraction unit.

[0068] Because the audio and video in the training samples are synchronous and follow-up, training with these samples enables the target audio feature extraction unit to extract entity features from the audio, and the target video feature extraction unit to extract entity features from the video. Therefore, using the target audio entity feature extraction unit as the audio entity feature extraction unit allows for the introduction of entity feature extraction during speech recognition, thereby improving the accuracy of entity recognition and the overall accuracy of the speech recognition results.

[0069] The speech recognition model is trained on audio training samples with text annotations, based on the audio entity feature extraction unit. That is, the structural parameters of the audio entity feature extraction unit are fixed, and the other units in the speech recognition model are trained. The text annotations can be processed into word form according to the word modeling of BPE segmentation; no specific limitation is made here.

[0070] Here, it can be understood that the speech recognition model is obtained by training an initial speech recognition model using audio training samples with text annotations. This initial speech recognition model includes parts that need to be trained and parts that do not need to be trained. The parts that need to be trained include the initial acoustic feature extraction unit, the initial splicing unit, and the initial speech recognition unit. The parts that do not need to be trained refer to the audio entity feature extraction unit, because this unit has already completed self-supervised training.

[0071] The speech recognition method provided in this embodiment of the invention first acquires the audio data to be processed; then, the audio data is input into the acoustic feature extraction unit and the audio entity feature extraction unit of the speech recognition model, respectively, to obtain the target acoustic features output by the acoustic feature extraction unit and the target entity features output by the audio entity feature extraction unit; subsequently, the target acoustic features and the target entity features are input into the concatenation unit of the speech recognition model to obtain the concatenation result output by the concatenation unit; finally, the concatenation result is input into the speech recognition unit of the speech recognition model to obtain the speech recognition result output by the speech recognition unit. This method, by extracting target entity features from the audio data, can greatly improve the accuracy of speech recognition results, enhance the efficiency of speech recognition, and reduce the cost of speech recognition. The audio entity feature extraction unit in the speech recognition model is trained based on unannotated audio and video training samples, which not only eliminates the need for manual annotation, reducing annotation costs, but also fully explores the inherent relationship between audio and video data, making the pre-training process more focused on entities in the audio. This can be applied to downstream tasks in speech recognition, enabling the speech recognition model to improve the performance of hot word recognition. Moreover, this speech recognition model can complete different types of speech recognition tasks, which can improve the generalization of the speech recognition model and expand the application scenarios of the speech recognition model.

[0072] Based on the above embodiments, the speech recognition method provided in this embodiment of the invention includes paired audio data samples and video data samples as the audio and video training samples;

[0073] The audio entity feature extraction unit is trained based on the following steps:

[0074] Based on the initial audio feature extraction unit, feature extraction is performed on the audio data sample to obtain audio sample features;

[0075] Based on the initial video feature extraction unit, feature extraction is performed on the video data samples to obtain video sample features;

[0076] Based on the audio sample features and the video sample features, a first loss function is calculated, and based on the first loss function, the structural parameters of the initial audio feature extraction unit and the initial video feature extraction unit are simultaneously iterated.

[0077] The target audio feature extraction unit obtained by iterating the structural parameters is used as the audio entity feature extraction unit.

[0078] Specifically, in this embodiment of the invention, the audio and video training samples may include pairs of audio data samples and video data samples. The number of pairs of audio and video data samples included in the audio and video training samples can be set as needed and is not specifically limited here. Pairs of audio and video data samples may include both positive and negative sample pairs. Positive sample pairs consist of audio and video data samples that correspond in time, while negative sample pairs consist of audio and video data samples that do not correspond in time. Here, "corresponding in time" can mean being in the same time period or being in a period that includes and is included within another period. For example, the audio and video data samples included in a positive sample pair may be in the same time period and have the same duration, both being 3.2 seconds. The time period of the audio data samples included in a positive sample pair may also be a combination of multiple 3.2-second time periods within a longer time period, including the time period containing the video data samples. The longer time period could be 9.6 seconds, 16 seconds, etc.

[0079] The frame rate of the video data samples can be set according to computational complexity and model performance; for example, it can be 10 pfs or other values, without specific limitations here. The number of channels for the video data samples can be RGB three channels, and the image size of each frame can be 112×112 or other sizes. The number of channels for the audio data samples can be single channel, and the sampling rate can be 16000Hz or other values.

[0080] Based on this, when training the audio entity feature extraction unit, audio data samples can be first input into the initial audio feature extraction unit. This initial audio feature extraction unit can then be used to extract features from the audio data samples to obtain the audio sample features. The network structure of this initial audio feature extraction unit can be an LSTM model.

[0081] Video data samples are input into an initial video feature extraction unit, which extracts features from the video data samples to obtain video sample features. The network structure of this initial video feature extraction unit can be a 3Dres18 network. The network size of this initial video feature extraction unit can be 32×3×112×112.

[0082] Subsequently, a first loss function can be calculated using audio and video sample features. Specifically, paired audio and video sample features are treated as positive examples, while unpaired audio and video sample features are treated as negative examples. Then, this first loss function can be used to simultaneously iterate the structure parameters of the initial audio feature extraction unit and the initial video feature extraction unit until a preset condition is met, resulting in the target audio feature extraction unit corresponding to the initial audio feature extraction unit and the target video feature extraction unit corresponding to the initial video feature extraction unit, respectively. This preset condition may include reaching a preset number of iterations or the convergence of the first loss function.

[0083] Subsequently, the target audio feature extraction unit obtained by iterating the structural parameters can be directly used as the audio entity feature extraction unit for extracting entity features from subsequent audio data.

[0084] In this embodiment of the invention, the initial audio feature extraction unit and the initial video feature extraction unit are jointly trained using audio and video training samples, which can fully explore the entity information associated with the audio and video, thereby improving the entity recognition accuracy in speech recognition.

[0085] Because commentary in sports, game, and live-streaming e-commerce (i.e., commentary videos) involves a degree of randomness, there may be a temporal misalignment between the commentator's narration and the actual video content. That is, the commentator's current narration might correspond to footage from a few seconds earlier or later, rather than directly to the current moment. This is due to the inherent divergence and lag in the commentator's narration, making it impossible to maintain perfect consistency with the current moment's video. However, the commentator's narration is not entirely random; therefore, corresponding video footage can usually be found within a short period before and after the narration. To address this, multiple positive sample pairs can be constructed around the corresponding video footage, assuming that each positive sample pair contains narration content that perfectly corresponds to the video footage.

[0086] Based on this, and building upon the above embodiments, the speech recognition method provided in this embodiment of the invention determines the positive sample pairs in the audio and video training samples based on the following steps:

[0087] Collect video segments from audio and video data within a preset time period, where the preset time period has a preset duration.

[0088] The audio and video data collected includes a preset number of audio segments of the preset duration, including the preset time period.

[0089] The video segment and the preset number of audio segments are determined as a positive sample pair.

[0090] Specifically, in this embodiment of the invention, when selecting positive sample pairs from the audio and video training samples, audio and video data can be obtained first. This audio and video data can be narration-type audio and video data. This audio and video data can first be processed using a voice VAD tool to remove silent video segments longer than 1 second from the audio and video data.

[0091] Subsequently, the video data in the audio and video data can be divided into multiple video segments of preset duration, and the time period corresponding to each video segment can be used as a preset time period. Then, a preset number of audio segments of preset duration, including those within the preset time periods, can be collected from the audio and video data. This preset number can be set as needed and can be either an odd number or an integer. For example, it could be 2, 3, 4, 5, etc., without specific limitations here.

[0092] Finally, a video segment within a preset time period and a preset number of audio segments of preset duration can be directly used as a positive sample pair. That is, the video segment within the preset time period is used as the input to the initial video feature extraction unit, and the preset number of audio segments of preset duration are used as the input to the initial audio feature extraction unit. This ensures that the audio and video training samples include audio segments within a preset time period before and after the video segments, resolving the audio-video misalignment problem and thus ensuring the accuracy of entity feature recognition by the audio entity feature extraction unit, thereby improving its performance.

[0093] Based on the above embodiments, the speech recognition method provided in this embodiment of the invention, wherein the acquisition of the audio and video data includes a preset number of audio segments of the preset duration, including the preset time period, comprises:

[0094] Determine the midpoint of the video segment;

[0095] Centered on the midpoint, and with a specified duration as the interval, select the preset number of audio segments;

[0096] Wherein, the specified duration is less than or equal to the preset duration.

[0097] Specifically, in this embodiment of the invention, when collecting audio and video data that includes a preset number of audio segments of preset duration, including a preset time period, such as... Figure 2 As shown, the midpoint time t0 of video segment v can be determined first. Then, with the midpoint time t0 as the center, a preset number of audio segments are selected at intervals of a specified duration. The specified duration is less than or equal to the preset duration, meaning that there are overlapping segments among the audio segments collected in adjacent time periods. This ensures the training effect of the audio entity feature extraction unit.

[0098] If the preset duration is 3.2s, then the specified duration can be less than or equal to 3.2s, for example, it can be 1.6s. Figure 2 The example shown illustrates the case where the preset quantity is 5. In this case, there are 5 audio segments, namely a1, a2, a3, a4, and a5. Furthermore, the video segment v and the audio segments a1, a2, a3, a4, and a5 are considered as a pair of audio data samples and video data samples.

[0099] In this embodiment of the invention, selecting a preset number of audio segments at intervals of a specified duration, centered on the middle moment of a video segment, allows the audio segments to be evenly distributed before and after the video segment, increasing the probability that each audio segment corresponds to the content of the video segment. Furthermore, specifying a duration less than or equal to the preset duration ensures overlapping segments between the audio segments, resulting in better performance of the trained audio entity feature extraction unit.

[0100] Based on the above embodiments, the speech recognition method provided in the embodiments of the present invention includes a first loss function comprising a multi-instance learning noise contrast estimation loss function.

[0101] Specifically, in this embodiment of the invention, during the training of the audio entity feature extraction unit, it is desirable that the video features learned by the audio entity feature extraction unit are close to the audio features in the positive sample pairs, while being far from the audio features in the negative sample pairs. However, in reality, it is impossible to determine which of the positive sample pairs is the true positive example. Therefore, the first loss function is set as the Multiple-instance learning Noise-Constrastive Estimation Loss (MIL-NCE loss), which transforms the above problem into the following optimization problem:

[0102]

[0103] Where n is the number of training batches (pitch), Let represent the set of positive sample pairs, where x and y represent a positive sample pair. x represents the audio data sample in the positive sample pair, y represents the video data sample in the positive sample pair, f(x) represents the audio sample feature in x, and g(y) represents the video sample feature in y. Let f(x) represent the set of negative sample pairs, x′, y′ represent a negative sample pair, x′ represent the audio data sample in the negative sample pair, y′ represent the video sample feature in the negative sample pair, and f(x) represent the video sample feature in the negative sample pair. ′ ) represents the audio sample features in x′, and g(y′) represents the video sample features in y′.

[0104] Its idea is to avoid the problem of unknown true positive pairs by narrowing the distance between all positive sample pairs. As training progresses, the true positive pair will gradually take the lead, thus achieving the goal of narrowing the distance between true positive sample pairs and widening the distance between negative sample pairs.

[0105] In commentary scenarios, the correspondence between video footage and narrated content is considered to be primarily determined by entity information. For example, in football commentary, the image of player "MX" in the video and the word "MX" mentioned in the commentator's speech are strongly correlated in both video and audio. The training process aims to ensure that both the target audio feature extraction unit and the target video feature extraction unit learn this key entity information, "MX." Therefore, through the above training steps, the resulting target audio and video feature extraction units can effectively learn to extract entity features from speech and video. This allows the target audio feature extraction units trained in this way to assist in speech recognition, improving the recognition performance of entity features such as proper nouns.

[0106] Based on the above embodiments, the speech recognition method provided in this embodiment of the invention, wherein the step of extracting features from the audio data samples based on the initial audio feature extraction unit to obtain audio sample features includes:

[0107] Extract the Fbank feature from the audio data sample;

[0108] The Fbank features are input into the initial audio feature extraction unit to obtain the audio sample features output by the initial audio feature extraction unit.

[0109] Specifically, in this embodiment of the invention, the input to the initial audio feature extraction unit can be either an audio data sample or the F-bank features within the audio data sample. Therefore, the F-bank features in the audio data sample can be extracted first, for example, an 80-dimensional F-bank feature. When extracting the F-bank features, the window length can be set to 25ms and the frame shift to 10ms.

[0110] Subsequently, the extracted Fbank features can be input into the initial audio feature extraction unit to obtain the audio sample features output by the initial audio feature extraction unit.

[0111] In this embodiment of the invention, there are two reasons for using Fbank features: 1) Fbank features transform audio from the time domain to the frequency domain, allowing for the extraction of stable audio features; 2) The spectrum of Fbank features is a two-dimensional feature, which can be processed using deep neural networks such as convolution and Transformer.

[0112] Based on the above embodiments, the speech recognition method provided in this embodiment of the invention determines the speech recognition model based on the following steps:

[0113] The audio training samples are input into the initial acoustic feature extraction unit and the audio entity feature extraction unit respectively to obtain the sample acoustic features output by the initial acoustic feature extraction unit and the sample entity features output by the audio entity feature extraction unit.

[0114] The acoustic features and entity features of the sample are input into the initial stitching unit to obtain the sample stitching result output by the initial stitching unit;

[0115] The sample splicing result is input into the initial speech recognition unit to obtain the sample recognition result output by the initial speech recognition unit;

[0116] Based on the sample recognition results and the text annotations, a second loss function is calculated, and based on the second loss function, the structural parameters of the initial acoustic feature extraction unit, the initial splicing unit, and the initial speech recognition unit are simultaneously iterated to obtain the target acoustic feature extraction unit, the target splicing unit, and the target speech recognition unit.

[0117] The speech recognition model is determined based on the target acoustic feature extraction unit, the target splicing unit, the target speech recognition unit, and the audio entity feature extraction unit.

[0118] Specifically, when training the initial recognition model to obtain the speech recognition model, the audio training samples can be input into the initial acoustic feature extraction unit and the audio entity feature extraction unit respectively to obtain the sample acoustic features output by the initial acoustic feature extraction unit and the sample entity features output by the audio entity feature extraction unit.

[0119] Then, the sample entity features are copied along the frame length dimension, and the copied result, along with the sample acoustic features, is input into the initial stitching unit. The initial stitching unit then stitches the copied result and the sample acoustic features along the channel dimension to obtain the sample stitching result. This result is then input into the initial speech recognition unit to obtain the sample recognition result output by the initial speech recognition unit. Finally, the sample recognition result and text annotations can be used to calculate the second loss function. The initial acoustic feature extraction unit can be an encoder, and the initial speech recognition unit can be a decoder, forming an encoder-decoder structure. In this case, the second loss function can be the cross-entropy loss function, such as CE loss. The initial speech recognition unit can also be a fully connected network, in which case the second loss function can be CTC loss.

[0120] Based on the second loss function, the structural parameters of the initial acoustic feature extraction unit, the initial splicing unit, and the initial speech recognition unit can be iterated simultaneously to finally obtain the target acoustic feature extraction unit, the target splicing unit, and the target speech recognition unit.

[0121] Finally, the target acoustic feature extraction unit, target splicing unit, and target speech recognition unit are combined with the audio entity feature extraction unit to form a speech recognition model.

[0122] In summary, such as Figure 3 The diagram shown illustrates a flowchart of a speech recognition method provided in an embodiment of the present invention. In this method, the acoustic feature extraction unit is an encoder, the audio entity feature extraction unit is an LSTM model, and the speech recognition unit is a decoder. The method includes:

[0123] 1) Acquire the audio data to be processed (Audio);

[0124] 2) Input the audio data into the encoder and the LSTM model respectively to obtain the target acoustic features output by the encoder and the target entity features output by the LSTM model;

[0125] 3) Input the target acoustic features and target entity features into the stitching unit to obtain the stitching result output by the stitching unit;

[0126] 4) Input the splicing result into the decoder, which performs autoregressive decoding to obtain and output the speech recognition result.

[0127] like Figure 4 As shown, during the training of the audio entity feature extraction unit, the initial video feature extraction unit uses a 3Dres18 network, while the initial audio feature extraction unit uses an LSTM model. The input to the initial video feature extraction unit is a 3.2s video clip, processed into a 32*3*112*112 format, which extracts video sample features g(y). The initial audio feature extraction unit's single input is five 3.2s audio clips (Audio*5) extracting Fbank features, which extract five audio sample features f(x). To address the audio-video misalignment issue, the initial audio feature extraction unit inputs five 3.2s audio clips before and after the time segment corresponding to the initial video feature extraction unit, extracting five audio features as subsequent positive samples.

[0128] During the training process, MIL-NCE loss is used as the first loss function for supervision.

[0129] This invention provides a scheme for improving speech recognition performance in narration scenarios, particularly hot word recognition, based on audio and video pre-training. First, a large amount of unsupervised narration video data is collected and then cleaned. Next, a network is constructed to extract video and audio features separately. MIL-NCE loss is used to address potential video-audio misalignment issues in narration videos, thus achieving audio and video pre-training. Since the most significant relevant information in narration videos and audio is generally entities such as people or objects, this pre-training is believed to significantly enhance the model's ability to extract entity information. After pre-training, the target audio feature extraction unit is extracted. In the downstream supervised speech recognition task, the information extracted by this target audio feature extraction unit is concatenated with the features extracted by the encoder in the original speech recognition network, and then fed into the decoder for speech recognition training. This scheme improves the speech recognition model's ability to capture entity information through audio and video pre-training, further enhancing the recognition performance of proper nouns in entity word classes during speech recognition.

[0130] like Figure 5 As shown, based on the above embodiments, this embodiment of the invention provides a voice recognition device, including:

[0131] Data acquisition module 51 is used to acquire audio data to be processed;

[0132] Feature extraction module 52 is used to input the audio data into the acoustic feature extraction unit and the audio entity feature extraction unit of the speech recognition model respectively, and obtain the target acoustic features output by the acoustic feature extraction unit and the target entity features output by the audio entity feature extraction unit.

[0133] The feature splicing module 53 is used to input the target acoustic features and the target entity features into the splicing unit of the speech recognition model to obtain the splicing result output by the splicing unit;

[0134] The speech recognition module 54 is used to input the splicing result into the speech recognition unit of the speech recognition model to obtain the speech recognition result output by the speech recognition unit;

[0135] The audio entity feature extraction unit is trained based on audio and video training samples without text annotations, and the speech recognition model is trained based on audio training samples with text annotations, on the basis of the audio entity feature extraction unit.

[0136] Based on the above embodiments, the speech recognition device provided in this embodiment of the invention includes paired audio data samples and video data samples as the audio and video training samples;

[0137] The speech recognition device includes an audio / video pre-training module, used for:

[0138] Based on the initial audio feature extraction unit, feature extraction is performed on the audio data sample to obtain audio sample features;

[0139] Based on the initial video feature extraction unit, feature extraction is performed on the video data samples to obtain video sample features;

[0140] Based on the audio sample features and the video sample features, a first loss function is calculated, and based on the first loss function, the structural parameters of the initial audio feature extraction unit and the initial video feature extraction unit are simultaneously iterated.

[0141] The target audio feature extraction unit obtained by iterating the structural parameters is used as the audio entity feature extraction unit.

[0142] Based on the above embodiments, the speech recognition device provided in this embodiment of the invention further includes a positive sample pair determination module, used for:

[0143] Collect video segments from audio and video data within a preset time period, where the preset time period has a preset duration.

[0144] The audio and video data collected includes a preset number of audio segments of the preset duration, including the preset time period.

[0145] The video segment and the preset number of audio segments are determined as a positive sample pair.

[0146] Based on the above embodiments, the speech recognition device provided in this embodiment of the invention, wherein the positive sample pair determination module is specifically used for:

[0147] Determine the midpoint of the video segment;

[0148] Centered on the midpoint, and with a specified duration as the interval, select the preset number of audio segments;

[0149] Wherein, the specified duration is less than or equal to the preset duration.

[0150] Based on the above embodiments, the speech recognition device provided in the embodiments of the present invention includes a first loss function comprising a multi-instance learning noise contrast estimation loss function.

[0151] Based on the above embodiments, the speech recognition device provided in this embodiment of the invention, wherein the audio and video pre-training module is specifically used for:

[0152] Extract the Fbank feature from the audio data sample;

[0153] The Fbank features are input into the initial audio feature extraction unit to obtain the audio sample features output by the initial audio feature extraction unit.

[0154] Based on the above embodiments, the speech recognition device provided in this embodiment of the invention further includes a model training module, used for:

[0155] The audio training samples are input into the initial acoustic feature extraction unit and the audio entity feature extraction unit respectively to obtain the sample acoustic features output by the initial acoustic feature extraction unit and the sample entity features output by the audio entity feature extraction unit.

[0156] The acoustic features and entity features of the sample are input into the initial stitching unit to obtain the sample stitching result output by the initial stitching unit;

[0157] The sample splicing result is input into the initial speech recognition unit to obtain the sample recognition result output by the initial speech recognition unit;

[0158] Based on the sample recognition results and the text annotations, a second loss function is calculated, and based on the second loss function, the structural parameters of the initial acoustic feature extraction unit, the initial splicing unit, and the initial speech recognition unit are simultaneously iterated to obtain the target acoustic feature extraction unit, the target splicing unit, and the target speech recognition unit.

[0159] The speech recognition model is determined based on the target acoustic feature extraction unit, the target splicing unit, the target speech recognition unit, and the audio entity feature extraction unit.

[0160] Specifically, the functions of each module in the speech recognition device provided in this embodiment of the invention correspond one-to-one with the operation flow of each step in the above method-like embodiments, and the achieved effects are also the same. For details, please refer to the above embodiments, and this will not be repeated in this embodiment of the invention.

[0161] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call logic instructions in the memory 630 to execute the speech recognition method provided in the above embodiments. The method includes: acquiring audio data to be processed; inputting the audio data into the acoustic feature extraction unit and the audio entity feature extraction unit of the speech recognition model respectively to obtain the target acoustic features output by the acoustic feature extraction unit and the target entity features output by the audio entity feature extraction unit; inputting the target acoustic features and the target entity features into the splicing unit of the speech recognition model to obtain the splicing result output by the splicing unit; inputting the splicing result into the speech recognition unit of the speech recognition model to obtain the speech recognition result output by the speech recognition unit; wherein, the audio entity feature extraction unit is trained based on audio and video training samples without text annotations, and the speech recognition model is trained based on audio training samples with text annotations, on the basis of the audio entity feature extraction unit.

[0162] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0163] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the speech recognition method provided in the above embodiments. The method includes: acquiring audio data to be processed; inputting the audio data into an acoustic feature extraction unit and an audio entity feature extraction unit of a speech recognition model to obtain target acoustic features output by the acoustic feature extraction unit and target entity features output by the audio entity feature extraction unit; inputting the target acoustic features and the target entity features into a splicing unit of the speech recognition model to obtain a splicing result output by the splicing unit; and inputting the splicing result into a speech recognition unit of the speech recognition model to obtain a speech recognition result output by the speech recognition unit. The audio entity feature extraction unit is trained based on audio and video training samples without text annotations, and the speech recognition model is trained based on audio training samples with text annotations, building upon the audio entity feature extraction unit.

[0164] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the speech recognition method provided in the above embodiments. The method includes: acquiring audio data to be processed; inputting the audio data into an acoustic feature extraction unit and an audio entity feature extraction unit of a speech recognition model, respectively, to obtain target acoustic features output by the acoustic feature extraction unit and target entity features output by the audio entity feature extraction unit; inputting the target acoustic features and the target entity features into a concatenation unit of the speech recognition model, to obtain a concatenation result output by the concatenation unit; and inputting the concatenation result into a speech recognition unit of the speech recognition model, to obtain a speech recognition result output by the speech recognition unit; wherein the audio entity feature extraction unit is trained based on audio and video training samples without text annotations, and the speech recognition model is trained based on audio training samples with text annotations, on the basis of the audio entity feature extraction unit.

[0165] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0166] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0167] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A voice recognition method, characterized by, include: Acquire the audio data to be processed; The audio data is input into the acoustic feature extraction unit and the audio entity feature extraction unit of the speech recognition model, respectively, to obtain the target acoustic features output by the acoustic feature extraction unit and the target entity features output by the audio entity feature extraction unit. The target acoustic features and the target entity features are input into the splicing unit of the speech recognition model to obtain the splicing result output by the splicing unit; The splicing result is input into the speech recognition unit of the speech recognition model to obtain the speech recognition result output by the speech recognition unit; The audio entity feature extraction unit is trained based on audio and video training samples without text annotations, and the speech recognition model is trained based on audio training samples with text annotations, on the basis of the audio entity feature extraction unit. The audio and video training samples include pairs of audio data samples and video data samples; The audio entity feature extraction unit is trained based on the following steps: Based on the initial audio feature extraction unit, feature extraction is performed on the audio data sample to obtain audio sample features; Based on the initial video feature extraction unit, feature extraction is performed on the video data samples to obtain video sample features; Based on the audio sample features and the video sample features, a first loss function is calculated, and based on the first loss function, the structural parameters of the initial audio feature extraction unit and the initial video feature extraction unit are simultaneously iterated. The target audio feature extraction unit obtained by iterating the structural parameters is used as the audio entity feature extraction unit.

2. The speech recognition method according to claim 1, characterized in that, The positive sample pairs in the audio and video training samples are determined based on the following steps: Collect video segments from audio and video data within a preset time period, where the preset time period has a preset duration. The audio and video data collected includes a preset number of audio segments of the preset duration, including the preset time period. The video segment and the preset number of audio segments are determined as a positive sample pair.

3. The speech recognition method according to claim 2, characterized in that, The acquisition of the audio and video data includes a preset number of audio segments of the preset duration, including the preset time period, comprising: Determine the midpoint of the video segment; Centered on the midpoint, and with a specified duration as the interval, select the preset number of audio segments; Wherein, the specified duration is less than or equal to the preset duration.

4. The speech recognition method according to claim 1, characterized in that, The first loss function includes a multi-instance learning noise contrastive estimation loss function.

5. The speech recognition method according to claim 1, characterized in that, The initial audio feature extraction unit extracts features from the audio data samples to obtain audio sample features, including: Extract the Fbank feature from the audio data sample; The Fbank features are input into the initial audio feature extraction unit to obtain the audio sample features output by the initial audio feature extraction unit.

6. The speech recognition method according to any one of claims 1-5, characterized in that, The speech recognition model is determined based on the following steps: The audio training samples are input into the initial acoustic feature extraction unit and the audio entity feature extraction unit respectively to obtain the sample acoustic features output by the initial acoustic feature extraction unit and the sample entity features output by the audio entity feature extraction unit. The acoustic features and entity features of the sample are input into the initial stitching unit to obtain the sample stitching result output by the initial stitching unit; The sample splicing result is input into the initial speech recognition unit to obtain the sample recognition result output by the initial speech recognition unit; Based on the sample recognition results and the text annotations, a second loss function is calculated, and based on the second loss function, the structural parameters of the initial acoustic feature extraction unit, the initial splicing unit, and the initial speech recognition unit are simultaneously iterated to obtain the target acoustic feature extraction unit, the target splicing unit, and the target speech recognition unit. The speech recognition model is determined based on the target acoustic feature extraction unit, the target splicing unit, the target speech recognition unit, and the audio entity feature extraction unit.

7. A voice recognition device, characterized in that, include: The data acquisition module is used to acquire the audio data to be processed; The feature extraction module is used to input the audio data into the acoustic feature extraction unit and the audio entity feature extraction unit of the speech recognition model, respectively, to obtain the target acoustic features output by the acoustic feature extraction unit and the target entity features output by the audio entity feature extraction unit. The feature splicing module is used to input the target acoustic features and the target entity features into the splicing unit of the speech recognition model to obtain the splicing result output by the splicing unit; A speech recognition module is used to input the splicing result into the speech recognition unit of the speech recognition model to obtain the speech recognition result output by the speech recognition unit; The audio entity feature extraction unit is trained based on audio and video training samples without text annotations, and the speech recognition model is trained based on audio training samples with text annotations, on the basis of the audio entity feature extraction unit. The audio and video training samples include pairs of audio data samples and video data samples; The audio entity feature extraction unit is trained based on the following steps: Based on the initial audio feature extraction unit, feature extraction is performed on the audio data sample to obtain audio sample features; Based on the initial video feature extraction unit, feature extraction is performed on the video data samples to obtain video sample features; Based on the audio sample features and the video sample features, a first loss function is calculated, and based on the first loss function, the structural parameters of the initial audio feature extraction unit and the initial video feature extraction unit are simultaneously iterated. The target audio feature extraction unit obtained by iterating the structural parameters is used as the audio entity feature extraction unit.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the speech recognition method as described in any one of claims 1-6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the speech recognition method as described in any one of claims 1-6.

Images