Multimodal speech recognition method based on visual scene, electronic device, and medium

Abstract

The present invention discloses a multimodal speech recognition method based on visual scenes, an electronic device, and a medium, comprising: obtaining speech audio and a scene video corresponding to the environment in which the speech audio is located; inputting the scene video into a pre-trained audio-visual feature coding model for feature extraction, respectively obtaining acoustic features and visual features, and obtaining audio-visual fusion features through splicing and fusion; wherein the training process of the audio-visual feature coding model comprises: obtaining initial acoustic features and initial visual features, and obtaining audio-visual fusion features through splicing and fusion; clustering the audio-visual fusion features in each time window to obtain clustering labels; randomly selecting the audio-visual fusion features of some time windows for masking; using the audio-visual fusion features of unmasked time windows to predict the clustering labels of masked time windows to complete the training process; inputting the audio-visual fusion features into a fine-tuned speech recognition decoding model to convert them into speech text to obtain a speech recognition result.

Description

Technical Field

[0001] The present invention belongs to the field of speech recognition technology, and in particular relates to a multimodal speech recognition method, electronic equipment, and medium based on visual scenes. Background Art

[0002] Speech is an important information carrier in human society. As an important means of human-computer interaction, speech recognition technology has wide application value in the fields of communication, vehicle-mounted, industry, etc. In recent years, with the rapid development of artificial intelligence algorithms, speech recognition methods based on deep neural networks have greatly improved the accuracy of speech recognition. Mainstream speech recognition technology mainly uses single-modal information (i.e., audio) for speech recognition. The research of Spille et al. (Spille C, Ewert SD, Kollmeier B, Meyer BT. Predicting speech intelligibility with deep neural networks. Computer Speech & Language, 2018, 48: 51-66) pointed out that the performance of mainstream single-modal speech recognition technology in simple and quiet scenes is close to that of humans, but the performance in complex and noisy scenes is still seriously insufficient; in contrast, humans can maintain stable and efficient speech recognition in complex scenes. Therefore, learning from the auditory speech processing mechanism of the human brain has gradually become one of the important ways to solve the insufficient performance of current speech models. In complex noisy environments, the human brain often uses visual information as a supplementary clue for speech processing. Specifically, the speaker's facial movements (such as expressions and lip movements) are interrelated with the acoustic fluctuations of speech, which can promote the human brain's perception and recognition of speech. Inspired by brain science research, multimodal speech recognition technology has been proposed in the field of speech recognition, which uses multimodal information (i.e., audio and video) for speech recognition. Xu et al.'s research (Xu B, Lu C, Guo Y, Wang J. Discriminative multi-modality speech recognition. Proceedings of the IEEE / CVF conference on Computer Vision and Pattern Recognition. 2020: 14433-14442) found that using the target speaker's video information can effectively improve the speech recognition performance of the speech recognition model in noisy environments.

[0003] Current multimodal speech recognition technology mainly uses the video information of the target speaker (e.g., facial movements and lip shapes) to improve the performance of speech recognition. Although it has high robustness and accuracy, the current multimodal speech recognition technology also shows a significant performance degradation in noisy scenes in the real world. Afouras et al. (Afouras T, Chung JS, Senior A, et al. Deep audio-visual speech recognition. IEEE transactions on pattern analysis and machine intelligence, 2018, 44 (12): 8717-8727) pointed out that the noise level in the audio can directly affect the improvement of speech recognition performance brought by video input such as facial movements or lip shapes. In addition, the current multimodal speech recognition technology requires the video of the target speaker as visual information input. In a natural auditory environment, it is difficult to obtain the video data of the target speaker's facial movements or lip shapes in real time, and the effectiveness of the video is easily affected by factors such as video clarity, shooting angle, and occlusions. Therefore, the current multimodal speech recognition technology also has certain limitations in the use scenario.

[0004] In summary, the speech recognition method in the prior art has the following problems:

[0005] 1. Traditional unimodal speech recognition technology has performance defects in complex noise environments. Although unimodal speech recognition systems can achieve high speech recognition accuracy in simple and quiet environments, the accuracy of speech recognition will be greatly reduced in complex noise environments. Therefore, unimodal speech recognition technology is often only suitable for low-noise speech environments, which greatly reduces its versatility in practical applications.

[0006] 2. At present, multimodal speech recognition technology needs to obtain clear and clean facial movements or lip shape videos of the target speaker as auxiliary visual input to improve the performance of the speech recognition system in complex noisy environments. On the one hand, it is difficult to obtain the facial movements or lip shape videos of the target speaker in real time in a natural auditory environment. On the other hand, even if the accuracy of speech recognition can be improved through the visual input of the target speaker, its performance is still easily affected by complex noise in the natural environment. Therefore, multimodal speech recognition technology also has certain limitations in actual application scenarios. Summary of the invention

[0007] In view of the deficiencies in the prior art, the present invention provides a multimodal speech recognition method, system, device and medium based on visual scenes.

[0008] According to a first aspect of an embodiment of the present invention, a multimodal speech recognition method based on a visual scene is provided, the method comprising:

[0009] Obtain the voice audio and the scene video corresponding to the environment in which the voice audio is located;

[0010] Input the speech audio and the scene video corresponding to the environment in which the speech audio is located into the pre-trained audio-visual feature encoding model, so as to extract the features of the speech audio and the scene video, and obtain the acoustic features and visual features respectively; splice and fuse the acoustic features with the visual features to obtain the audio-visual fusion features;

[0011] The training process of the audio-visual feature coding model includes: obtaining initial acoustic features and initial visual features, and concatenating them to obtain initial audio-visual fusion features; clustering the initial audio-visual fusion features in each time window to obtain cluster labels; randomly selecting the initial audio-visual fusion features of some time windows for masking; using the audio-visual fusion feature vector of the unmasked time window to predict the cluster label of the masked time window, thereby training the audio-visual feature coding model;

[0012] The audio-visual fusion features are input into the fine-tuned speech recognition decoding model to convert them into speech text to obtain the speech recognition results.

[0013] A second aspect of an embodiment of the present invention provides an electronic device, comprising a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-mentioned visual scene-based multimodal speech recognition method.

[0014] According to a third aspect of an embodiment of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the multimodal speech recognition method based on visual scenes is implemented.

[0015] According to a fourth aspect of an embodiment of the present invention, a computer program product is provided, including a computer program / instruction, which implements the above-mentioned visual scene-based multimodal speech recognition method when executed by a processor.

[0016] Compared with the prior art, the present invention has the following beneficial effects:

[0017] (1) The present invention obtains speech audio and a scene video corresponding to the environment in which the speech audio is located, and obtains acoustic clues related to environmental noise through a neural network based on the visual scene information of the environment in which the speech audio is located, thereby effectively enhancing the ability of the speech recognition system to suppress environmental noise and improving the stability and accuracy of speech recognition.

[0018] (2) In the training process of the audio-visual feature encoding model, the present invention clusters the initial audio-visual fusion features in each time window to obtain cluster labels, and clusters the audio-visual fusion features of the scene in a data-driven manner, so that the model learns the cluster labels of the audio-visual fusion features in different auditory scenes during the audio-visual feature encoding process, and realizes the model's pattern recognition of different scenes (types, attributes, locations, movements, etc. of different noise sources), thereby improving the model's speech recognition performance in different noise environments.

[0019] (3) Different from the prior art, the present invention does not need to obtain visual information related to the speaker, such as facial movements, lip shapes, etc. Therefore, the video acquisition device can be placed at any position in the speaker's environment, or directly worn by the speaker, so as to collect scene videos corresponding to the environment where the voice audio is located, with higher degrees of freedom and wider application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0021] Figure 1 is a first schematic diagram of a multimodal speech recognition method based on a visual scene provided by an embodiment of the present invention;

[0022] Figure 2 is a second schematic diagram of a multimodal speech recognition method based on a visual scene provided by an embodiment of the present invention;

[0023] Figure 3 is a schematic diagram of a training process of an audio-visual feature coding model provided by an embodiment of the present invention;

[0024] Figure 4 is an example diagram of the architecture of a multimodal speech recognition model provided by an embodiment of the present invention;

[0025] Figure 5 is a schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] It should be noted that, in the absence of conflict, the features in the following embodiments and implementations may be combined with each other.

[0028] like Figure 1 and Figure 2 As shown, an embodiment of the present invention provides a multimodal speech recognition method based on a visual scene, and the method specifically includes the following steps:

[0029] Step S1, obtaining voice audio and a scene video corresponding to the environment in which the voice audio is located.

[0030] Furthermore, the audio acquisition and recording device should be close to the speaker to collect effective voice audio signals. The video acquisition and recording device can be placed anywhere in the speaker's environment, or directly worn by the speaker, so as to collect visual information of the speaking scene and collect as complete a scene video signal as possible.

[0031] It should be noted that in real auditory scenes, acoustic noise in speech audio usually comes from the background environment, and the environmental noise is related to the properties, position, movement, etc. of objects in the environment. Therefore, the visual scene information of the environment where the speech audio is located can provide acoustic clues related to the environmental noise, thereby effectively enhancing the speech recognition system's ability to suppress environmental noise and improving the stability and accuracy of speech recognition.

[0032] Step S2, input the speech audio and the scene video corresponding to the environment in which the speech audio is located into a pre-trained audio-visual feature encoding model, so as to extract features of the speech audio and scene video to obtain acoustic features and visual features respectively; splice and fuse the acoustic features with the visual features to obtain audio-visual fusion features.

[0033] like Figure 3 and Figure 4 As shown, specifically, the step S2 specifically includes the following steps:

[0034] Step S201, constructing and training an audio-visual feature coding model; the audio-visual feature coding model includes: a feedforward network layer, a residual network layer, a splicing network layer, and a fusion network layer.

[0035] Step S202, dividing the speech audio and the scene video corresponding to the environment in which the speech audio is located according to the time window to obtain a plurality of audio input units and video input units.

[0036] Specifically, the video signal can be segmented into each frame or multiple frames according to the video frame rate as a video input unit. The audio signal can be segmented into time windows according to the duration of each video frame or multiple frames, and an audio signal with the same duration as each video frame is obtained as an audio input unit.

[0037] Exemplarily, the input audio signal and video signal are segmented into time windows to obtain small segments of audio and video signals as input units. The video signal is segmented according to the video frame rate. Since the video frame rate is 30 frames per second, the time window length is 1 / 30 second, and each time window contains 1 frame of the original picture; the audio signal is consistent with the video signal and is segmented according to the 1 / 30 second time window. Since the sampling rate of the audio signal is 48000 Hz, each time window contains 1600 audio signal sampling points.

[0038] Step S203, the Mel-Frequency Cepstral Coefficients (MFCC) of the speech audio signal (i.e., an audio input unit) in each time window is calculated through the Feed-Forward Network (FFN) layer in the audio-visual feature coding model as the acoustic feature a. n , n is the nth time window.

[0039] Step S204, obtaining the visual features v corresponding to the scene video signal in each time window through the residual network layer (Residual Networks, ResNets) in the audio-visual feature coding model n , n is the nth time window.

[0040] Specifically, the RGB value of each pixel point of each frame of scene video image is extracted as the visual feature. When the time window contains multiple frames of video images, the RGB mean value of each pixel point of the multiple frames of video images is extracted as the visual feature.

[0041] Furthermore, the acoustic feature a n With visual features v n The dimensions are the same.

[0042] Step S205, concatenate the acoustic features with the visual features through the concatenation network layer in the audio-visual feature encoding model to obtain the initial audio-visual fusion feature; the expression is as follows:

[0043]

[0044] Step S206, calculating the initial audio-visual fusion features through the fusion network layer in the audio-visual feature encoding model to obtain effective audio-visual fusion features.

[0045] Specifically, in this example, the fusion network layer uses a 24-layer Transformer network, and each Transformer layer consists of a feedforward network layer and a self-attention layer. n When input into a layer of Transformer, the self-attention layer can simultaneously calculate all the audio-visual fusion features x n The attention weights between them are used to obtain the relationship between the auditory features and the visual features in different time windows, so as to update the audiovisual fusion features in the current time window using the audiovisual fusion features in the previous and next time windows to obtain new audiovisual fusion features.

[0046] It should be noted that, through the fusion network, the audiovisual fusion features of the current time window can form a mapping relationship with the audiovisual fusion features of the previous and next time windows, and the relationship between the visual features and the auditory features can be captured by using the temporal correlation of the input features, thereby obtaining more effective audiovisual fusion features.

[0047] Specifically, Figure 3 As shown, the training process of the audio-visual feature encoding model includes:

[0048] S100, obtaining initial acoustic features and initial visual features, and concatenating them to obtain initial audio-visual fusion features.

[0049] S200, clustering the initial audio-visual fusion features in each time window to obtain cluster labels.

[0050] Specifically, first, this embodiment uses the K-means clustering algorithm to cluster the initial audio-visual fusion features x in different time windows. n Perform clustering. Set the number of cluster labels k, and the cluster labels calculated by the clustering algorithm are:

[0051] Z = h(x n )=[z1,···,z n ]

[0052] Among them, h is the clustering algorithm model, z n ∈[C k ], C k is the set of clustering labels, the number of labels is k.

[0053] S300, randomly selecting initial audio-visual fusion features of a part of the time window for masking.

[0054] Secondly, this embodiment randomly selects the initial audio-visual feature vector of the p% time window for masking at the input end of the audio-visual feature encoding model, and assumes that M is the feature subset to be masked by the initial audio-visual feature vector X:

[0055] M=[m1,···,m o]

[0056] Then the subset of masked features removed is

[0057]

[0058] The sum of o and q is the total number of time windows N.

[0059] S400, predicting the clustering label of the masked time window using the audio-visual fusion feature vector of the unmasked time window, thereby training the audio-visual feature encoding model.

[0060] Specifically, this embodiment uses the audio-visual feature vector of the unmasked time window to predict the clustering label of the masked time window, that is, to predict the clustering label of each masked time window in the input end. The probability distribution of , assuming the output of the fusion network layer is:

[0061] Y=[y1,···,y n ]

[0062] Then the probability distribution of the fused features output by each Transformer encoding layer c is:

[0063]

[0064] Among them, A represents the projection matrix of the Transformer encoding layer, e c represents the feature vector of the c-th encoding layer, sim is the cosine similarity, and τ is the hyperparameter.

[0065] According to the above label prediction task, the loss function of the audio-visual feature encoding model can be constructed, where the cross entropy loss function of the masked time window is:

[0066]

[0067] The cross entropy loss function for the unmasked time window is:

[0068]

[0069] The weighted sum of the loss function of the masked time window and the loss function of the unmasked time window is used as the cross entropy loss function of the audio-visual feature encoding model, that is:

[0070] L=αL m +(1-α)L u

[0071] Among them, α is a hyperparameter. The network parameter optimization update of the audio-visual feature encoding model is achieved by minimizing the loss function L.

[0072] It should be noted that the noise forms of different auditory scenes in natural environments are different (including: the type, attributes, location, movement, etc. of the noise source). The present invention clusters the audio-visual fusion features of the scene in a data-driven manner, so that the model learns the clustering labels of the audio-visual fusion features in different auditory scenes during the audio-visual feature encoding process, thereby realizing the model's pattern recognition of different scenes, thereby improving the model's speech recognition performance in different noise environments.

[0073] Step S3, input the audio-visual fusion features into the fine-tuned speech recognition decoding model to convert them into speech text to obtain the speech recognition result.

[0074] Specifically, Figure 4 As shown in the example, the connectionist temporal classification method (Connectionist Temporal Classification, CTC) or the sequence to sequence method (Sequence To Sequence, Seq2Seq) is used to realize the mapping of the audio-visual fusion feature vector to the speech text. The output end of the audio-visual feature encoding model is connected to the input end of the speech recognition encoding model, and the audio-visual fusion feature y is converted into speech text in an autoregressive manner. 1:n The sequence mapped to the target word unit probability is the speech text sequence.

[0075] Specifically, the process of fine-tuning the speech recognition decoding model includes:

[0076] Step S100, inputting speech audio with text labels and scene video corresponding to the environment in which the speech audio is located at the input end of the audio-visual feature coding model;

[0077] Step S200, inputting a text label corresponding to the audio signal at the output end of the speech recognition decoding model;

[0078] Step S300, constructing a loss function through a text conversion task to implement parameter updates of the audio-visual feature encoding model and the speech recognition decoding network.

[0079] The expression of the loss function constructed by the text conversion task is as follows:

[0080]

[0081] In the formula, y 1:n represents the audio-visual fusion feature, w 1:s represents the label of the target word, S represents the number of word labels, and p(w t |w 1:t-1 ,y 1:n ) represents the conditional probability of the speech recognition encoding model converting the audio-visual fusion feature vector into the target word.

[0082] Freeze N% of the network parameters of the audio-visual feature encoding model and use the cross entropy loss function L constructed for the text conversion task s2s The unfrozen network parameters of the audio-visual feature encoding model and all network parameters of the speech recognition decoding model are updated. Preferably, the value range of N is 0% to 100%.

[0083] Example 1

[0084] The following is an example to further illustrate the visual scene-based multimodal speech recognition method proposed by the present invention.

[0085] In this embodiment, in order to demonstrate the versatility of the present invention, a binaural microphone (Soundman OKMII), an Android smartphone (Samsung GALAXY A52), an iOS smartphone (iPhone SE), and a sports camera (DJI Action 2) are used as audio acquisition and recording devices, and the audio sampling frequency is set to 48000 Hz; an Android smartphone (Samsung GALAXY A52), an iOS smartphone (iPhone SE), and a sports camera (DJI Action 2) are used as video acquisition and recording devices, and the video frame rate is set to 30 frames per second; a computing server is equipped for the storage and analysis of audio and video data, and the implementation of the multimodal speech recognition method. The server is equipped with 1 Intel Xeon Silver 4214 GPU, 2 Nvidia V100 GPUs, 256GB DDR4-2666MHz memory, a Crucial 4TB SSD solid state drive, and a Lenovo T24A-10 LCD display.

[0086] Collection using binaural microphones The binaural microphones are worn at the speaker's ears; Android smartphones and iOS smartphones are worn on the left and right chests of the speaker through backpacks; sports cameras are worn on the front of the speaker's neck. This embodiment synchronously collects and records audio and video for 11 audio-visual scenes in daily life, including: streets (2.5 hours), subways (0.8 hours), subway stations (2.1 hours), airports (1.9 hours), buses (2.3 hours), markets (1.7 hours), shopping malls (3.5 hours), canteens (1.6 hours), playgrounds (2.2 hours), parks (9.0 hours), and the wild (3.4 hours), for a total of 31 hours. Among them, 75% of the data were randomly selected as the training set of the multimodal speech recognition model algorithm, 15% of the data were used as the verification set of the multimodal speech recognition model algorithm, and 10% of the data were used as the test set of the multimodal speech recognition model algorithm.

[0087] The specific steps include:

[0088] Step S1, obtaining voice audio and a scene video corresponding to the environment in which the voice audio is located.

[0089] Step S2, input the speech audio and the scene video corresponding to the environment in which the speech audio is located into a pre-trained audio-visual feature encoding model, so as to extract features of the speech audio and scene video to obtain acoustic features and visual features respectively; splice and fuse the acoustic features with the visual features to obtain audio-visual fusion features.

[0090] Among them, the training process of the audio-visual feature encoding model includes:

[0091] Step S100, obtaining initial acoustic features and initial visual features, and concatenating them to obtain initial audio-visual fusion features;

[0092] Step S200, clustering the initial audio-visual fusion features in each time window to obtain cluster labels;

[0093] Furthermore, in this example, during the initial iterative training of the audio-visual feature coding model, the number of cluster labels k is set to 100. Based on the above cluster training process, the audio-visual feature coding model will be iteratively trained for 5 times, and each iteration will use the audio-visual fusion features output by the previous generation of audio-visual feature coding to recalculate new cluster labels for training. The number of cluster labels used in the 5 iterations of audio-visual feature coding are {100, 100, 500, 1000, 2000} respectively.

[0094] Step S300, randomly selecting initial audio-visual fusion features of a part of the time window for masking;

[0095] Step S400, using the audio-visual fusion feature vector of the unmasked time window to predict the clustering label of the masked time window, so as to train the audio-visual feature encoding model;

[0096] Step S3, input the audio-visual fusion features into the fine-tuned speech recognition decoding model to convert them into speech text to obtain the speech recognition result.

[0097] In the embodiment, 50% of the parameters of the audio-visual feature encoding model are frozen, and the cross entropy loss function L constructed by the text conversion task is used. s2s Update the remaining 50% of the network parameters of the audio-visual feature encoding model and all the network parameters of the speech recognition decoding model.

[0098] The multimodal speech recognition model (i.e., audio-visual feature encoding model-speech recognition decoding model) constructed in this embodiment has speech recognition error rates in 11 natural environments in the test set: 9.1% on the street; 7.2% on the subway; 9.0% on the subway station; 8.2% at the airport; 6.5% on the bus; 20.0% on the market; 8.5% at the shopping mall; 16.0% in the cafeteria; 13.8% on the playground; 6.4% in the park; and 4.3% in the wild, with an average error rate of 9.9%. As a control condition, this embodiment uses the same network architecture, masks all video signals, and constructs a unimodal speech recognition model with only unimodal audio input. The speech recognition error rates in 11 natural environments are: 9.6% on the street; 7.6% on the subway; 8.1% on the subway station; 10.8% at the airport; 7.3% on the bus; 22.3% at the market; 9.9% at the mall; 17.2% at the canteen; 16.1% at the playground; 7.9% at the park; 5.6% in the wild, with an average error rate of 11.3%. The error rate of the multimodal speech recognition model is 1.4% lower than that of the unimodal speech recognition model. This embodiment shows that the visual scene information related to audio noise in the natural environment can effectively suppress the noise in the audio, thereby improving the speech recognition performance of the speech recognition system in a complex noise environment.

[0099] Accordingly, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the multimodal speech recognition method based on visual scenes as described above. Figure 5 As shown, it is a hardware structure diagram of any device with data processing capability in which the multimodal speech recognition method based on visual scenes provided by an embodiment of the present invention is located, except Figure 5 In addition to the processor, memory and network interface shown, any device with data processing capability in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capability, which will not be described in detail.

[0100] Accordingly, the present application also provides a computer-readable storage medium on which computer instructions are stored, and when the instructions are executed by a processor, the multimodal speech recognition method based on visual scenes as described above is implemented. The computer-readable storage medium can be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), an SD card, a flash card (Flash Card), etc. equipped on the device. Furthermore, the computer-readable storage medium can also include both an internal storage unit of any device with data processing capabilities and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and can also be used to temporarily store data that has been output or is to be output.

[0101] Those skilled in the art will readily appreciate other embodiments of the present application after considering the description and practicing the contents disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application. The description and examples are intended to be exemplary only.

[0102] It should be understood that the present application is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.

Claims

1. A multimodal speech recognition method based on visual scenes, characterized in that: The method comprises: Obtain the voice audio and the scene video corresponding to the environment in which the voice audio is located; Input the speech audio and the scene video corresponding to the environment in which the speech audio is located into the pre-trained audio-visual feature encoding model, so as to extract the features of the speech audio and the scene video, and obtain the acoustic features and visual features respectively; splice and fuse the acoustic features with the visual features to obtain the audio-visual fusion features; The training process of the audio-visual feature coding model includes: obtaining initial acoustic features and initial visual features, and concatenating them to obtain initial audio-visual fusion features; clustering the initial audio-visual fusion features in each time window to obtain cluster labels; randomly selecting the initial audio-visual fusion features of some time windows for masking; using the audio-visual fusion feature vector of the unmasked time window to predict the cluster label of the masked time window, thereby training the audio-visual feature coding model; The process of randomly selecting the initial audio-visual fusion features of some time windows for masking includes: Let M be the feature subset to be masked by the initial audio-visual feature vector X: ； Then the subset of masked features removed is : ； Among them, the sum of o and q is the total number of time windows N; The process of using the audio-visual fusion feature vector of the unmasked time window to predict the clustering label of the masked time window, thereby training the audio-visual feature encoding model includes: Each masked time window The probability distribution of , assuming the output of the fusion network layer is: ； Then the probability distribution of the fusion features output by each coding layer c is: = ； Among them, A represents the projection matrix of the coding layer, e c represents the feature vector of the cth coding layer, sim is the cosine similarity, is a hyperparameter; The weighted sum of the loss function of the masked time window and the loss function of the unmasked time window is used as the loss function of the audio-visual feature encoding model; The expression of the loss function for the masked time window is as follows: ； In the formula, Indicates the label type of the audio-visual fusion feature in the nth time window; The expression of the loss function for the unmasked time window is as follows: ； Minimize the loss function of the audio-visual feature encoding model, thereby optimizing and updating the network parameters of the audio-visual feature encoding model; The audio-visual fusion features are input into the fine-tuned speech recognition decoding model to convert them into speech text to obtain the speech recognition results.

2. The multimodal speech recognition method based on visual scene according to claim 1, characterized in that: The process of obtaining audio-visual fusion features includes: The audio-visual feature encoding model includes: a feedforward network layer, a residual network layer, a splicing network layer, and a fusion network layer; The speech audio and the scene video corresponding to the environment in which the speech audio is located are divided according to time windows; The Mel-frequency cepstral coefficients of the speech audio in each time window are calculated as acoustic features through the feedforward network layer; The visual features corresponding to the scene video in each time window are obtained through the residual network layer; The acoustic features are concatenated with the visual features through the concatenated network layer to obtain the initial audio-visual fusion features; The initial audio-visual fusion features are calculated through the fusion network layer to obtain effective audio-visual fusion features.

3. The multimodal speech recognition method based on visual scene according to claim 1, characterized in that: The process of fine-tuning the speech recognition decoding model includes: Inputting speech audio with text labels and scene video corresponding to the environment in which the speech audio is located into the input end of the audio-visual feature encoding model; Input the text label corresponding to the audio signal at the output end of the speech recognition decoding model; The loss function is constructed through the text conversion task to realize the parameter update of the audio-visual feature encoding model and the speech recognition decoding network.

4. The multimodal speech recognition method based on visual scene according to claim 1, characterized in that: The expression of the loss function constructed by the text conversion task is as follows: ； In the formula, represents the audio-visual fusion feature, w 1:s represents the label of the target word, S represents the number of word labels, It represents the conditional probability that the speech recognition encoding model transforms the audio-visual fusion feature vector into the target word.

5. An electronic device comprising a memory and a processor, characterized in that: The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the multimodal speech recognition method based on visual scenes as described in any one of claims 1-4 above.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the multimodal speech recognition method based on visual scenes as described in any one of claims 1 to 4 is implemented.

7. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the multimodal speech recognition method based on visual scenes described in any one of claims 1 to 4 is implemented.

Images