Model training method and device, speech recognition method and device and storage medium

Abstract

The invention provides a model training method, a voice recognition method and device and a storage medium, and the method comprises the steps: obtaining a plurality of pieces of voice image pair data, and enabling target voice image pair data in the plurality of pieces of voice image pair data to comprise target voice data and target image data which are consistent in corresponding content; for any target voice image pair data in the multiple pieces of voice image pair data, feature extraction is carried out on the target voice data through a voice feature extraction model, voice features of the target voice data are obtained, and the voice features comprise voice global features and / or voice local features; performing feature extraction on the target image data through a visual feature extraction model to obtain image features of the target image data, the image features including image global features and / or image local features; according to the speech features and the image features, a speech feature extraction model is trained, and the speech feature extraction accuracy and the speech recognition result accuracy are improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence (AI) technology, and in particular to a model training method, a speech recognition method, a device, and a storage medium. Background Art

[0002] At present, speech feature extraction models are widely used in various speech vertical fields such as speech-to-text and illegal speech recognition. For example, the speech to be recognized is usually first extracted based on the speech feature extraction model to obtain the speech features of the speech to be recognized, and then the recognition result corresponding to the speech to be recognized is determined based on the speech features, such as the text content corresponding to the speech to be recognized or whether the speech to be recognized is illegal speech.

[0003] However, since the relevant model training method only uses speech data to train the speech feature extraction model, the accuracy of the extracted speech features is not high. Summary of the invention

[0004] The present application provides a model training method, a speech recognition method, a device and a storage medium, which can improve the accuracy of speech feature extraction and the accuracy of speech recognition results.

[0005] In a first aspect, a model training method is provided, comprising: acquiring a plurality of speech-image pair data, wherein target speech-image pair data in the plurality of speech-image pair data comprises target speech data and target image data with consistent corresponding contents; for any target speech-image pair data in the plurality of speech-image pair data, performing feature extraction on the target speech data through a speech feature extraction model to obtain speech features of the target speech data, wherein the speech features comprise global speech features and / or local speech features; performing feature extraction on the target image data through a visual feature extraction model to obtain image features of the target image data, wherein the image features comprise global image features and / or local image features; and training the speech feature extraction model based on the speech features and the image features.

[0006] In a second aspect, a speech recognition method is provided, comprising: obtaining speech data to be recognized; inputting the speech data to be recognized into a speech feature extraction model trained by the first aspect or the optional method of the first aspect to obtain speech features of the speech data to be recognized, or a speech recognition model to obtain a recognition result of the speech data to be recognized.

[0007] In a third aspect, a model training device is provided, including: an acquisition module, used to: acquire multiple speech image pair data, target speech image pair data in the multiple speech image pair data include: target speech data and target image data with consistent corresponding content; a first feature extraction module, used to extract features of the target speech data through a speech feature extraction model for any target speech image pair data in the multiple speech image pair data, and obtain speech features of the target speech data, the speech features include: global speech features and / or local speech features; a second feature extraction module, used to extract features of the target image data through a visual feature extraction model, and obtain image features of the target image data, the image features include: global image features and / or local image features; a training module, used to train the speech feature extraction model based on the speech features and image features.

[0008] In a fourth aspect, a speech recognition device is provided, comprising: an acquisition module for acquiring speech data to be recognized; and a recognition module for: inputting the speech data to be recognized into a speech feature extraction model trained by the first aspect or the optional method of the first aspect to obtain speech features of the speech data to be recognized, or a speech recognition model to obtain a recognition result of the speech data to be recognized.

[0009] In a fifth aspect, an electronic device is provided, comprising: a processor and a memory, the memory being used to store a computer program, the processor being used to call and run the computer program stored in the memory to execute the method described in the first aspect or the second aspect.

[0010] In a sixth aspect, a computer-readable storage medium is provided for storing a computer program, wherein the computer program enables a computer to execute the method described in the first aspect or the second aspect.

[0011] In summary, the present application uses a visual feature extraction model to extract features from image data that is consistent with the content of speech data, obtains image features, and uses the image features and the speech features obtained by extracting speech data using a speech feature extraction model to train a speech feature extraction model. Since speech data and image data (including image data that is consistent with or inconsistent with the content of speech data) are comprehensively considered, the speech feature extraction model can understand the speech data more accurately and output a more accurate feature representation, which helps to improve the prediction accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for describing the embodiments are briefly introduced below.

[0013] Figure 1 A flowchart of a model training method provided in an embodiment of the present application;

[0014] Figure 2 A schematic diagram of a model training method provided in an embodiment of the present application;

[0015] Figure 3 A schematic diagram of another model training method provided in an embodiment of the present application;

[0016] Figure 4 A flow chart of a speech recognition method provided in an embodiment of the present application;

[0017] Figure 5 A schematic diagram of a model training device 500 provided in an embodiment of the present application;

[0018] Figure 6 A schematic diagram of a speech recognition device 600 provided in an embodiment of the present application;

[0019] Figure 7 It is a schematic diagram of an electronic device 700 provided in an embodiment of the present application. DETAILED DESCRIPTION

[0020] The technical solution of the present application will be described below in conjunction with the accompanying drawings in the embodiments of the present application.

[0021] It should be noted that the information, data (including, but not limited to: data for analysis, data for storage, data for display, etc., such as individual images, voice data, and voice recognition tasks, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions. For example, the voice-image pair data and the operations on the voice-image pair data involved in this application are all obtained with full authorization.

[0022] In one embodiment, the technical solution of the present application can be used in scenarios in the vertical field of voice, specifically, in voice recognition scenarios, voice content security detection scenarios, voice-to-text scenarios, etc., but not limited thereto.

[0023] Among them, the speech feature extraction model trained by the technical solution of this application can output the speech features of speech data; the speech recognition model trained based on the speech feature extraction model and other technical solutions can output the recognition results of speech data, which will be introduced in subsequent embodiments.

[0024] Exemplarily, the technical solution of the present application can be applied to the scenario of identifying whether a voice involves specific content, or to the scenario of identifying whether a voice is a specific voice, wherein the specific voice can refer to a voice containing specific content. For example, the specific content can refer to sensitive, disharmonious or unhealthy content such as pornography and violence; the specific voice can refer to sensitive voice, disharmonious voice or unhealthy voice, and the present application does not limit this.

[0025] Exemplarily, the technical solution of the present application can be applied to a scenario of identifying whether a voice is that of a specific person.

[0026] Exemplarily, the technical solution of the present application can be applied to a scenario of converting speech into text in a specific language.

[0027] Similarly, the technical solution of the present application can also be applied to the field of image detection or recognition. Specifically, the technical solution of the present application can be combined with the visual feature extraction model based on speech features and image features. Specifically, when training the visual feature extraction model based on speech features and image features, the speech features can be replaced with image features and the image features can be replaced with speech features, that is, a training method similar to that for the speech feature extraction model is used to train the visual feature extraction model in combination with speech features.

[0028] In one embodiment, the solution provided by the present application can be executed by any electronic device with data processing capabilities. For example, the electronic device can be a server, specifically an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services. For another example, the electronic device can be a terminal device, specifically a tablet computer, a laptop computer, or a desktop computer. For another example, the electronic device can be a combination of a server and a terminal device, wherein the server and the terminal device in the combination can communicate wirelessly or wired, and the present application does not impose specific restrictions on the electronic device.

[0029] Figure 1 A flowchart of a model training method provided in an embodiment of the present application, which method can be executed by the electronic device in the above embodiment, such as Figure 1 As shown, the method includes:

[0030] S110: Acquire a plurality of voice-image pair data, wherein target voice-image pair data in the plurality of voice-image pair data includes: target voice data and target image data with consistent corresponding contents;

[0031] S120: for any target speech image pair data among the multiple speech image pair data, extract features of the target speech data by using a speech feature extraction model to obtain speech features of the target speech data, where the speech features include: speech global features and / or speech local features;

[0032] S130: extracting features from the target image data using a visual feature extraction model to obtain image features of the target image data, where the image features include: global image features and / or local image features;

[0033] S140: Training a speech feature extraction model based on speech features and image features.

[0034] It should be noted that the speech-image pair data in this application refers to a pair of speech data and image data with consistent corresponding content. The speech data and image data with consistent corresponding content can be understood as: the content represented by the speech data is consistent with the content represented by the image data, or the speech data is used to describe the image data, or the image content displayed in the image data is consistent with the information conveyed by the speech data.

[0035] For example, the image data is an image of a group of people gathered next to a swimming pool, and the voice data consistent with the content corresponding to the image data may be the voice "a group of people gathered next to a swimming pool".

[0036] In one embodiment, the scenarios or fields involved in the multiple voice-image pair data may be different, and this application does not impose any limitation on this.

[0037] Exemplarily, the multiple voice-image pair data include: voice-image pair data 1 and voice-image pair data 2, the voice data 1 and image data 1 included in the voice-image pair data 1 may respectively be: the commentary voice of the news anchor and the news screen image in a certain frame in the news playback interface; the voice data 2 and image data 2 included in the voice-image pair data 2 may respectively be: the product introduction voice of product 1 and the product image of product 1 in the advertising playback page of product 1.

[0038] In one embodiment, the input of the speech feature extraction model (speech encoder) may be the target speech data, and the output may be the speech features of the target speech data. The speech features may include at least one of the global speech features and the local speech features. Alternatively, the speech features may be features obtained by concatenating the global speech features and the local speech features, for example, features obtained by concatenating the global speech features and the local speech features head to tail or weighted concatenation.

[0039] Exemplarily, the global speech feature can be at least one of a vector corresponding to the average energy of the entire speech segment of the target speech data, a vector corresponding to the fundamental frequency, a vector corresponding to the spectral envelope, a Mel Frequency Cepstrum Coefficient (MFCC) feature, log-mel, and linear predictive coding (LPC).

[0040] Similarly, the local feature of speech can be at least one of the vector corresponding to the average energy of each speech segment in the target speech data, the vector corresponding to the fundamental frequency, the vector corresponding to the spectral envelope, MFCC features, logarithmic Merp, and LPC.

[0041] In one embodiment, the input of the visual feature extraction model (Image encoder) may be the target image data, and the output may be the image features of the target image data. The image features may include at least one of the global image features and the local image features. Alternatively, the image features may be features obtained by splicing the global image features and the local image features, for example, features obtained by splicing the global image features and the local image features head to tail or weighted splicing.

[0042] Exemplarily, the global features of an image may be at least one of the color features of the entire image of the target image data (specifically, they may be: vectors corresponding to the distribution of each color in the image), texture features (specifically, they may be: grayscale co-occurrence matrices), shape features (specifically, they may be: edge features and contour features of each object in the image), and spatial relationship features (specifically, they may be: vectors or matrices corresponding to the relative positional relationships between different objects or regions in the image).

[0043] Similarly, the local features of the image can be at least one of the color features of each image block in the target image data (specifically, they can be: the vectors corresponding to the color distributions in the image blocks), texture features (specifically, they can be: the grayscale co-occurrence matrix of the image blocks), shape features (specifically, they can be: the edge features and contour features of each object in the image blocks), and spatial relationship features (specifically, they can be: the vectors or matrices corresponding to the relative positional relationships between different objects or regions in the image blocks).

[0044] It should be noted that the present application does not limit the specific contents of the voice features and image features and the execution order of S120 and S130.

[0045] In one embodiment, Figure 2As shown in (c) in FIG. 1 , the speech feature extraction model may include: a Hubert (Hidden Unit BERT) module and a transformer module. When training the speech feature extraction model, the parameters corresponding to the Hubert module may be set to Fixed parameters, so that only the parameters corresponding to the transformer module are adjusted according to the loss, thereby reducing the processing workload.

[0046] Among them, the Hubert module can extract features of the target speech data to obtain the speech intermediate representation of the target speech data; the transformer module can further process the speech intermediate representation, capture the long-distance dependency between the elements in the speech intermediate representation, and obtain the speech features of the target speech data.

[0047] Exemplarily, the Hubert module may include: a feature extractor, an intermediate encoder and a final encoder. The feature extractor may include a multi-layer one-dimensional convolutional network for extracting primary feature representations from target speech data; the intermediate encoder includes multiple encoder layers, each of which includes a self-attention mechanism and a feedforward neural network for obtaining the initial feature representation output by the feature extractor and generating a high-level speech representation; the final encoder includes a discrete content encoder and a soft content encoder, the discrete content encoder can divide the high-level speech representation into a series of discrete speech units through a clustering algorithm, and the soft content encoder can predict the probability distribution of discrete speech units, and select part of the high-level speech representation from the high-level speech representation according to the probability distribution to obtain the speech intermediate representation of the target speech data.

[0048] Exemplarily, the transformer module may include several layers of transformer blocks, wherein each transformer block may include an attention mechanism and a feedforward neural network, and each layer of transformer blocks may be connected through residual connection and layer normalization. The attention mechanism may combine each element in the intermediate representation of speech and the weight of each element to output the initial features of the intermediate representation of speech; the feedforward neural network may include at least one fully connected layer, which may map the initial features of the intermediate representation of speech to a preset feature space, and the feedforward neural network of the last layer of transformer block may output the speech features of the target speech data.

[0049] In one embodiment, Figure 2As shown in (c) in FIG. 5 , the visual feature extraction model can be a model based on multimodal vision and text learning (Contrastive Language–Image Pre-training, CLIP).

[0050] Exemplarily, the visual feature extraction model may include: an image encoder, a picture-text converter, a text encoder, and a picture-text mixing module. Among them, the image encoder may include a convolutional neural network (CNN) or a Transformer-based visual model, which can encode the target image data to obtain initial image features. The picture-text converter may include a recurrent neural network for converting the target image data into target text data consistent with the corresponding content of the target image data. The text encoder may be a BERT (Bidi rectional Encoder Representations from Transformers) model, which is used to encode the target text data to obtain initial text features. The picture-text mixing module can process the initial image features by feature splicing or feature fusion in combination with the initial text features to obtain image features of the target image data.

[0051] The following introduces the process of training the speech feature extraction model according to the speech features and the image features, that is, S140 .

[0052] In one embodiment, the speech feature extraction model may be trained according to the speech features and the image features through the following steps:

[0053] S140-1: performing noise processing on the image features to obtain noisy image features;

[0054] S140-2: De-noising the noisy image features according to the speech features to obtain de-noised image features;

[0055] S140-3: Determine a first loss according to the denoised image features and the speech features;

[0056] S140-4: Determine a second loss according to the image feature and the speech feature;

[0057] S140-5: Train a speech feature extraction model based on the first loss and / or the second loss.

[0058] For S140-1, in one embodiment, the image features can be denoised by the following steps to obtain noisy image features: determining some image features in the image features; determining the noise value to determine the noisy image features; adding the noise value to some image features in the image features, or replacing some image features in the image features with the noisy image features to obtain the noisy image features.

[0059] Exemplarily, partial features (patch, feature block) can be randomly selected from the image features to obtain partial image features; or, features corresponding to the background part in the target image data can be determined as the above-mentioned partial image features, but it is not limited thereto.

[0060] Exemplarily, the noise value may be a noise value of any noise type, such as Gaussian Noise and Salt and Pepper Noise, but is not limited thereto.

[0061] Exemplarily, the noise image features may be partial features of other image features in multiple speech image pair data except the target image features, or may be a randomly generated noise matrix having the same size as partial image features in the image features, but is not limited thereto.

[0062] Exemplarily, the adding of the noise value to the partial image features in the image features may be to add the numerical value corresponding to the noise value and the matrix or vector corresponding to the partial image features.

[0063] For S140-2, in one embodiment, the noisy image features can be denoised according to the speech features to obtain the denoised image features through the following steps: the speech features and the noisy image features are weighted by the attention mechanism to obtain the attention representation corresponding to the noisy image features; the attention representation is processed through a fully connected layer (Fully Connected Layer, FC) to obtain the integrated features corresponding to the attention representation; the attention representation and the integrated features are processed through a normalization layer (Layer norm) to obtain the denoised image features.

[0064] Specifically, Figure 2 As shown in (c), the pooling layer (Att.pooling), fully connected layer and normalization layer based on the attention mechanism can be connected to obtain a denoising model (also called a fusion encoder); the denoising model denoises the noisy image features according to the speech features to obtain denoised image features. Among them, the pooling layer based on the attention mechanism can perform attention mechanism weighting on the speech features and the noisy image features to obtain the attention representation corresponding to the noisy image features.

[0065] For example, the speech features and the noisy image features can be weighted by the attention mechanism through the scaled dot product attention to obtain the attention representation corresponding to the noisy image features, as shown in formula (1). * |s) represents the attention representation, I *refers to the noisy image feature, √D refers to the scaling factor, Q s It refers to the query corresponding to the voice feature. refers to the key (Ke y) corresponding to the noisy image feature, V I* is the value matrix, which contains the vectors used for weighted summation according to the attention weights.

[0066]

[0067] Then, the denoised image features can be obtained by formula (2). * |s)) represents the integrated features corresponding to the attention representation obtained by processing the attention representation through the fully connected layer; LN(FC(Attn(I * |s))+Attn(I * |s)), that is, F I′ , represents the denoised image features obtained by processing the attention representation and integrated features through the normalization layer.

[0068] F I′ =LN(FC(Attn(I * |s))+Attn(I * |s)) Formula (2)

[0069] In the above process, by adding noise to the image features, we can simulate various interferences that the image may suffer in the real environment, increase the diversity of training data, and thus improve the robustness and generalization ability of the model; by denoising the noisy image features through speech features, the model can fully explore the intrinsic connection and complementarity between image data and speech data, promote the model's deep understanding and fusion of multimodal information, and improve the generalization ability and performance of the model.

[0070] With respect to S140-3 and S140-4, in one embodiment, the first loss and the second loss may be determined by the following steps:

[0071] Among them, this application does not limit the order of S140-3 and S140-4, step two and step three.

[0072] Step 1: for target image data, determine a first actual matching vector between the target image data and speech data in multiple speech-image pair data; for target speech data, determine a second actual matching vector between the target speech data and image data in multiple speech-image pair data.

[0073] Exemplarily, One-Hot encoding may be used to determine the first actual matching vector and the second actual matching vector.

[0074] For example, assuming that the multiple voice-image pair data include B voice-image pair data, B is a positive integer, and the voice data that is consistent with the corresponding content of the target image data is the target voice data, then it can be determined that the first actual matching vector is a B-dimensional vector, and the value of the dimension corresponding to the target voice data in the B-dimensional vector is a first value (for example, 1), and the value of the dimension corresponding to the voice data other than the target voice data in the multiple voice-image pair data is a second value (for example, 0), and the second value is different from the first value. Similarly, it can be determined that the second actual matching vector is a B-dimensional vector, and the value of the dimension corresponding to the target image data in the B-dimensional vector is a first value (for example, 1), and the value of the dimension corresponding to the image data other than the target image data in the multiple voice-image pair data is a second value (for example, 0), and the second value is different from the first value.

[0075] Step 2: For the denoised image features, determine the similarity value of the denoised image features to the speech features to obtain the first image-to-sound similarity; for the speech features, determine the similarity value of the speech features to the denoised image features to obtain the first sound-to-image similarity; determine the first sub-loss based on the first actual matching vector and the first image-to-sound similarity, and determine the second sub-loss based on the second actual matching vector and the first sound-to-image similarity; determine the first loss based on the first sub-loss and the second sub-loss.

[0076] Exemplarily, the above-mentioned determining the similarity value of the denoised image feature to the speech feature to obtain the first image-to-sound similarity may include: determining the similarity value of the denoised image feature to the speech feature based on the speech features of each speech data in multiple speech image pair data to obtain the first image-to-sound similarity.

[0077] For example, as shown in formula (3), for the denoised image feature, the first sub-image-sound similarity between the denoised image feature and the speech feature is calculated, the sum of the similarities between the denoised image feature and the speech feature of each speech data in the multiple speech image pair data is determined to obtain the second sub-image-sound similarity, and the ratio of the first sub-image-sound similarity to the second sub-image-sound similarity is calculated to obtain the first image-sound similarity. represents the first image-sound similarity; I′ represents the denoised image feature; S represents the speech feature, i.e., the speech feature of the target speech data; j=1, ..., B; Sj represents the speech feature of the jth speech data in the B speech-image pair data; exp(s(I′, S) / t) represents the first sub-image-sound similarity; Indicates the sound similarity of the second sub-graph.

[0078]

[0079] Exemplarily, the above-mentioned method of determining the similarity value of the speech feature to the denoised image feature for the speech feature to obtain the first sound-image similarity may include: determining the similarity value of the speech feature to the denoised image feature based on the denoised image feature of each image data in multiple speech image pair data for the speech feature to obtain the first sound-image similarity.

[0080] For example, as shown in formula (4), for the speech feature, the first consonant graph similarity between the speech feature and the denoised image feature is calculated, the sum of the similarities between the speech feature and the denoised image feature of each image data in the plurality of speech image pair data is determined to obtain the second consonant graph similarity, and the ratio of the first consonant graph similarity to the second consonant graph similarity is calculated to obtain the first consonant graph similarity. represents the first sound-image similarity; I′ represents the denoised image feature; S represents the speech feature, i.e., the speech feature of the target speech data; I′j represents the denoised image feature of the jth image data in the B speech-image pair data; exp(s(S, I′) / t) represents the first consonant-image similarity; Represents the second consonant graph similarity.

[0081]

[0082] Exemplarily, the first sub-loss, the second sub-loss and the first loss can be calculated according to formula (5). s2i (S) represents the second actual matching vector; y i2s (I) represents the first actual matching vector; H(y s2i (S), ) represents the second sub-loss, which can be the cross entropy loss; H(y i2s (I) represents the first sub-loss, which may be specifically the cross entropy loss; Lcmd represents the first loss, which may be specifically the average of the first sub-loss and the second sub-loss, and of course, may also be the maximum or minimum value thereof, which is not limited in the present application.

[0083]

[0084] Step three: for image features, determine the similarity value of image features relative to speech features to obtain a second image-sound similarity; for speech features, determine the similarity value of speech features relative to image features to obtain a second sound-image similarity; determine the third sub-loss based on the first actual matching vector and the second image-sound similarity, and determine the fourth sub-loss based on the second actual matching vector and the second sound-image similarity; determine the second loss based on the third sub-loss and the fourth sub-loss.

[0085] Exemplarily, the above-mentioned determining the similarity value of the image feature relative to the speech feature to obtain the second image-sound similarity may include: determining the similarity value of the image feature relative to the speech feature based on the speech features of each speech data in multiple speech-image pair data to obtain the second image-sound similarity.

[0086] For example, as shown in formula (6), for the image feature, the third sub-image-sound similarity between the image feature and the speech feature is calculated, the sum of the similarities between the image feature and the speech feature of each speech data in the multiple speech-image pair data is determined to obtain the fourth sub-image-sound similarity, and the ratio of the third sub-image-sound similarity to the fourth sub-image-sound similarity is calculated to obtain the second image-sound similarity. represents the second image-sound similarity; I represents image features; S represents speech features, i.e., speech features of the target speech data; Sj represents the speech features of the jth speech data in the B speech-image pair data; exp(s(I, S) / t) represents the third sub-image-sound similarity; Indicates the sound similarity of the fourth sub-graph.

[0087]

[0088] Exemplarily, the above-mentioned method of determining the similarity value of the speech feature relative to the image feature for the speech feature to obtain the second sound-image similarity may include: determining the similarity value of the speech feature relative to the denoised image feature based on the image features of each image data in multiple speech-image pair data for the speech feature to obtain the second sound-image similarity.

[0089] For example, as shown in formula (7), for the speech feature, the third consonant graph similarity of the speech feature and the image feature is calculated, the sum of the similarities between the speech feature and the image feature of each image data in the plurality of speech image pair data is determined, the fourth consonant graph similarity is obtained, and the ratio of the third consonant graph similarity to the fourth consonant graph similarity is calculated to obtain the second consonant graph similarity. represents the second sound-image similarity; I represents the image feature; S represents the speech feature, i.e., the speech feature of the target speech data; Ij represents the image feature of the jth image data in the B speech-image pair data; exp(s(S, I) / t) represents the third consonant-image similarity; Indicates the similarity of the fourth consonant graph.

[0090]

[0091] For example, the third sub-loss, the fourth sub-loss and the second loss can be calculated according to formula (8). s2i (S), ) represents the fourth sub-loss, which can be specifically the cross entropy loss; H(yi2s (I) represents the third sub-loss, which may be specifically the cross entropy loss; Lsic represents the second loss, which may be specifically the average of the third sub-loss and the fourth sub-loss, and of course, may also be the maximum or minimum value thereof, which is not limited in this application.

[0092]

[0093] For S140-5, as shown in formula (9), a balance factor can be determined; the product of the balance factor and the first loss is calculated to obtain a weighted first loss; the sum of the weighted first loss and the second loss is calculated to obtain a third loss; and the speech feature extraction model is trained according to the third loss. Wherein, a represents a balance factor, which is used to balance the magnitude of the first loss and the second loss, for example, it can be 0.1; and L represents the third loss.

[0094] L=Lsic+a*Lcmd Formula (9)

[0095] Exemplarily, when training the speech feature extraction model according to the third loss, back propagation (BP) can be combined to allow matching image features or denoised image features and speech features to be close to each other in the feature space, while mismatched ones are far away from each other in the feature space, wherein the image data and speech data corresponding to the features are consistent in content and are considered to match each other, otherwise they are considered to be mismatched.

[0096] Exemplarily, the training of the speech feature extraction model can be completed when the third loss is less than a preset loss; or, the training of the speech feature extraction model can be completed when the number of training times is greater than a preset number of times.

[0097] In the above process, when training the speech feature extraction model, not only the image features after denoising and denoising are considered, but also the unprocessed image features are considered at the same time; moreover, when calculating the loss, the loss of image features or denoised image features relative to speech features, and the loss of speech features relative to image features or denoised image features are considered respectively. This can not only improve the robustness and generalization ability of the model, but also facilitate the model to fully explore the intrinsic connection and complementarity between image data and speech data, that is, multimodal data, promote the model's deep understanding and fusion of multimodal information, and improve the model's feature extraction capabilities. In addition, since the above process is a self-supervised learning task, the performance of the model can be improved without introducing additional annotation costs.

[0098] In one embodiment, in combination with the above, Figure 2As shown in (a), after obtaining the speech features and image features, two tasks can be used to align the speech features and image features or denoised image features in the feature space to complete the training of the speech feature extraction model. Among them, the first task is cross-modality denoise (CMD), corresponding to S140-1 to S140-3, and the corresponding loss is the first loss; the second task is speech-image contrast (SIC), corresponding to S140-4, and the corresponding loss is the second loss. Specifically, as Figure 2 As shown in (b), through the first task, the denoised image features are determined in combination with the speech features, and the first loss is calculated based on the denoised image features and the speech features; through the second task, the second loss is calculated based on the image features and the speech features; finally, the speech feature extraction model can be trained based on BP according to the first loss and the second loss.

[0099] In one embodiment, after obtaining the speech feature extraction model, the sample speech data corresponding to the sample speech recognition task and the language label corresponding to the sample speech data can also be obtained; based on the sample speech data and the language label, the trained speech feature extraction model is fine-tuned to obtain a speech recognition model; wherein the speech recognition model is used to recognize the sample speech data to obtain the recognition result of the sample speech recognition task.

[0100] Exemplarily, the sample speech recognition task can be any of the following tasks, but is not limited to: sensitive audio recognition task, human interactive chat recognition task, voice signature recognition task, news content recognition task, advertising content recognition task, game speech recognition task, song recognition task.

[0101] Exemplarily, the language label refers to the label of the language corresponding to the sample speech data, and the language may be Mandarin, dialect, etc.

[0102] For example, Figure 3 As shown, the speech feature extraction model can be fine-tuned in combination with the prompt data, wherein the prompt data is used to indicate the output content or form of the prompt speech feature extraction model. Alternatively, the speech feature extraction model can be fine-tuned by using a fine-tuning method such as Lora, so that the obtained speech recognition model can output recognition results of different sample speech recognition tasks.

[0103] The following is an introduction to the speech recognition method involved in this application.

[0104] Figure 4 A flow chart of a speech recognition method provided in an embodiment of the present application, which method can be executed by the electronic device in the above embodiment, such as Figure 4 As shown, the method includes:

[0105] S410: Acquire voice data to be recognized;

[0106] S420: Inputting the speech data to be recognized into a speech feature extraction model to obtain speech features of the speech data to be recognized;

[0107] S430: Input the speech data to be recognized into the speech recognition model to obtain the recognition result of the speech data to be recognized.

[0108] It should be understood that the speech feature extraction model and the speech recognition model can be trained by the above-mentioned model training method, and the contents and effects involved in the speech recognition method can refer to the contents and effects involved in the above-mentioned model training method. To avoid repetition, this application does not elaborate on this. In addition, since the speech feature extraction model and the speech recognition model obtained by the above-mentioned training method have higher accuracy in feature extraction and recognition of speech, based on this, this speech recognition method can realize speech recognition more accurately.

[0109] It should be noted that all the above technical solutions can be combined in any way to form optional embodiments of the present application, which will not be described one by one here.

[0110] Figure 5 A schematic diagram of a model training device 500 provided in an embodiment of the present application is shown as follows: Figure 5 As shown, the device 500 includes: an acquisition module 501, a first feature extraction module 502, a second feature extraction module 503, a training module 504, a task acquisition module 505, and a fine-tuning module 506.

[0111] In one embodiment, an acquisition module 501 is used to: acquire multiple speech-image pair data, wherein target speech-image pair data in the multiple speech-image pair data include target speech data and target image data with consistent corresponding contents; a first feature extraction module 502 is used to, for any target speech-image pair data in the multiple speech-image pair data, perform feature extraction on the target speech data through a speech feature extraction model to obtain speech features of the target speech data, wherein the speech features include global speech features and / or local speech features; a second feature extraction module 503 is used to perform feature extraction on the target image data through a visual feature extraction model to obtain image features of the target image data, wherein the image features include global image features and / or local image features; a training module 504 is used to train a speech feature extraction model based on speech features and image features.

[0112] Exemplarily, the training module 504 is specifically used to: perform noise processing on image features to obtain noisy image features; perform denoising processing on the noisy image features according to speech features to obtain denoised image features; determine a first loss according to the denoised image features and the speech features; determine a second loss according to the image features and the speech features; and train a speech feature extraction model according to the first loss and / or the second loss.

[0113] Exemplarily, the training module 504 is specifically used to: determine part of the image features in the image features; determine the noise value and determine the noise image features; add the noise value to part of the image features in the image features, or use the noise image features to replace part of the image features in the image features to obtain the noisy image features.

[0114] Exemplarily, the training module 504 is specifically used to: perform attention mechanism weighting on speech features and noisy image features to obtain attention representations corresponding to noisy image features; process the attention representations through a fully connected layer to obtain integrated features corresponding to the attention representations; and process the attention representations and integrated features through a normalization layer to obtain denoised image features.

[0115] Exemplarily, the training module 504 is specifically used to: determine, for target image data, a first actual matching vector between the target image data and the speech data in a plurality of speech-image pair data; determine, for target speech data, a second actual matching vector between the target speech data and the image data in a plurality of speech-image pair data; determine, for denoised image features, a numerical value of the similarity between the denoised image features and the speech features to obtain a first image-sound similarity; determine, for speech features, a numerical value of the similarity between the speech features and the denoised image features to obtain a first sound-image similarity; determine, based on the first actual matching vector and the first image-sound similarity, and determine, based on the second actual matching vector and the first sound-image similarity, a first sub-loss; determine, based on the first sub-loss and the second sub-loss, a first loss; determine, for image features, a numerical value of the similarity between the image features and the speech features to obtain a second image-sound similarity; determine, for speech features, a numerical value of the similarity between the speech features and the image features to obtain a second sound-image similarity; determine, based on the first actual matching vector and the second image-sound similarity, a fourth sub-loss; determine, based on the third sub-loss and the fourth sub-loss.

[0116] Exemplarily, the training module 504 is specifically used to: for denoised image features, based on the speech features of each speech data in multiple speech image pair data, determine the similarity value of the denoised image features to the speech features, and obtain a first image-to-sound similarity; for speech features, based on the denoised image features of each image data in multiple speech image pair data, determine the similarity value of the speech features to the denoised image features, and obtain a first sound-to-image similarity.

[0117] Exemplarily, the training module 504 is specifically used to: for denoised image features, calculate the first sub-graph sound similarity between the denoised image features and the speech features, determine the sum of the similarities between the denoised image features and the speech features of each speech data in multiple speech image pair data to obtain the second sub-graph sound similarity, calculate the ratio of the first sub-graph sound similarity to the second sub-graph sound similarity to obtain the first graph-sound similarity; for speech features, calculate the first sub-sound graph similarity between the speech features and the denoised image features, determine the sum of the similarities between the speech features and the denoised image features of each image data in multiple speech image pair data to obtain the second sub-sound graph similarity, calculate the ratio of the first sub-sound graph similarity to the second sub-sound graph similarity to obtain the first sound-graph similarity.

[0118] Exemplarily, the training module 504 is specifically used to: for image features, based on the speech features of each speech data in multiple speech image pair data, determine the similarity value of the image features to the speech features, and obtain a second image-to-sound similarity; for speech features, based on the image features of each image data in multiple speech image pair data, determine the similarity value of the speech features to the denoised image features, and obtain a second sound-to-image similarity.

[0119] Exemplarily, the training module 504 is specifically used to: for image features, calculate the third sub-graph-sound similarity between the image features and the speech features, determine the sum of the similarities between the image features and the speech features of each speech data in multiple speech-image pair data, obtain the fourth sub-graph-sound similarity, calculate the ratio of the third sub-graph-sound similarity to the fourth sub-graph-sound similarity, and obtain the second image-sound similarity; for speech features, calculate the third sub-sound-graph similarity between the speech features and the image features, determine the sum of the similarities between the speech features and the image features of each image data in multiple speech-image pair data, obtain the fourth sub-sound-graph similarity, calculate the ratio of the third sub-sound-graph similarity to the fourth sub-sound-graph similarity, and obtain the second sound-graph similarity.

[0120] Exemplarily, the training module 504 is specifically used to: determine a balance factor; calculate the product of the balance factor and the first loss to obtain a weighted first loss; calculate the sum of the weighted first loss and the second loss to obtain a third loss; and train a speech feature extraction model based on the third loss.

[0121] Exemplarily, the task acquisition module 505 is used to: obtain sample speech data corresponding to the sample speech recognition task and the language label corresponding to the sample speech data; the fine-tuning module 506 is used to fine-tune the trained speech feature extraction model based on the sample speech data and the language label to obtain a speech recognition model; wherein the speech recognition model is used to recognize the sample speech data to obtain the recognition result of the sample speech recognition task.

[0122] Figure 6 A schematic diagram of a speech recognition device 600 provided in an embodiment of the present application is shown as follows: Figure 6 As shown, the device 600 includes: an acquisition module 601 and an identification module 602.

[0123] In one embodiment, the acquisition module 601 is used to acquire speech data to be recognized; the recognition module 602 is used to: input the speech data to be recognized into a speech feature extraction model to obtain speech features of the speech data to be recognized; or input the speech data to be recognized into a speech recognition model to obtain recognition results of the speech data to be recognized.

[0124] It should be understood that the device embodiment and the method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, they will not be described here.

[0125] The above describes the apparatus 600 and 500 of the embodiment of the present application from the perspective of the functional module in conjunction with the accompanying drawings. It should be understood that the functional module can be implemented in hardware form, can be implemented by instructions in software form, and can also be implemented by a combination of hardware and software modules. Specifically, the steps of the method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and / or software form instructions in the processor, and the steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or a combination of hardware and software modules in the decoding processor to perform. Optionally, the software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory, and completes the steps in the above-mentioned model training method or speech recognition method embodiment in conjunction with its hardware.

[0126] Figure 7 Schematic diagram of an electronic device 700 provided in an embodiment of the present application. Figure 7 As shown, the electronic device 700 may include: a memory 710 and a processor 720, wherein the memory 710 is used to store a computer program and transmit the program code to the processor 720. In other words, the processor 720 may call and run the computer program from the memory 710 to implement the method in the embodiment of the present application.

[0127] For example, the processor 720 may be configured to execute the above method embodiments according to instructions in the computer program.

[0128] In some embodiments of the present application, the processor 720 may include but is not limited to:

[0129] General-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, etc.

[0130] In some embodiments of the present application, the memory 710 includes but is not limited to:

[0131] Volatile memory and / or non-volatile memory. Among them, the non-volatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM) or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLD RAM) and direct memory bus random access memory (DR RAM).

[0132] In some embodiments of the present application, the computer program may be divided into one or more modules, which are stored in the memory 710 and executed by the processor 720 to complete the method provided by the present application. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device.

[0133] like Figure 7 As shown, the electronic device may also include:

[0134] The transceiver 730 may be connected to the processor 720 or the memory 710 .

[0135] The processor 720 may control the transceiver 730 to communicate with other devices, specifically, to send information or data to other devices, or to receive information or data sent by other devices. The transceiver 730 may include a transmitter and a receiver. The transceiver 730 may further include an antenna, and the number of antennas may be one or more.

[0136] It should be understood that the various components in the electronic device are connected via a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus and a status signal bus.

[0137] The present application also provides a computer storage medium on which a computer program is stored, and when the computer program is executed by a computer, the computer can perform the method of the above method embodiment. In other words, the present application embodiment also provides a computer program product containing instructions, and when the instructions are executed by a computer, the computer can perform the method of the above method embodiment.

[0138] When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instruction is loaded and executed on a computer, the computer can be made to perform the corresponding flow in each method in the embodiment of the present application in whole or in part, and generate the functions that can be realized by each method in the embodiment of the present application. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instruction can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instruction can be transmitted from a website site, computer, server or data center by wired (such as coaxial cable, optical fiber, digital subscriber line (Digital Subscriber Line, DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server, a data center, etc. that contains one or more available media integrations. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital video disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0139] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0140] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the system, device or module can be electrical, mechanical or other forms.

[0141] The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. For example, each functional module in each embodiment of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

Claims

1. A model training method, characterized in that: include: Acquire a plurality of voice-image pair data, wherein target voice-image pair data in the plurality of voice-image pair data includes: target voice data and target image data with consistent corresponding contents; For any target speech image pair data among the plurality of speech image pair data, extracting features from the target speech data by using a speech feature extraction model to obtain speech features of the target speech data, wherein the speech features include: speech global features and / or speech local features; Extracting features from the target image data using a visual feature extraction model to obtain image features of the target image data, wherein the image features include: global image features and / or local image features; The speech feature extraction model is trained according to the speech features and the image features.

2. The method according to claim 1, characterized in that The step of training the speech feature extraction model according to the speech feature and the image feature comprises: Performing noise processing on the image features to obtain noisy image features; Performing denoising processing on the noisy image feature according to the speech feature to obtain a denoised image feature; Determining a first loss according to the denoised image feature and the speech feature; Determining a second loss according to the image feature and the speech feature; The speech feature extraction model is trained according to the first loss and / or the second loss.

3. The method according to claim 2, characterized in that The performing noise processing on the image feature to obtain the noisy image feature includes: Determining some of the image features; Determine the noise value and determine the noise image characteristics; The noise value is added to part of the image features in the image features, or the noise image features are used to replace part of the image features in the image features to obtain the noisy image features.

4. The method according to any one of claims 1 to 3, characterized in that: After training the speech feature extraction model according to the speech feature and the image feature, the method further comprises: Obtaining sample speech data corresponding to a sample speech recognition task and a language label corresponding to the sample speech data; Based on the sample speech data and the language label, fine-tuning the trained speech feature extraction model to obtain a speech recognition model; The speech recognition model is used to recognize the sample speech data to obtain the recognition result of the sample speech recognition task.

5. A speech recognition method, characterized in that: include: Obtaining voice data to be recognized; Inputting the speech data to be recognized into a speech feature extraction model trained by the method according to any one of claims 1 to 3 to obtain speech features of the speech data to be recognized; or, The speech data to be recognized is input into the speech recognition model trained by the method as claimed in claim 4 to obtain the recognition result of the speech data to be recognized.

6. A model training device, characterized in that: include: An acquisition module is used to: acquire a plurality of voice-image pair data, wherein target voice-image pair data in the plurality of voice-image pair data includes: target voice data and target image data with consistent corresponding contents; A first feature extraction module is used to extract features of any target speech image pair data among the plurality of speech image pair data by using a speech feature extraction model to obtain speech features of the target speech data, wherein the speech features include: speech global features and / or speech local features; A second feature extraction module is used to extract features from the target image data through a visual feature extraction model to obtain image features of the target image data, wherein the image features include: global image features and / or local image features; A training module is used to train the speech feature extraction model according to the speech features and the image features.

7. A speech recognition device, characterized in that: include: An acquisition module, used to acquire speech data to be recognized; A recognition module, used to: input the speech data to be recognized into a speech feature extraction model trained by the method according to any one of claims 1 to 6 to obtain speech features of the speech data to be recognized; or The speech data to be recognized is input into the speech recognition model trained by the method as claimed in claim 7 to obtain the recognition result of the speech data to be recognized.

8. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 5 by executing the executable instructions.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

10. A computer program product comprising instructions, characterized in that When the computer program product is executed on an electronic device, the electronic device is enabled to execute the method according to any one of claims 1 to 5.

Images