Speech recognition method and device, computer equipment and storage medium
By fusing the modal features and sub-features of multimodal data in speech recognition technology and fusion of weights, the problem of insufficient speech recognition accuracy in noise environments is solved, and higher speech information accuracy is achieved.
Patent Information
- Application Number
- CN202411522416.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-29
- Publication Date
- 2025-02-25
AI Technical Summary
Existing voice recognition technology is difficult to ensure the accuracy of voice recognition results in a noise-rich environment, especially when the cab is affected by engine sounds, wind noises, etc.
By acquiring multiple modal data (such as audio data and video data), the modal features and modal sub-features of modal data are determined in the target model, the weights of each modal sub-feature are calculated and fused, and the fused features are generated to improve the accuracy of speech information.
It effectively reduces noise in the fusion feature, improves the accuracy of speech information, and significantly improves the speech recognition effect in a noisy environment.
Smart Images
Figure CN119517011A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a speech recognition method, apparatus, computer equipment and storage medium. Background Art
[0002] Although speech recognition has been achieved through deep learning models, the accuracy of speech recognition results is still greatly affected by factors such as the rich types of noise and high volume. Taking in-cab speech recognition as an example, noise includes but is not limited to engine noise, wind noise, road noise, other vehicles, thunder and other sounds from the environment. The above noises all affect the clarity and quality of speech signals to varying degrees, resulting in insufficient accuracy of current speech recognition. Summary of the invention
[0003] Based on this, it is necessary to provide a speech recognition method, apparatus, computer device and storage medium to address the above technical problems so as to improve the accuracy of speech information obtained by speech recognition.
[0004] In a first aspect, an embodiment of the present application provides a speech recognition method, comprising:
[0005] Acquire data to be processed, and input the data to be processed into a target model; wherein the data to be processed includes multiple modal data, the modal data includes voice information, and the modal data of at least two modalities corresponds to the same time segment;
[0006] Determining, by means of the target model, a modal feature of the modal data and a feature set consisting of the modal sub-features of the same time segment; wherein the modal feature includes a plurality of the modal sub-features;
[0007] Determine a first weight corresponding to the modal sub-feature in the feature set, and based on the first weight, fuse the modal sub-features in the feature set to obtain a fused feature; wherein the fused feature corresponds to the feature set;
[0008] The speech information is determined based on the fusion features of the multiple time segments.
[0009] In one embodiment, determining a first weight corresponding to the modality sub-feature in the feature set includes:
[0010] Performing convolution processing on the modal sub-feature in the feature set to obtain an intermediate feature corresponding to the modal sub-feature;
[0011] The first weight is obtained by processing the product of the intermediate features through a first sigmoid function.
[0012] In one embodiment, the plurality of modal data are composed of first modal data and second modal data; the modal feature is composed of a first modal feature and a second modal feature, and a first modal sub-feature in the first modal feature and a second modal sub-feature in the second modal feature constitute the feature set;
[0013] Then the first weight is calculated by the following formula:
[0014] a1=sigmoid(A′1+β2A′2), a2=sigmoid(ω1A′1+A′2);
[0015] Among them, a1 is the first weight corresponding to the first modal sub-feature, a2 is the first weight corresponding to the second modal sub-feature, A′1 is the first intermediate feature corresponding to the first modal sub-feature, A′2 is the second intermediate feature corresponding to the second modal sub-feature, β2 is the first preset hyperparameter, and ω1 is the second preset hyperparameter.
[0016] In one of the embodiments, the plurality of modality data includes audio data;
[0017] Then determining the modal features of the modal data includes:
[0018] Inputting the audio data into at least two convolutional layers, and concatenating the audio convolution features output by the at least two convolutional layers to obtain concatenated features; wherein the at least two convolutional layers have different receptive fields;
[0019] Performing a pooling operation on the spliced features in a channel dimension to obtain a channel descriptor of the spliced features;
[0020] A second weight is determined by inputting the channel descriptor into two sequentially connected fully connected layers, and a product of the second weight and the concatenated feature is determined as a first modality feature corresponding to the audio data.
[0021] In one embodiment, performing a pooling operation on the splicing feature in a channel dimension to obtain a channel descriptor of the splicing feature includes:
[0022] Performing an average pooling operation on the concatenated features in a channel dimension to obtain a first descriptor in the channel descriptor;
[0023] A maximum pooling operation is performed on the concatenated features in the channel dimension to obtain a second descriptor in the channel descriptor.
[0024] In one embodiment, the second weight is calculated by the following formula:
[0025]
[0026] Where γ is the second weight, W1 corresponds to the first fully connected layer, and W2 corresponds to the second fully connected layer. For the first descriptor, is the second descriptor.
[0027] In one of the embodiments, the plurality of modality data includes video data;
[0028] Then determining the modal features of the modal data includes:
[0029] Performing convolution processing on the lip image in the video data to obtain a global feature; wherein the global feature includes a plurality of global sub-features;
[0030] Inputting the global feature into a feature slicing module to obtain a first number of the global sub-features, and subtracting the global feature from the first number of the global sub-features to obtain a differential feature;
[0031] Performing convolution processing on the differential feature to obtain a differential convolution feature, and processing the differential convolution feature through a second sigmoid function to obtain a third weight;
[0032] A Hadamard product of the third weight and the differential feature is determined as a second modality feature corresponding to the video data.
[0033] In a second aspect, an embodiment of the present application provides a speech recognition device, including:
[0034] An input module, used to obtain data to be processed and input the data to be processed into a target model; wherein the data to be processed includes multiple modal data, the modal data includes voice information, and the modal data of at least two modalities corresponds to the same time segment;
[0035] A feature module, used to determine the modal feature of the modal data and a feature set consisting of the modal sub-features of the same time segment through the target model; wherein the modal feature includes a plurality of the modal sub-features;
[0036] A fusion module, used to determine a first weight corresponding to the modal sub-feature in the feature set, and based on the first weight, fuse the modal sub-features in the feature set to obtain a fused feature; wherein the fused feature corresponds to the feature set;
[0037] A speech module is used to determine the speech information based on the fusion features of the multiple time segments.
[0038] In one of the embodiments, the fusion module is specifically used to perform convolution processing on the modal sub-features in the feature set to obtain intermediate features corresponding to the modal sub-features; and process the product of the intermediate features through a first sigmoid function to obtain the first weight.
[0039] In one embodiment, the multiple modal data consist of first modal data and second modal data; the modal feature consists of a first modal feature and a second modal feature, and the first modal sub-feature in the first modal feature and the second modal sub-feature in the second modal feature constitute the feature set; then the first weight is calculated by the following formula: a1=sigmoid(A′1+β2A′2), a2=sigmoid(ω1A′1+A′2); wherein a1 is the first weight corresponding to the first modal sub-feature, a2 is the first weight corresponding to the second modal sub-feature, A′1 is the first intermediate feature corresponding to the first modal sub-feature, A′2 is the second intermediate feature corresponding to the second modal sub-feature, β2 is the first preset hyperparameter, and ω1 is the second preset hyperparameter.
[0040] In one embodiment, the multiple modal data include audio data; the feature module is specifically used to input the audio data into at least two convolutional layers, and splice the audio convolution features output by the at least two convolutional layers to obtain a spliced feature; wherein the receptive fields of the at least two convolutional layers are different; a pooling operation is performed on the spliced feature in the channel dimension to obtain a channel descriptor of the spliced feature; a second weight is determined by inputting the channel descriptor into two sequentially connected fully connected layers, and the product of the second weight and the spliced feature is determined as a first modal feature corresponding to the audio data.
[0041] In one of the embodiments, the feature module is specifically used to perform an average pooling operation on the spliced features in the channel dimension to obtain a first descriptor in the channel descriptor; and perform a maximum pooling operation on the spliced features in the channel dimension to obtain a second descriptor in the channel descriptor.
[0042] In one embodiment, the second weight is calculated by the following formula: γ= Where γ is the second weight, W1 corresponds to the first fully connected layer, and W2 corresponds to the second fully connected layer. For the first descriptor, is the second descriptor.
[0043] In one embodiment, the multiple modal data include video data; the feature module is further used to perform convolution processing on the lip picture in the video data to obtain a global feature; wherein the global feature includes multiple global sub-features; the global feature is input into a feature slicing module to obtain a first number of the global sub-features, and the global feature is subtracted from the first number of the global sub-features to obtain a differential feature; convolution processing is performed on the differential feature to obtain a differential convolution feature, and the differential convolution feature is processed by a second sigmoid function to obtain a third weight; and the Hadamard product of the third weight and the differential feature is determined as a second modal feature corresponding to the video data.
[0044] In a fourth aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in the first aspect and any possible implementation manner when executing the computer program.
[0045] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method described in the first aspect and any possible implementation manner are implemented.
[0046] The above-mentioned speech recognition method, apparatus, computer equipment and storage medium, after determining the modal features of the multimodal data, determine the first weights of the modal sub-features in each feature set respectively, based on the feature set, so as to fuse the modal sub-features in the feature set using the first weights, thereby effectively reducing the noise in the fused features and enabling the feature information corresponding to the speech information to be highlighted in the fused features, thereby effectively improving the accuracy of the speech information obtained through the fused features. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.
[0048] Figure 1 A diagram of an application environment of a speech recognition method in an embodiment;
[0049] Figure 2 A flowchart of a speech recognition method in one embodiment;
[0050] Figure 3A schematic diagram of a flow chart of a step of determining a first weight corresponding to a modal feature in a feature set in one embodiment;
[0051] Figure 4 A schematic diagram of a flow chart of determining a first modal feature corresponding to audio data in one embodiment;
[0052] Figure 5 A schematic diagram of a method for determining a first modal feature corresponding to audio data in one embodiment;
[0053] Figure 6 A schematic diagram of a flow chart of determining a second modality feature corresponding to video data in one embodiment;
[0054] Figure 7 A schematic diagram of a method for determining a second modality feature corresponding to video data in one embodiment;
[0055] Figure 8 is a structural block diagram of a speech recognition device in one embodiment;
[0056] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0057] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application. Obviously, the described embodiment is only a part of the embodiment of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be arbitrarily combined with each other. In addition, although the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in an order different from that here.
[0058] The terms "first" and "second" in the specification and claims of the present application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of their variations are intended to cover non-exclusive protection. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices. "Multiple" in the present application can mean at least two, for example, two, three or more, and the embodiments of the present application are not limited.
[0059] To facilitate understanding of the technical solution provided by the embodiments of the present application, the following first introduces the design concept of the embodiments of the present application:
[0060] Currently, in order to reduce the impact of noise on speech recognition results, a common method is to process the original speech data to convert it into different forms of data such as text data and Mel-spectrograms, and then fuse the different forms of data to extract features; thereby reducing the impact of noise through speech enhancement. However, due to the large number and rich types of features, this method is prone to losing useful information when extracting features, resulting in insufficient speech recognition accuracy.
[0061] To this end, an embodiment of the present application provides a speech recognition method, which takes a feature set as a processing unit, and for each modal sub-feature in the feature set (i.e., the same time segment), weighted processing is performed on each modal sub-feature by determining a first weight corresponding to each modal sub-feature in the feature set, thereby achieving effective fusion between the modal sub-features in the feature set, thereby effectively reducing the noise content introduced in the fused feature, and significantly improving the accuracy of the speech information determined by the fused feature.
[0062] The speech recognition method provided in this application can be applied to Figure 1 In the application environment shown, the terminal 102 communicates with the server 104 through the network. The terminal 102 can be used to collect data to be processed and send the data to be processed to the server 104 through the network so that the server processes the data to be processed to determine the voice information contained in the data to be processed.
[0063] Alternatively, the server 104 may send the speech recognition method or target model to the terminal in advance in the form of SDK (Software Development Kit) via the network, so that the terminal 102 can directly process the data to be processed after collecting the data to be processed to determine the speech information in the data to be processed.
[0064] The terminal 102 may be different types of data acquisition devices. The terminal 102 may be, but is not limited to, a vehicle-mounted camera, various personal computers, laptop computers, smart phones, tablet computers, and portable wearable devices. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.
[0065] In one embodiment, Figure 2 As shown, a speech recognition method is provided, which is applied to Figure 1 The server in the example is used to illustrate the following steps:
[0066] Step 201: Obtain the data to be processed and input the data to be processed into the target model.
[0067] The data to be processed includes multiple modal data, and each modal data in the multiple modal data includes voice information.
[0068] Furthermore, the modal data carries a time tag, and since each modal data corresponds to a time segment containing at least one preset time period, the time tag can be a time segment.
[0069] Among the above-mentioned multiple modal data, the modal data of at least two modalities correspond to the same time segment.
[0070] For example, a time segment is represented by the start time of speech and the end time of speech. Then the same time segment refers to a time segment with the same start time and end time.
[0071] The above-mentioned multiple modal data may include at least one of audio data, video data containing the target person (or lips) emitting the sound, text data (for example, by converting audio into text), and physiological data of the target person emitting the sound (for example, heart rate).
[0072] The above target model at least includes multiple convolutional layers for feature extraction.
[0073] Step 202: determine the modal features of the modal data and a feature set consisting of modal sub-features of the same time segment through the target model.
[0074] Wherein, the modal feature includes a plurality of modal sub-features;
[0075] Specifically, corresponding to the modal data, the modal sub-feature contains the time mark corresponding to the time segment, then the modal feature A m Contains modal sub-features A of multiple time segments i mi .
[0076] Among them, m and i are natural numbers, m corresponds to the mode of the modal data, A m represents the mth mode feature of the mth mode, A mi Represents the mth modal feature of time segment i.
[0077] And, the modal feature A m At least two modal sub-features A corresponding to the same time segment i but corresponding to different modal data mi .
[0078] In this way, the modal characteristics A of each modal data can be determined separately m Then, according to the time stamp contained in the modal data, the modal feature A m Determine the modal sub-feature A of each time segmentmi Finally, the modal sub-feature A with the same time segment mi Divide into the same feature set and obtain multiple feature sets.
[0079] Among them, feature sets are distinguished from each other by their respective time segments.
[0080] Step 203, determining a first weight corresponding to the modal sub-feature in the feature set, and based on the first weight, fusing the modal sub-features in the feature set to obtain a fused feature.
[0081] Among them, the fusion feature corresponds to the feature set.
[0082] Specifically, the fused features correspond one-to-one to the feature sets; and also correspond one-to-one to the time segments of each feature set.
[0083] The fusion feature of each feature set can be calculated by the following formula:
[0084]
[0085] Among them, H i represents the fusion feature of the feature set of time segment i; n is the number of modal sub-features in the feature set of time segment i; A mi is the mth modal sub-feature in the feature set of time segment i; a m is the mth modal sub-feature A mi The first weight of .
[0086] The fusion feature can be obtained through the fusion module of the target model; the fusion module can be used to first determine the first weight corresponding to the modal sub-features one by one, and then the sum of the products between the first weight and the modal sub-features is calculated to obtain the fusion feature.
[0087] Step 204: Determine speech information based on fusion features of multiple time segments.
[0088] Specifically, the voice information contained in the data to be processed may be determined according to the time segments of the preset interval time or the fusion characteristics of the continuous time segments.
[0089] Then the fusion feature H of multiple time segments all It can be expressed as: H all =[H1,H2,…,H i ], where i represents the time segment i, then H i The fused feature H represents the feature set of time segment t.
[0090] In one embodiment, the voice information can be determined by processing the fused features according to the time segments corresponding to the fused features through the LSTM (Long Short-Term Memory) in the target model, so as to capture the connection between the fused features of each time segment through LSTM and more accurately determine the above voice information.
[0091] Then the fusion features H of the above multiple fusion fragments can be all Input LSTM to obtain speech information.
[0092] In the above-mentioned speech recognition method, by determining the first weights of the modal sub-features of multiple modal data of the same time segment, the modal sub-features of the same time segment are fused, and then the speech information is determined by the fused features obtained by fusion. In this speech recognition method, information complementation is achieved through different modal data, and adaptive fusion of modal sub-features of the same time segment is achieved through the first weight, thereby avoiding the loss of effective information caused by direct fusion and / or the decrease in the accuracy of the speech information obtained by recognition due to the introduction of noise.
[0093] The first weights can adapt to the characteristics of different modal data and different speech occurrence scenarios, and thus can achieve personalized weighting corresponding to different modal sub-features. The first weights can be obtained by convolution processing the modal sub-features and using preset hyperparameters to control the higher-level abstract features obtained by the convolution processing.
[0094] The following is a detailed description of the determination of the first weight, please refer to Figure 3 :
[0095] Step 301, performing convolution processing on the modal sub-features in the feature set to obtain intermediate features corresponding to the modal sub-features.
[0096] Specifically, the modal sub-features can be convolved through a one-dimensional convolution layer to obtain intermediate features. 1i , the second modal sub-feature A 2i For example, i indicates that the time segment of the feature set is time segment i:
[0097] A'1=Conv1D(A1); A'2=Conv1D(A2);
[0098] Then A′1 is the first intermediate feature corresponding to the first modal sub-feature, and A′2 is the second intermediate feature corresponding to the second modal sub-feature.
[0099] In this way, more abstract high-level features in the modal sub-features can be obtained through the one-dimensional convolution layer.
[0100] In the embodiment of the present application, the target model is a pre-trained speech recognition model. The one-dimensional convolution layer used to process each modal sub-feature can be the same one-dimensional convolution layer, or it can be a one-dimensional convolution layer corresponding to the modal sub-feature. The difference between the two settings mainly lies in the different training data sets used in the training process. If the one-dimensional convolution layer used to process each modal sub-feature is the same one-dimensional convolution layer, the corresponding training data set includes the modal sub-features of all modal data at the same time. If the one-dimensional convolution layer used to process each modal sub-feature corresponds to the modal sub-feature one-to-one, the corresponding training data set is the modal sub-feature of the corresponding modal data.
[0101] Step 302: Process the product of the intermediate features through a first sigmoid function to obtain a first weight.
[0102] Specifically, the first sigmoid function can make each first weight be located in the interval (0, 1).
[0103] It should be noted that since the first weight is actually used to regulate the interaction between the modal sub-features in the same feature set (i.e., the modal sub-features of the same time segment), when the product of the intermediate features is processed by the first sigmoid function, the feature set is used as a unit, and the processing object each time is the intermediate feature of the modal sub-features in the same feature set.
[0104] Optionally, in order to achieve differentiation between the first weight and the second weight, so as to obtain intermediate features corresponding to the modal sub-features one by one, the product of the intermediate features can be calculated by adjusting the order of each intermediate feature. Take the first intermediate feature A′1 corresponding to the first modal sub-feature, the second intermediate feature A′2 corresponding to the second modal sub-feature, and the third intermediate feature A′3 corresponding to the third modal sub-feature as examples: to obtain the first weight a1 corresponding to the first modal sub-feature, calculate by the following formula: a1=sigmoid(A′1×A′2×A′3);
[0105] To obtain the first weight a2 corresponding to the second modal sub-feature, it is calculated by the following formula: a2=simoid(A′2×A′1×A′3).
[0106] To obtain the first weight a3 corresponding to the third modal sub-feature, it is calculated by the following formula: a3=sigmoid(A′3×A′1×A′2).
[0107] Optionally, the first weight corresponding to each modal sub-feature may be determined by presetting a hyperparameter. The following is an explanation of the method for determining the first weight by presetting a hyperparameter:
[0108] The first weight of each modal sub-feature corresponds to a set of preset hyper-parameters. Still taking the first intermediate feature A′1 corresponding to the first modal sub-feature, the second intermediate feature A′2 corresponding to the second modal sub-feature, and the third intermediate feature A′3 corresponding to the third modal sub-feature as an example:
[0109] To obtain the first weight a1 corresponding to the first modal sub-feature, the following formula is used for calculation: a1=sigmoid(β1A′1×β2A′2×β3A′3); wherein β1>β2, and β1>β3; β1-β3 are a set of preset hyperparameters corresponding to the first weight a1;
[0110] To obtain the first weight a2 corresponding to the second modal sub-feature, the following formula is used for calculation: a2=sigmoid(ω1A′1×ω2A′2×ω3A′3); wherein ω2>ω1, and ω2>ω3; ω1-ω3 are a set of preset hyperparameters corresponding to the first weight a2;
[0111] To obtain the first weight a3 corresponding to the third modal sub-feature, the following formula is used for calculation: a3=sigmoid(σ1A′1×σ2A′2×σ3A′3); wherein, and are both less than 0.5, σ3>σ1, and σ3>σ2; σ1-σ3 are a set of preset hyperparameters corresponding to the first weight a3.
[0112] In one embodiment, the modal data is composed of the first modal data and the second modal data. Correspondingly, the modal feature is composed of the first modal feature A1 and the second modal feature A2.
[0113] The modal sub-feature is composed of the first modal sub-feature A 1i and the second modal sub-feature A 2i The first modal sub-feature A corresponding to time segment i is 1i and the second modal sub-feature A 2i A feature set is formed, and the intermediate feature is composed of the first intermediate feature A′1 and the second intermediate feature A′2. 1i The corresponding first weight a1 is calculated by the following formula:
[0114] a1=sigmoid(β1A′1×β2A′2) where β1>β2; preferably, β1=1.
[0115] With the second modal sub-feature A 2i The corresponding first weight a2 is calculated by the following formula: a2=sigmoid(ω1A′1×ω2A′2), wherein ω2>ω1; preferably, ω2=1.
[0116] Furthermore, the above-mentioned multiple modal data may include audio data; the following takes the first modal data as audio data as an example to illustrate the determination of the first modal feature, please refer to Figure 4 :
[0117] Step 401: input audio data into at least two convolutional layers, and concatenate audio convolution features output by the at least two convolutional layers to obtain concatenated features.
[0118] Among them, the receptive fields of at least two convolutional layers are different.
[0119] Specifically, at least two convolutional layers may both be one-dimensional convolutional layers, for example, a first convolutional layer and a second convolutional layer, the receptive field of the first convolutional layer is 20×20, and the receptive field of the second convolutional layer is 10×10.
[0120] The concatenation of the above audio convolution features refers to concatenating the audio convolution features in the channel dimension. For example, if two audio convolution features have 20 channels respectively, the number of channels of the concatenated feature is 40.
[0121] The first audio signal includes time markers corresponding to time segments.
[0122] Optionally, before the audio data is input into at least two convolutional layers, the audio data may be first converted into cepstral coefficients, and then noise in the cepstral coefficients may be removed to obtain a first audio signal, which is then input into the convolutional layer.
[0123] Step 402: Perform a pooling operation on the spliced features in the channel dimension to obtain a channel descriptor of the spliced features.
[0124] The channel descriptor is used to describe the spatial information contained in the splicing feature.
[0125] Specifically, the channel descriptor may include a first descriptor obtained by average pooling and a second descriptor obtained by maximum pooling.
[0126] Alternatively, the channel descriptor is composed of the first descriptor and the second descriptor. In one embodiment, the concatenated features may be firstly subjected to an average pooling operation in the channel dimension to obtain the first descriptor. Then, the concatenated features may be subjected to a maximum pooling operation in the channel dimension to obtain the second descriptor.
[0127] Step 403: Input the channel descriptor into two sequentially connected fully connected layers, determine a second weight, and determine the product of the second weight and the concatenated feature as a first modal feature corresponding to the audio data.
[0128] Specifically, when the above audio data is the first modal data, the modal feature of the audio data is the first modal feature A1, A1=γ×F; wherein γ is the second weight, and F is the splicing feature.
[0129] The second weight γ can be obtained by confusing the first descriptor and the second descriptor through the fully connected layer. It can be calculated by the following formula:
[0130]
[0131] Among them, γ is the second weight, W1 corresponds to the first fully connected layer, and W2 corresponds to the second fully connected layer. is the first descriptor, is the second descriptor.
[0132] Among them, W1 may specifically be a parameter of the first connection layer, and W2 may specifically be a parameter of the second connection layer.
[0133] in, L is the length of the time segment of the first audio signal, F c is the channel feature of the cth channel in the splicing feature F. Since the number of channels of the splicing feature F is C, c satisfies: 1≤c≤C.
[0134] In one embodiment, the first audio signal may be obtained by using the target sparse weight w obtained by pre-training. s0 , obtained by removing the cepstral coefficients of the audio data.
[0135] but: in is the first audio signal, and x is the aforementioned cepstral coefficient.
[0136] In one embodiment, the target sparse weight w s0 Determined by the following method: First, the cepstral coefficients of the audio data are normalized to obtain the target cepstral coefficients corresponding to the time segment. The audio data contains time tags corresponding to the time segment. The method for determining the cepstral coefficients is not limited in the present embodiment of the application, for example, it can be obtained by taking the logarithm of the spectrum of the audio data and then performing Fourier transform. The above normalization can be specifically implemented by the following formula:
[0137] Among them, X a is the audio data, Xa_min is the minimum value of the cepstral coefficients of the corresponding time segment, Xa_max is the maximum value of the cepstral coefficients of the corresponding time segment, and X′ a is the target cepstrum coefficient.
[0138] Then, the SR (Sparse Representation) loss function Q is optimized. When the value of Q drops to the minimum value, or Q converges, the corresponding sparse weight w can be determined. s is the target sparse weight w s0 .
[0139] Furthermore, the loss function Q can be set as a cross entropy loss function, which can be specifically expressed as:
[0140] Q = min C + min Q1 + min Q2;
[0141] in, Q2=β||w s ||1; β is the sparse regularization coefficient, w s is the sparse weight to be optimized. Q1 is used to retain the pulse feature, Q2 is used to remove noise, and minCy is used to minimize the classification error. is the first audio signal, and the relationship between it and the inverse spectrum coefficient x is: N is the noise.
[0142] The following combination Figure 5 , the method for determining the first modal feature described in 401-402 is illustrated by way of example:
[0143] First, the first modal data is input into the one-dimensional convolution layer with a receptive field of 20×20: the first convolution layer CNN_1; the one-dimensional convolution layer with a receptive field of 10×10: the second convolution layer CNN_2; the one-dimensional convolution layer with a receptive field of 5×5: the third convolution layer CNN_3, so that the audio convolution features in different frequency domains are extracted through the above three convolution layers, which are the first convolution feature, the second convolution feature, and the third convolution feature. The above first modal data can be w in the aforementioned SR s0 The first audio signal is processed.
[0144] Then, the first convolution feature, the second convolution feature, and the third convolution feature are concatenated in the channel dimension to obtain the concatenated feature F.
[0145] Next, the concatenated features are subjected to a Max (Max Pooling) operation and an Ave (Average Pooling) operation to obtain a first channel description and a second descriptor, respectively.
[0146] The first descriptor and the second descriptor are mixed and input into two fully connected layers and sigmoid to generate a second weight γ.
[0147] Finally, the second weight γ is multiplied by the concatenated feature F to obtain the first modal feature.
[0148] Furthermore, the above-mentioned multiple modal data may also include video data. To maintain time consistency, the frame ratio between the audio data and the video data may be set within a preset range. The preset range may be (2-8): 1. The target of the audio in the video data is the sound source of the above-mentioned audio data.
[0149] The following describes a method for determining the second modal features corresponding to the second modal data, taking the second modal data as video data. Figure 6 :
[0150] Step 601: Perform convolution processing on the lip image in the video data to obtain the global feature X v .
[0151] The global feature includes multiple global sub-features.
[0152] Specifically, the global sub-features may be sub-features on different channels obtained by dividing the channels as units.
[0153] The lip image can be obtained by taking the area where the lips are located in the video data as the area of interest, so as to extract the area of interest for each video frame, thereby reducing the amount of feature calculation.
[0154] In one embodiment, a toolkit such as DLib may be used to extract the lip region in the video frame to obtain a ROI (Region of Interest) region with a preset aspect ratio as the image to be processed. The preset aspect ratio may be 160×80.
[0155] Then, the pixels in each frame ROI can be normalized: Where X v is the pixel value of the pixel in ROI after normalization. The pixel value of the pixel point in the ROI obtained by extracting the ROI.
[0156] Then, the pixel set of the normalized ROI area can be Perform convolution processing.
[0157] Step 602: Input the global feature to the feature slicing module to obtain a first number of global sub-features, and perform a difference between the global feature and the first number of global sub-features to obtain a differential feature X v，d .
[0158] Specifically, the first number of global sub-features are channel features corresponding to speech information.
[0159] That is, the first number of global sub-features are selected after the global features are divided in the channel dimension by the feature slicing module.
[0160] Exemplarily, the first number of global sub-features may include: one or more of: a color histogram, a texture feature, a hysteresis feature, and a moving average feature.
[0161] The feature slicing module is part of the target model, so it is trained together with the target model. The loss function of the target model may include a sub-function corresponding to the feature slicing module. As the above-mentioned loss function of the target model is iterated, the parameters in the feature slicing module are optimized. When the loss function of the target model in the training stage converges or drops to a minimum value, the feature slicing module determines the aforementioned first quantity, and the performance of the feature slicing module in identifying the first number of global sub-features among multiple global sub-features reaches a preset standard. The preset standard means that a number of global sub-features identified by the feature slicing module highly overlap with the channel features in the global features that characterize the voice information of the target person.
[0162] Further, the global feature and the first number of global sub-features are expressed as a difference, and the global feature and each global sub-feature in the first number of global sub-features are subtracted element by element to obtain a differential feature X v，d ∈R a×b .
[0163] In one embodiment, the differential feature X v，d It can be calculated by the following formula:
[0164] X v,d =Conv2D(X v )! CS(X v );
[0165] Or, X v,d =θ(BN(Conv2D(X v )))! CS(X v );
[0166] Among them, Conv2D(X v ) is a global feature, "!" means element-by-element subtraction (i.e., difference), BN (BatchNormalization) is batch normalization, CS (X v ) represents the first number of global sub-features obtained by the feature slicing module, θ represents the activation function ReLu, and Conv2D represents a two-dimensional convolutional layer.
[0167] Step 603: performing convolution processing on the differential feature to obtain a differential convolution feature, and processing the differential convolution feature through a second sigmoid function to obtain a third weight.
[0168] Specifically, the third weight can be expressed as follows:
[0169] sigmoid(BN(Cοпv1D(X v,d ))).
[0170] In the submodule for identifying the second modal features of video data contained in the target model, since the aforementioned differential features are obtained by subtracting the global subfeatures selected on the channel dimension from the global features, the third weight can help the above submodule focus on processing the channel features corresponding to the voice information, thereby further improving the accuracy of the second modal features obtained in step 604.
[0171] Step 604: Determine the Hadamard product of the third weight and the differential feature as the second modal feature corresponding to the video data.
[0172] When the video data is the second modal data, its modal feature is the second modal feature A2, and the expression of the second modal feature is as follows:
[0173] A2=σ(BN(Сοпv1D(X v,d )))⊙CS(X v ), σ represents the second sigmoid function, ⊙ represents the Hadamard product, and Conv1D represents a one-dimensional convolutional layer.
[0174] The following combination Figure 7 , the modal feature (i.e., the second modal feature) of determining the video data described in steps 601-604 is further described:
[0175] First, the ROI area obtained by the above normalization process is used as the image to be processed and input into the pre-trained two-dimensional convolution layer Conv2D to obtain the global feature. The two-dimensional convolution layer is then connected with the BN layer and the activation function ReLu in sequence, so that the stability of the global feature is improved after passing through the BN layer and the activation function ReLu, and it has stronger nonlinear expression ability.
[0176] At the same time, the feature slicing module performs feature slicing processing on the global features to split the global features from the channel dimension and extract a first number of global sub-features.
[0177] Then, the global features after the BN layer and the activation function ReLu are subtracted element by element from the global sub-features obtained by feature slicing to obtain differential features.
[0178] Next, the differential feature is input into the one-dimensional convolution layer Conv1D, and passes through the BN layer and the second activation function sigmoid in sequence to obtain the third weight.
[0179] Finally, the Hadamard product of the third weight and the aforementioned first number of global sub-features is calculated to obtain the second modal feature.
[0180] It should be understood that although Figure 2 , Figure 3-4 , Figure 6 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2 , Figure 3-4 , Figure 6 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.
[0181] Based on the same inventive concept, the embodiment of the present application provides a speech recognition device, including: an input module 801, a feature module 802, a fusion module 803 and a speech module, wherein:
[0182] The input module 801 is used to obtain data to be processed and input the data to be processed into the target model, wherein the data to be processed includes multiple modal data, the modal data includes voice information, and the modal data of at least two modalities corresponds to the same time segment.
[0183] The feature module 802 is used to determine the modal features of the modal data and the feature set consisting of the modal sub-features of the same time segment through the target model.
[0184] Wherein, the modal feature includes a plurality of the modal sub-features;
[0185] The fusion module 803 is used to determine a first weight corresponding to the modal sub-feature in the feature set, and based on the first weight, fuse the modal sub-features in the feature set to obtain a fused feature; wherein the fused feature corresponds to the feature set.
[0186] The speech module 804 is used to determine the speech information based on the fusion features of the multiple time segments.
[0187] In one embodiment, the fusion module 803 is specifically used to perform convolution processing on the modal sub-features in the feature set to obtain intermediate features corresponding to the modal sub-features; and process the product of the intermediate features through a first sigmoid function to obtain the first weight.
[0188] In one embodiment, the multiple modal data consist of first modal data and second modal data; the modal feature consists of a first modal feature and a second modal feature, and the first modal sub-feature in the first modal feature and the second modal sub-feature in the second modal feature constitute the feature set; then the first weight is calculated by the following formula: a1=sigmoid(A′1+β2A′2), a2=sigmoid(ω1A′1+A′2); wherein a1 is the first weight corresponding to the first modal sub-feature, a2 is the first weight corresponding to the second modal sub-feature, A′1 is the first intermediate feature corresponding to the first modal sub-feature, A′2 is the second intermediate feature corresponding to the second modal sub-feature, β2 is the first preset hyper-parameter, and ω1 is the second preset hyper-parameter.
[0189] In one embodiment, the multiple modal data include audio data; the feature module 802 is specifically used to input the audio data into at least two convolutional layers, and splice the audio convolution features output by the at least two convolutional layers to obtain a spliced feature; wherein the receptive fields of the at least two convolutional layers are different; a pooling operation is performed on the spliced feature in the channel dimension to obtain a channel descriptor of the spliced feature; a second weight is determined by inputting the channel descriptor into two sequentially connected fully connected layers, and the product of the second weight and the spliced feature is determined as the first modal feature corresponding to the audio data.
[0190] In one embodiment, the feature module 802 is specifically used to perform an average pooling operation on the spliced features in the channel dimension to obtain a first descriptor in the channel descriptor; and perform a maximum pooling operation on the spliced features in the channel dimension to obtain a second descriptor in the channel descriptor.
[0191] In one embodiment, the second weight is calculated by the following formula: Where γ is the second weight, W1 corresponds to the first fully connected layer, and W2 corresponds to the second fully connected layer. For the first descriptor, is the second descriptor.
[0192] In one embodiment, the multiple modal data include video data; the feature module 802 is also used to perform convolution processing on the lip picture in the video data to obtain a global feature; wherein the global feature includes multiple global sub-features; the global feature is input into the feature slicing module to obtain a first number of the global sub-features, and the global feature is subtracted from the first number of the global sub-features to obtain a differential feature; convolution processing is performed on the differential feature to obtain a differential convolution feature, and the differential convolution feature is processed by a second sigmoid function to obtain a third weight; the Hadamard product of the third weight and the differential feature is determined as the second modal feature corresponding to the video data.
[0193] For the specific definition of the speech recognition device, please refer to the definition of the speech recognition method above, which will not be repeated here. Each module in the above-mentioned speech recognition device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
[0194] Based on the same inventive concept, the embodiment of the present application provides a computer device, which may be a server, and its internal structure diagram may be as follows: Fig. 9 As shown. The computer device includes a processor, a memory, and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a speech recognition method is implemented.
[0195] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0196] Based on the same inventive concept, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are implemented:
[0197] Acquire data to be processed, and input the data to be processed into a target model; wherein the data to be processed includes multiple modal data, the modal data includes voice information, and the modal data of at least two modalities corresponds to the same time segment;
[0198] Determining, by means of the target model, a modal feature of the modal data and a feature set consisting of the modal sub-features of the same time segment; wherein the modal feature includes a plurality of the modal sub-features;
[0199] Determine a first weight corresponding to the modal sub-feature in the feature set, and based on the first weight, fuse the modal sub-features in the feature set to obtain a fused feature; wherein the fused feature corresponds to the feature set;
[0200] The speech information is determined based on the fusion features of the multiple time segments.
[0201] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0202] Convolution processing is performed on the modal sub-features in the feature set to obtain intermediate features corresponding to the modal sub-features; and the product of the intermediate features is processed by a first sigmoid function to obtain the first weight.
[0203] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0204] The multiple modal data are composed of first modal data and second modal data; the modal features are composed of first modal features and second modal features, and the first modal sub-feature in the first modal feature and the second modal sub-feature in the second modal feature constitute the feature set; then the first weight is calculated by the following formula: a1=sigmoid(A′1+β2A′2), a2=sigmoid(ω1A′1+A′2); wherein a1 is the first weight corresponding to the first modal sub-feature, a2 is the first weight corresponding to the second modal sub-feature, A′1 is the first intermediate feature corresponding to the first modal sub-feature, A′2 is the second intermediate feature corresponding to the second modal sub-feature, β2 is the first preset hyper-parameter, and ω1 is the second preset hyper-parameter.
[0205] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0206] The multiple modal data include audio data; the audio data is input into at least two convolutional layers, and the audio convolution features output by the at least two convolutional layers are concatenated to obtain a concatenated feature; wherein the receptive fields of the at least two convolutional layers are different; a pooling operation is performed on the concatenated feature in the channel dimension to obtain a channel descriptor of the concatenated feature; a second weight is determined by inputting the channel descriptor into two sequentially connected fully connected layers, and the product of the second weight and the concatenated feature is determined as a first modal feature corresponding to the audio data.
[0207] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0208] An average pooling operation is performed on the concatenated features in a channel dimension to obtain a first descriptor in the channel descriptor; and a maximum pooling operation is performed on the concatenated features in a channel dimension to obtain a second descriptor in the channel descriptor.
[0209] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0210] The multiple modal data include video data; convolution processing is performed on the lip picture in the video data to obtain a global feature; wherein the global feature includes multiple global sub-features; the global feature is input into a feature slicing module to obtain a first number of the global sub-features, and the global feature is subtracted from the first number of the global sub-features to obtain a differential feature; convolution processing is performed on the differential feature to obtain a differential convolution feature, and the differential convolution feature is processed by a second sigmoid function to obtain a third weight; the Hadamard product of the third weight and the differential feature is determined as a second modal feature corresponding to the video data.
[0211] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:
[0212] Acquire data to be processed, and input the data to be processed into a target model; wherein the data to be processed includes multiple modal data, the modal data includes voice information, and the modal data of at least two modalities corresponds to the same time segment;
[0213] Determining, by means of the target model, a modal feature of the modal data and a feature set consisting of the modal sub-features of the same time segment; wherein the modal feature includes a plurality of the modal sub-features;
[0214] Determine a first weight corresponding to the modal feature in the feature set, and based on the first weight, fuse the modal features in the feature set to obtain a fused feature; wherein the fused feature corresponds to the feature set;
[0215] The speech information is determined based on the fusion features of the multiple time segments.
[0216] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0217] Convolution processing is performed on the modal sub-features in the feature set to obtain intermediate features corresponding to the modal sub-features; and the product of the intermediate features is processed by a first sigmoid function to obtain the first weight.
[0218] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0219] The multiple modal data are composed of first modal data and second modal data; the modal features are composed of first modal features and second modal features, and the first modal sub-feature in the first modal feature and the second modal sub-feature in the second modal feature constitute the feature set; then the first weight is calculated by the following formula: a1=sigmoid(A′1+β2A′2), a2=sigmoid(ω1A′1+A′2); wherein a1 is the first weight corresponding to the first modal sub-feature, a2 is the first weight corresponding to the second modal sub-feature, A′1 is the first intermediate feature corresponding to the first modal sub-feature, A′2 is the second intermediate feature corresponding to the second modal sub-feature, β2 is the first preset hyper-parameter, and ω1 is the second preset hyper-parameter.
[0220] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0221] The multiple modal data include audio data; the audio data is input into at least two convolutional layers, and the audio convolution features output by the at least two convolutional layers are concatenated to obtain a concatenated feature; wherein the receptive fields of the at least two convolutional layers are different; a pooling operation is performed on the concatenated feature in the channel dimension to obtain a channel descriptor of the concatenated feature; a second weight is determined by inputting the channel descriptor into two sequentially connected fully connected layers, and the product of the second weight and the concatenated feature is determined as a first modal feature corresponding to the audio data.
[0222] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0223] An average pooling operation is performed on the concatenated features in a channel dimension to obtain a first descriptor in the channel descriptor; and a maximum pooling operation is performed on the concatenated features in a channel dimension to obtain a second descriptor in the channel descriptor.
[0224] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0225] The second weight is calculated by the following formula: Where γ is the second weight, W1 corresponds to the first fully connected layer, and W2 corresponds to the second fully connected layer. For the first descriptor, is the second descriptor.
[0226] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0227] The multiple modal data include video data; convolution processing is performed on the lip picture in the video data to obtain a global feature; wherein the global feature includes multiple global sub-features; the global feature is input into a feature slicing module to obtain a first number of the global sub-features, and the global feature is subtracted from the first number of the global sub-features to obtain a differential feature; convolution processing is performed on the differential feature to obtain a differential convolution feature, and the differential convolution feature is processed by a second sigmoid function to obtain a third weight; the Hadamard product of the third weight and the differential feature is determined as a second modal feature corresponding to the video data.
[0228] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
[0229] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0230] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.
Claims
1. A speech recognition method, characterized in that: include: Acquire data to be processed, and input the data to be processed into a target model; wherein the data to be processed includes multiple modal data, the modal data includes voice information, and the modal data of at least two modalities corresponds to the same time segment; Determining, by means of the target model, a modal feature of the modal data and a feature set consisting of modal sub-features of the same time segment; wherein the modal feature includes a plurality of the modal sub-features; Determine a first weight corresponding to the modal sub-feature in the feature set, and fuse the modal sub-features in the feature set based on the first weight to obtain a fused feature; The speech information is determined based on the fusion features of the multiple time segments.
2. The method according to claim 1, characterized in that The determining a first weight corresponding to the modal sub-feature in the feature set includes: Performing convolution processing on the modal sub-feature in the feature set to obtain an intermediate feature corresponding to the modal sub-feature; The first weight is obtained by processing the product of the intermediate features through a first sigmoid function.
3. The method according to claim 1 or 2, characterized in that The multiple modal data are composed of first modal data and second modal data; the modal feature is composed of a first modal feature and a second modal feature, and a first modal sub-feature in the first modal feature and a second modal sub-feature in the second modal feature correspond to the same time segment; Then the first weight is calculated by the following formula: a1=sigmoid(A′1+β2A′2), a2=sigmoid(ω1A′1+A′2); Among them, a1 is the first weight corresponding to the first modal sub-feature, a2 is the first weight corresponding to the second modal sub-feature, A′1 is the first intermediate feature corresponding to the first modal sub-feature, A′2 is the second intermediate feature corresponding to the second modal sub-feature, β2 is the first preset hyperparameter, and ω1 is the second preset hyperparameter.
4. The method according to claim 1, characterized in that The plurality of modality data includes audio data; Then determining the modal features of the modal data includes: Inputting the audio data into at least two convolutional layers, and concatenating the audio convolution features output by the at least two convolutional layers to obtain concatenated features; wherein the at least two convolutional layers have different receptive fields; Performing a pooling operation on the spliced features in a channel dimension to obtain a channel descriptor of the spliced features; A second weight is determined by inputting the channel descriptor into two sequentially connected fully connected layers, and a product of the second weight and the concatenated feature is determined as a first modality feature corresponding to the audio data.
5. The method according to claim 4, characterized in that Performing a pooling operation on the splicing feature in the channel dimension to obtain a channel descriptor of the splicing feature includes: Performing an average pooling operation on the concatenated features in a channel dimension to obtain a first descriptor in the channel descriptor; A maximum pooling operation is performed on the concatenated features in the channel dimension to obtain a second descriptor in the channel descriptor.
6. The method according to claim 5, characterized in that The second weight is calculated by the following formula: Where γ is the second weight, W1 corresponds to the first fully connected layer, and W2 corresponds to the second fully connected layer. For the first descriptor, is the second descriptor.
7. The method according to any one of claims 1-2, 4-6, characterized in that: The plurality of modality data includes video data; Then determining the modal features of the modal data includes: Performing convolution processing on the lip image in the video data to obtain a global feature; wherein the global feature includes a plurality of global sub-features; Inputting the global feature into a feature slicing module to obtain a first number of the global sub-features, and subtracting the global feature from the first number of the global sub-features to obtain a differential feature; Performing convolution processing on the differential feature to obtain a differential convolution feature, and processing the differential convolution feature through a second sigmoid function to obtain a third weight; A Hadamard product of the third weight and the differential feature is determined as a second modality feature corresponding to the video data.
8. A speech recognition device, characterized in that: include: An input module, used to obtain data to be processed and input the data to be processed into a target model; wherein the data to be processed includes multiple modal data, the modal data includes voice information, and the modal data of at least two modalities corresponds to the same time segment; A feature module, used to determine the modal feature of the modal data and a feature set consisting of the modal sub-features of the same time segment through the target model; wherein the modal feature includes a plurality of the modal sub-features; a fusion module, configured to determine a first weight corresponding to the modal sub-feature in the feature set, and fuse the modal sub-features in the feature set based on the first weight to obtain a fused feature; A speech module is used to determine the speech information based on the fusion features of the multiple time segments.
9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.