Speech recognition method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of natural language processing, and provides a voice recognition method and device, electronic equipment and a storage medium, and the method comprises the steps: predicting the number of frame skips according to the coding state of a current frame and the decoding state of a previous frame in voice data; and under the condition that the number of the skipped frames indicates that the frames which do not need to be decoded exist, taking the frames corresponding to the number of the skipped frames as skipped frames, stopping decoding calculation of the skipped frames, determining the decoding state of the previous frame as the decoding state of the skipped frames, taking the next frame of a frame sequence formed by the skipped frames as the current frame, and returning to execute the prediction step. Until the decoding states of all frames are obtained; and determining a voice recognition result of the voice data according to the decoding states of all the frames. According to the method, invalid frames are predicted and directly skipped through joint coding and historical decoding states, so that the decoding rate is remarkably improved while memory occupation and computing power consumption are greatly reduced, and the problems of high resource consumption and low efficiency in long-sequence speech recognition are effectively solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, and in particular to a speech recognition method and device, electronic equipment and a storage medium. BACKGROUND

[0002] With the continuous development of artificial intelligence technology, speech recognition systems based on encoder-decoder architecture have been widely used in human-computer interaction and other scenarios.

[0003] In current speech recognition, the encoder usually converts the input speech signal sequence into an encoded state using a long short-term memory network, and the decoder then decodes frame by frame based on the encoded state. However, speech data often contains a large number of blank frames, and the above method requires decoding calculation for each frame and storing the corresponding hidden state. This frame-by-frame processing method consumes unnecessary computing power when processing a large number of blank frames, severely reducing the decoding rate, and the large amount of accumulated intermediate state also causes a huge memory occupation, especially in long sequence processing. SUMMARY

[0004] The present application provides a speech recognition method, device, electronic equipment and storage medium to solve the defects in the prior art.

[0005] The present application provides a speech recognition method, comprising the following steps: encoding the speech data to be recognized to obtain the encoded state corresponding to each frame of the speech data; predicting the number of skipped frames according to the encoded state of the current frame and the decoded state of the previous frame; in the case where the number of skipped frames indicates that there are frames that do not need to be decoded, the frames corresponding to the number of skipped frames are taken as skipped frames, the decoding calculation of the skipped frames is stopped, the decoded state of the previous frame is determined as the decoded state of the skipped frames, and the next frame of the frame sequence formed by the skipped frames is taken as the current frame, and the prediction step is returned to be executed until the decoded state of all frames is obtained; determining the speech recognition result of the speech data according to the decoded state of all frames.

[0006] According to the speech recognition method provided by the present application, the number of skipped frames is predicted according to the encoded state of the current frame and the decoded state of the previous frame, comprising: splicing the encoded state of the current frame and the decoded state of the previous frame to obtain a joint feature; based on an inference network, the joint feature is sequentially subjected to feature extraction and classification prediction to obtain a classification probability distribution of a plurality of candidate skipped frame numbers, and the number of skipped frames is determined from the plurality of candidate skipped frame numbers according to the classification probability distribution.

[0007] According to a speech recognition method provided by the present invention, the step of sequentially extracting features and classifying and predicting classifications from the joint features to obtain a classification probability distribution of the number of candidate skip frames includes: Feature extraction is performed on the joint features to obtain a high-dimensional feature representation; The probability of the classification is obtained by performing probability calculation on the high-dimensional feature representation.

[0008] According to a speech recognition method provided by the present invention, the inference network includes a shallow feature extraction module, at least two dense connection modules, a transition module connecting two adjacent dense connection modules, and a classification module connected in sequence. The shallow feature extraction module is used to extract shallow features from the joint features; Each densely connected module is used to perform deep feature extraction on the received input features; The transition module is used to adjust the output feature dimension of the preceding dense module in two adjacent dense connection modules so that the adjusted output feature dimension is consistent with the input feature dimension of the following dense module in two adjacent dense connection modules. The classification module is used to perform probability calculations based on the high-dimensional feature representation output by the last densely connected module to obtain the classification probability distribution.

[0009] According to a speech recognition method provided by the present invention, the dense connection module includes multiple processing layers, each processing layer is used to perform deep feature extraction on the received input features, and the input features of each layer include the output features of all layers before the current layer and the input features of the corresponding dense connection module.

[0010] According to a speech recognition method provided by the present invention, the training steps of the inference network include: Acquire sample speech data, and extract the encoding status of each frame in the sample speech data and the label indicating whether the decoding result of each frame is invalid; The actual number of skipped frames for each frame is determined by the number of subsequent frames that are consecutive in time from each frame and whose labels are all invalid. The encoding state of each frame and the decoding state of the previous frame are used as inputs to the inference network to obtain the predicted number of skip frames for each frame. Based on the difference between the predicted number of skipped frames and the actual number of skipped frames in each frame, the inference network is iteratively trained until a preset convergence condition is met.

[0011] According to a speech recognition method provided by the present invention, after predicting the number of frame skips based on the encoding state of the current frame and the decoding state of the previous frame, the method further includes: If the frame skipping count indicates that there are no frames that do not require decoding calculation, the encoding state of the current frame is decoded to obtain the decoding state of the current frame; Take the next frame of the current frame as the current frame, return to the prediction step, and continue until the decoding status of all frames is obtained.

[0012] The present invention also provides a voice recognition device, comprising the following units: The encoding unit is used to encode the speech data to be recognized to obtain the encoding state corresponding to each frame in the speech data; The prediction unit is used to predict the number of frame skips based on the encoding state of the current frame and the decoding state of the previous frame. The decoding unit is configured to, when the number of skipped frames indicates that there are frames that do not require decoding calculation, take the frame corresponding to the number of skipped frames as a skipped frame, stop the decoding calculation of the skipped frame, and after determining the decoding state of the previous frame as the decoding state of the skipped frame, take the next frame of the frame sequence formed by the skipped frames as the current frame, and return to execute the prediction step until the decoding state of all frames is obtained. The recognition unit is used to determine the speech recognition result of the speech data based on the decoding status of all the frames.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the above-described speech recognition methods.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the speech recognition method as described above.

[0015] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the speech recognition method as described above.

[0016] The speech recognition method, apparatus, electronic device, and storage medium provided by this invention jointly predict the number of consecutive frames, including the current frame, that do not require decoding calculations by utilizing the encoding state of the current frame and the decoding state of the previous frame. When the prediction result indicates the existence of skipped frames, the decoding calculation of the corresponding frame is directly stopped and the state of the previous frame is reused. Since invalid frames usually account for a large proportion of speech data, and this invention can accurately predict and skip these invalid frames in batches, it significantly reduces unnecessary decoding calculations and significantly improves the decoding speed. At the same time, by reusing states instead of storing independent states at each intermediate moment, memory consumption is effectively reduced, solving the problem of high memory consumption and slow decoding in long sequence processing of related technologies. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the speech recognition method provided by the present invention.

[0019] Figure 2 This is a schematic diagram of the speech recognition network provided by the present invention.

[0020] Figure 3 This is a schematic diagram of the transition module provided by the present invention.

[0021] Figure 4 This is a structural schematic diagram of the classification module provided by the present invention.

[0022] Figure 5 This is a schematic diagram of the inference network provided by the present invention.

[0023] Figure 6 This is a structural schematic diagram of the dense connection module provided by the present invention.

[0024] Figure 7 This is a schematic diagram of the training process of the inference network provided by the present invention.

[0025] Figure 8 This is a flowchart illustrating another speech recognition method provided by the present invention.

[0026] Figure 9 This is a schematic diagram of the structure of the voice recognition device provided by the present invention.

[0027] Figure 10 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0029] In speech recognition systems, the input speech signal is typically encoded by an encoding network to obtain an encoded state, then decoded by a decoding network, and the final decoded result is used as the speech recognition result. To more accurately capture the correspondence between the input and output sequences in tasks such as speech recognition, the decoding network obtains two states from the posterior calculation of the encoder: a valid state and a blank state. If the speech recognition model determines that the current frame lacks valid acoustic features, or if the state of this frame cannot be clearly identified as belonging to any specific phoneme, the model outputs a blank state; if the current frame contains valid acoustic features, the model outputs a valid state.

[0030] The encoder typically uses a Long Short-Term Memory (LSTM) network to convert the speech signal into a fixed-shape coded state. It reads one frame of the speech signal at each time step and updates its hidden state. The decoder uses the coded state of the current frame and the decoded state of the previous frame to predict the decoded state of the current frame.

[0031] However, before the decoder begins decoding, it cannot determine whether the current frame is in a valid state or a blank state. Since blank state frames constitute a large proportion of all frames, and each frame requires decoding and computation, and most decoded frames are invalid blank state frames, the computational data increases exponentially due to the need to store and process the decoding state of each frame. This results in a slower decoding speed as sentences become longer. Furthermore, after decoding, the decoding state of each frame needs to be stored for computation in subsequent frames, and the accumulation of these states leads to significant memory consumption.

[0032] To address this, the present invention provides a speech recognition method that, during the decoding process of the encoding state of each frame, predicts the number of consecutive frames, including the current frame, that do not require decoding calculation from the current frame onwards, based on the encoding state of the current frame and the decoding state of the previous frame. If the prediction result indicates the existence of a skipped frame, the decoding calculation of the corresponding frame is directly stopped and the decoding state of the previous frame is reused, thereby achieving rapid skipping of blank frames. This significantly reduces memory usage and eliminates the need to calculate invalid frames, thus significantly improving the decoding speed of speech recognition.

[0033] It should be noted that all actions involving the acquisition of signal information or data in this invention are carried out in compliance with the relevant data protection laws and regulations of the country where the invention is located, and with the authorization granted by the owner of the relevant device.

[0034] in, Figure 1 This is a flowchart illustrating the speech recognition method provided by the present invention, as shown below. Figure 1 As shown, the method includes steps 110, 120, 130 and 140.

[0035] Step 110: Encode the speech data to be recognized to obtain the encoding state corresponding to each frame in the speech data.

[0036] Here, the speech data to be recognized can be understood as the audio signal that needs to be processed for speech recognition. It can be a real-time speech stream input by the user or a pre-stored audio file. In order to convert continuous analog or digital audio signals into discrete feature sequences that can be processed by computer models, after acquiring the speech data to be recognized, preprocessing operations such as framing, windowing, and feature extraction can be performed to obtain an acoustic feature sequence containing time-frequency information, providing an effective data foundation for subsequent encoding and decoding processes.

[0037] Encoding speech data refers to using an encoding network to transform the feature sequence of the speech data into a higher-dimensional feature representation more suitable for subsequent processing. As an optional implementation, the encoding network can be a Long Short-Term Memory (LSTM) network, which can capture long-range dependencies in the speech data. At each time step, the encoding network reads one frame of feature data from the acoustic feature sequence and updates its internal hidden states. After processing by the encoding network, the speech data is converted into a sequence of encoded states, which is formed by arranging the encoded states corresponding to each frame of the speech data in chronological order.

[0038] In this context, the encoded state refers to the hidden state or feature vector output by the encoding network, which contains the acoustic feature information of the corresponding frame. The encoded state corresponding to each frame in the speech data is each element in the encoded state sequence, and they correspond one-to-one with each frame of the speech data in time steps. For example, if the speech data contains n frames, then after passing through the encoding network, n fixed-length encoded states can be encoded, corresponding to frames 1 to n respectively.

[0039] Step 120: Based on the encoding state of the current frame and the decoding state of the previous frame, predict the number of skipped frames. The number of skipped frames refers to the number of consecutive frames, including the current frame, from which no decoding calculation is required.

[0040] Specifically, decoding refers to the process of converting an encoded state into a target text or phoneme sequence using a decoding network. Decoding networks typically employ structures such as recurrent neural networks or LSTMs, starting from an initial state and generating an output sequence through iterative loops. During decoding, a decoding state needs to be maintained, which is continuously updated as time steps progress and serves as the input for the next time step. The decoding state can be understood as the hidden state or cell state in the decoding network.

[0041] The current frame refers to the frame corresponding to the time step currently being processed in the decoding loop. The previous frame refers to the frame corresponding to the immediately preceding time step. For the first frame, the decoding state of the previous frame can be initialized to a specific initial state, such as an all-zero vector or a preset start flag.

[0042] Predicting the number of skipped frames refers to using the encoding state of the current frame and the decoding state of the previous frame to pre-determine how many consecutive frames, starting from the current frame, will be invalid, thus allowing the tedious calculations for these frames to be skipped. The number of skipped frames refers to the number of consecutive frames, including the current frame, from which no decoding calculations are required.

[0043] Here, "no decoding calculation needed" usually means that the decoding result of the corresponding frame is an invalid state, such as a Blank state. In speech recognition tasks, the model may output a Blank state at certain time steps, indicating that there are no valid characters or that the character cannot be clearly assigned. Since the frames corresponding to Blank states account for a large proportion of all audio frames, if it is possible to predict which frames are Blank states before decoding, meaningless calculations can be avoided.

[0044] As an optional embodiment, a pre-built inference network can be used to receive the encoding state of the current frame and the decoding state of the previous frame as input, and output a scalar value or classification result as the predicted number of frame skips. This number of frame skips can be 0 or an integer greater than 0.

[0045] Step 130: If the number of skipped frames indicates that there are frames that do not require decoding calculation, take the frame corresponding to the number of skipped frames as the skipped frame, stop the decoding calculation of the skipped frame, and determine the decoding state of the previous frame as the decoding state of the skipped frame. Then, take the next frame of the frame sequence formed by the skipped frames as the current frame, return to the prediction step, and continue until the decoding state of all frames is obtained.

[0046] Specifically, if the number of skipped frames indicates the existence of frames that do not require decoding calculations, it means that the current frame is an invalid frame, and that there may also be invalid frames in the subsequent consecutive frames of the current frame. In other words, these frames that do not require decoding calculations are skipped frames. In this case, to avoid redundant processing of invalid information and thus save computing power and time, this embodiment stops calling the decoding network to perform complex decoding calculations for these skipped frames. This means that the decoding network does not need to perform time-consuming operations such as matrix multiplication and nonlinear activation to update its internal state.

[0047] For example, if the number of skipped frames is N and N>0, it means that the consecutive N frames starting from the current frame are invalid frames. In this case, the current frame and the consecutive N-1 frames after the current frame are treated as skipped frames.

[0048] Furthermore, to ensure the integrity and continuity of the decoding state sequence while stopping decoding calculations, it is necessary to determine the decoding state of these skipped frames. Considering that these skipped frames are predicted to be invalid states, meaning that their corresponding decoding outputs have not undergone substantial state transitions or information updates, the decoding state of the skipped frames is kept consistent with the decoding state of the previous frame. This significantly reduces memory usage by replacing complex decoding operations with simple state reuse.

[0049] For example, if the number of skipped frames is N, and the decoding state of the previous frame is d. t-1 Then the decoding state from the current frame t to frame t+N-1 is directly assigned the value d. t-1 .

[0050] After processing this batch of skipped frames, in order to continue the subsequent decoding process, the pointer needs to be moved to the next frame to be processed. Specifically, the next frame after the frame sequence consisting of skipped frames (i.e., frame t+N) is taken as the new current frame, and the prediction step described above is returned to be executed. This process will be continuously repeated until all frames have been processed, thereby obtaining a decoding state sequence of all frames with the same length as the input speech data. Here, the frame sequence consisting of skipped frames refers to the set of frames that are consecutive in time and covered by the predicted number of skipped frames.

[0051] It should be noted that if the predicted frame skipping count indicates that there are no frames that do not require decoding calculations (e.g., a frame skipping count of 0), it means that the current frame is either a valid frame or an undetermined frame. In this case, normal decoding calculations can be performed on the current frame, the decoding state can be updated, and the next frame can be used as the new current frame for further processing.

[0052] Step 140: Determine the speech recognition result of the speech data based on the decoding status of all frames.

[0053] Specifically, after obtaining the decoding state corresponding to each frame, a classifier can map these decoding states into a probability distribution of characters or phonemes. Finally, through a search algorithm or greedy decoding, the final text sequence is obtained from the probability distribution, which is the speech recognition result of the speech data.

[0054] The speech recognition method provided in this embodiment uses the encoding state of the current frame and the decoding state of the previous frame to jointly predict the number of consecutive frames, including the current frame, that do not require decoding calculations. When the prediction result indicates that there are skipped frames, the decoding calculation of the corresponding frame is stopped directly and the state of the previous frame is reused. Since invalid frames usually account for a large proportion of speech data, and this embodiment can accurately predict and skip these invalid frames in batches, it significantly reduces unnecessary decoding calculations and significantly improves the decoding speed. At the same time, by reusing the state instead of storing the independent state of each intermediate moment, memory consumption is effectively reduced, solving the problem of high memory consumption and slow decoding in long sequence processing of related technologies.

[0055] Considering that directly relying on isolated encoding or decoding states may not accurately determine whether the current frame is an invalid frame, in order to more fully capture the correlation between speech features and decoding states, thereby achieving accurate prediction of the number of skipped frames, this embodiment refines the specific implementation process for predicting the number of skipped frames.

[0056] As an optional embodiment, step 120, predicting the number of frame skips based on the encoding state of the current frame and the decoding state of the previous frame, includes: Step 121: Concatenate the encoding state of the current frame with the decoding state of the previous frame to obtain the joint feature.

[0057] Considering that the encoding state of the current frame represents the current acoustic input information, while the decoding state of the previous frame represents the historical semantic context information, fusing the two can provide a direct acoustic basis for the current prediction, and can also use historical context constraints to help determine whether the current moment is in an invalid state of pronunciation. Therefore, this embodiment adopts a concatenation method to generate joint features. The joint features contain the acoustic feature information of the current moment and the decoding semantic information of the previous moment, thus forming a complete contextual basis for predicting the state of future frames.

[0058] As an optional embodiment, the feature vector corresponding to the encoding state of the current frame and the feature vector corresponding to the decoding state of the previous frame can be directly concatenated in the feature dimension direction to obtain joint features.

[0059] For example, suppose the encoding state of the current frame is e t The decoding state of the previous frame is d. t-1 Then, they can be combined into a longer vector using vector concatenation operations [e]. t d t-1 ], which is a joint feature.

[0060] Step 122: Based on the inference network, perform feature extraction and classification prediction on the joint features in sequence to obtain the classification probability distribution of the number of candidate skip frames, and determine the number of skip frames from the number of candidate skip frames according to the classification probability distribution.

[0061] Considering the powerful nonlinear mapping and pattern recognition capabilities of neural networks, this embodiment introduces a specially designed inference network to perform the prediction task in order to extract the implicit frame skipping patterns from the joint features. Here, the inference network can be understood as a pre-trained neural network model whose input is the aforementioned joint features and whose output is the predicted number of frame skips.

[0062] Specifically, performing feature extraction and classification prediction on the joint features in sequence means that the inference network performs deep abstraction and transformation on the input joint features to extract a high-dimensional feature representation that can characterize the length of future invalid frames; then, the high-dimensional feature is mapped onto a preset category space.

[0063] The number of candidate frame skips refers to a predefined set of possible frame skip step sizes. According to research and experimental statistics, invalid states usually appear within a specific length range after valid frames. Furthermore, based on actual decoding statistics, invalid states usually appear in the 1st, 2nd, 4th, or 8th frame after the valid frames. That is, the probability of invalid states appearing in this frame number is the highest. Therefore, in this embodiment, the set of multiple candidate frame skips is preferably {0, 1, 2, 4, 8}.

[0064] The classification probability distribution refers to the prediction confidence for each of the above candidate frame skipping numbers. For example, the inference network may output a probability vector P=[p0, p1, p2, p4, p8] containing 5 elements, where pi represents the probability of skipping number i.

[0065] After obtaining the classification probability distribution, the number of skip frames is determined from multiple candidate skip frame numbers based on the classification probability distribution. Usually, the candidate skip frame number with the highest probability value can be selected as the final number of skip frames. For example, if p4 has the highest value in the probability distribution, then the current number of skip frames is determined to be 4.

[0066] In this embodiment, the entire process of predicting the number of frame skips can be formally described as: e represents the encoding state e of the current frame. t Decoding state d of the previous frame t-1 The input is the inference network, which outputs the number of skipped frames, N, where N ∈ {0, 1, 2, 4, 8}. If N = 0, the current frame is valid and no skipping is performed; if N > 0, the current frame and the subsequent N-1 frames are invalid and skipping is performed. This classification and prediction method can flexibly and accurately adapt to silence or invalid segments of different lengths in the speech signal, significantly improving decoding efficiency.

[0067] in, Figure 2 This is a schematic diagram of the speech recognition network provided by the present invention, as shown below. Figure 2 As shown, the speech recognition network includes an encoding network, an inference network, and a decoding network. The encoding network encodes the speech data to be recognized, obtaining the encoding state corresponding to each frame in the speech data. The inference network receives the encoding state of the current frame and the decoding state of the previous frame, and predicts the number of consecutive frames, including the current frame, that do not require decoding calculation from the current frame onwards, i.e., the number of skipped frames. The decoding network stops calculation and reuses the state of the previous frame when the number of skipped frames predicted by the inference network indicates the existence of invalid frames, or performs normal decoding calculation on the encoding state of the current frame to update the decoding state when the number of skipped frames indicates the existence of invalid frames.

[0068] Considering that directly mapping from the original joint features to the final classification result may be too far off and difficult to capture complex nonlinear relationships, in order to improve the expressive power and prediction accuracy of the inference network, this embodiment clearly divides the inference process into a feature extraction stage and a probability calculation stage.

[0069] Specifically, the above-mentioned joint features are sequentially extracted and classified for prediction to obtain the classification probability distribution of multiple candidate skip frames, including: Feature extraction is performed on the joint features to obtain a high-dimensional feature representation; The probability of the classification is obtained by performing probability calculation on the high-dimensional feature representation.

[0070] Considering that joint features are simply concatenated from encoded and decoded states, although they contain necessary information, their semantic level is low and they contain a large amount of raw data details. In order to extract abstract features that play a key role in determining the length of invalid frames from these low-level features, deep feature extraction is required.

[0071] Here, feature extraction of the joint features can be achieved by utilizing the nonlinear transformation capability of neural networks to map the input joint features to a higher-dimensional, more abstract feature space. The high-dimensional feature representation is the output of this mapping process, which, compared to the original joint features, more robustly characterizes the temporal dependence and state attributes of the speech signal at the current moment.

[0072] Furthermore, considering that predicting the number of skipped frames is essentially a classification problem—that is, determining whether the current moment belongs to the category of "skipping 0 frames," "skipping 1 frame," or "skipping N frames"—while high-dimensional feature representations contain rich discriminative information, they are still in an abstract feature space and cannot directly reflect the confidence level of specific category affiliation. However, by performing probability calculations on high-dimensional feature representations, these abstract features can be quantitatively mapped to a predefined candidate category space, quantifying the degree of confidence in predicting the number of skipped frames for each candidate, thereby obtaining an intuitive classification probability distribution.

[0073] Here, probabilistic calculation of the high-dimensional feature representation refers to mapping the abstract feature vector to the category space of the candidate skip frames and calculating the probability of belonging to each category. The classification probability distribution is a set of values, where each value represents the probability that the corresponding candidate skip frame is predicted to be selected.

[0074] As an optional implementation, a fully connected layer can be used to linearly weight the high-dimensional feature representation, transforming the feature dimension into a category dimension. For example, if there are 5 candidate skip frames, it can be transformed into a vector of dimension 5. Then, the Softmax function is used to normalize the transformed vector to obtain the classification probability distribution of all candidate skip frames.

[0075] Considering that a single feature extraction layer often struggles to capture both shallow details and deep semantics of features, and that gradient vanishing or low feature utilization can easily occur as network depth increases, this embodiment further clarifies the specific composition structure of the inference network in order to construct an efficient inference network architecture that can fully utilize multi-level features.

[0076] As an optional embodiment, the inference network includes a shallow feature extraction module, at least two dense connection modules, a transition module connecting two adjacent dense connection modules, and a classification module connected in sequence. The shallow feature extraction module is used to extract shallow features from joint features; Each densely connected module is used to perform deep feature extraction on the received input features; The transition module is used to adjust the output feature dimension of the preceding dense module in two adjacent dense connection modules so that the adjusted output feature dimension is consistent with the input feature dimension of the following dense module in two adjacent dense connection modules. The classification module is used to calculate the probability based on the high-dimensional feature representation output by the last densely connected module, and obtain the classification probability distribution.

[0077] Considering that the joint features have high dimensionality and may contain redundant information, direct deep processing would lead to excessive computation and difficulty in capturing basic features. Therefore, this embodiment sets up a shallow feature extraction module to perform preliminary dimensionality reduction and feature abstraction on the joint features.

[0078] Here, the shallow feature extraction module is used to perform shallow feature extraction on the joint features. This means that this module is mainly responsible for extracting low-level, local acoustic semantic association features and reducing the size of the feature map.

[0079] The shallow feature extraction module can be composed of a convolutional layer and a max-pooling layer connected in series. The joint features are first input into the convolutional layer, where the convolutional kernel performs filtering to extract features; then they enter the max-pooling layer, where downsampling is used to retain the strongest feature responses in local regions while reducing the data dimensionality.

[0080] Considering the efficiency of information transmission and the backpropagation of gradients in deep networks, this embodiment introduces a dense connection module in order to maximize feature reuse and improve the network's ability to distinguish frame skipping patterns.

[0081] Here, each densely connected module is used to perform deep feature extraction on the received input features. The core mechanism of the densely connected module lies in the fact that it adopts a dense connection method inside the module, that is, the input of each layer comes from the output of all previous layers. This structure can enhance feature propagation, encourage feature reuse, and reduce the number of parameters.

[0082] Specifically, the beneficial effect of dense connections lies in breaking the limitation of traditional networks where layers only transmit the output of the current layer. By establishing direct connections across layers, back-end layers can directly see the features of front-end layers. This not only alleviates the vanishing gradient problem common in deep network training, as gradients can be directly propagated to shallower layers through cross-layer connections, but also greatly improves the efficiency of feature utilization. For example, when determining whether a frame is skipped, it is necessary to use both shallow detailed features to identify phoneme boundaries and deep semantic features to determine whether it is a silent segment. Dense connections enable the simultaneous fusion and utilization of these features from different levels for comprehensive decision-making, thereby significantly improving prediction accuracy.

[0083] For example, suppose a densely connected module contains three processing layers (Layer 1, Layer 2, Layer 3). In dense connection mode, the output of Layer 1 serves not only as the input of Layer 2 but also directly as the input of Layer 3; similarly, the input of Layer 2 includes not only the output of Layer 1 but also the initial input of the module; the input of Layer 3 is composed of the initial input of the module, the output of Layer 1, and the output of Layer 2.

[0084] Considering that densely connected modules significantly increase the dimensionality of the output features by concatenating features along the channel dimension, directly connecting the output of one densely connected module to the next would not only lead to dimensionality mismatch but also greatly increase the computational burden. Therefore, transition modules are needed between adjacent densely connected modules.

[0085] Here, the transition module is used to adjust the output feature dimension of the preceding dense module in two adjacent dense connection modules so that the adjusted output feature dimension is consistent with the input feature dimension of the following dense module in two adjacent dense connection modules. That is, the transition module mainly plays the role of connection, dimensionality reduction and dimension adaptation.

[0086] in, Figure 3 This is a schematic diagram of the transition module provided by the present invention, as shown below. Figure 3 As shown, the transition module can include a batch normalization layer, an activation function layer, a convolutional layer (such as a 1×1 convolution), and an average pooling layer (such as a 2×2 pooling) connected in sequence. The output features from the previous densely connected module enter the batch normalization layer to normalize the feature distribution. Then, the normalized features pass through the activation function layer, where a non-linear transformation enhances the expressive power of the features. Next, the non-linearly transformed features enter the convolutional layer, which primarily compresses the number of feature channels, thereby adjusting the output feature dimension to match the input feature dimension of the next densely connected module, solving the dimensionality expansion problem caused by dense connections. Finally, the dimensionally adjusted features enter the average pooling layer, where downsampling operations (such as 2×2 region averaging) reduce the spatial size of the feature map, further compressing the data volume and retaining the average response with background information. The processed features are then output to the next densely connected module.

[0087] Considering that the features obtained after processing by the above multiple modules are high-dimensional abstract representations, a classification module is set up in this embodiment to transform them into specific prediction results, i.e., determining how many frames to skip. Here, the classification module is used to perform probability calculations based on the high-dimensional feature representation output by the last densely connected module to obtain the classification probability distribution. This module is responsible for mapping the feature space to the class probability space.

[0088] in, Figure 4 This is a structural diagram of the classification module provided by the present invention, as shown below. Figure 4 As shown, the classification module can include a globally average pooling layer, two fully connected layers, and an activation function layer connected in sequence. The high-dimensional feature representation from the output of the last densely connected module enters the globally average pooling layer. This layer performs global average pooling on each feature channel, compressing the spatial feature map into a one-dimensional feature vector, thus significantly reducing the number of parameters and preventing overfitting. Then, the processed feature vector passes through two fully connected layers (FC). The first fully connected layer performs preliminary linear mapping and combination of the features. Next, the second fully connected layer maps the features to the class space, outputting values ​​corresponding to the number of candidate skip frames. Finally, the output of the second fully connected layer enters the Softmax activation function layer, which converts the input values ​​into a probability distribution—a classification probability distribution for the number of candidate skip frames—allowing the final number of skip frames to be determined based on the term with the highest probability in this distribution.

[0089] As an optional embodiment, Figure 5 This is a schematic diagram of the inference network provided by the present invention, as shown below. Figure 5 As shown, the inference network can include convolutional layers, max pooling layers, a first dense connection module, a transition module, a second dense connection module, and a classification module connected in sequence. The workflow of the inference network is as follows: First, the joint features are input into the convolutional layer for preliminary feature extraction to capture the local patterns of the joint features and obtain preliminary features; then, the preliminary features are entered into the max pooling layer for pooling, and the dimensionality of the preliminary features is reduced by downsampling while retaining the most significant feature information.

[0090] Then, the pooled preliminary features sequentially enter the first dense connection module, the transition module, and the second dense connection module. Specifically, the pooled preliminary features first undergo deep feature extraction in the first dense connection module; then, they enter the transition module, which connects adjacent dense connection modules and adjusts the output feature dimension of the first dense connection module to match the dimension of the feature vector of the second dense connection module, ensuring dimensionality matching; subsequently, the features with adjusted dimensions enter the second dense connection module for further deep feature extraction, resulting in a high-dimensional feature representation containing rich semantic information.

[0091] Finally, the high-dimensional feature representation output by the second densely connected module enters the classification module. The fully connected layer and the Softmax activation function in the classification module process the high-dimensional feature representation in turn, and finally output the classification probability distribution for the number of candidate skip frames, thereby completing the prediction of the number of skip frames.

[0092] Considering that traditional convolutional neural networks are prone to information and gradient loss during transmission as the number of layers increases, resulting in difficult network training and low feature utilization, this embodiment specifies the internal structure of the dense connection module in order to maximize feature reuse and improve the inference network's ability to capture subtle changes and long-term dependencies in speech data.

[0093] Specifically, in this embodiment, the dense connection module includes multiple processing layers. Each processing layer is used to perform deep feature extraction on the received input features, and the input features of each layer include the output features of all layers before the current layer and the input features of the corresponding dense connection module.

[0094] Considering that a single or simple network layer is often insufficient to extract highly discriminative deep semantic features from complex joint features, this embodiment sets up multiple processing layers inside the dense connection module.

[0095] Here, the processing layer can be understood as the basic unit that performs specific feature transformation operations, typically including operations such as normalization, nonlinear activation, and convolution. Each processing layer is used to perform deep feature extraction on the received input features, that is, through multi-level nonlinear mapping, gradually extracting abstract patterns that are highly relevant to the frame skipping prediction task from the features.

[0096] Considering that maintaining the smooth flow of information in deep networks is crucial, this embodiment adopts a densely connected topology to solve the gradient vanishing problem and improve parameter efficiency. That is, the input features of each layer include the output features of all layers before the current layer and the input features of the corresponding densely connected modules.

[0097] Here, the input features of each layer include the output features of all layers preceding the current layer as well as the input features of the corresponding densely connected modules. This means that for the l-th layer, its input includes not only the output of the (l-1)-th layer, but also all the outputs of the (l-2)-th, ..., and the 0th layer, as well as the input features of the corresponding densely connected modules.

[0098] Through this densely connected topology, each layer in the densely connected module can directly access the original input and features from all previous layers, allowing information to flow more smoothly within the network. Furthermore, it avoids relearning redundant features at each layer; instead, features extracted from previous layers can be reused, achieving higher performance with fewer parameters. Building on this, the gradient of the loss function can be directly propagated to previous layers through cross-layer connections, effectively mitigating the vanishing gradient problem and making deep networks easier to train.

[0099] in, Figure 6 This is a structural schematic diagram of the dense connection module provided by the present invention, as shown below. Figure 6As shown, the dense connection module includes a first processing layer and a second processing layer. The first processing layer includes a cascaded batch normalization layer, an activation function layer, and a 1×1 convolutional layer. The second processing layer includes a cascaded batch normalization layer, an activation function layer, and a 3×3 convolutional layer. The workflow of this module is as follows: First, the input features enter the batch normalization layer of the first processing layer. This layer separates the mean and variance from the weights, normalizes the input features, and makes the loss function curve smoother, thus making it easier for the feature distribution and weights to match and accelerating network training convergence. Then, the normalized features pass through the activation function layer of the first processing layer. This layer introduces non-linear factors, which can provide the model with more powerful learning and expressive capabilities, and enhance the network's fitting of complex patterns. Next, the features enter a 1×1 convolutional layer to reduce the feature dimension, reduce the amount of computation and the number of parameters, and increase non-linear combinations.

[0100] The data then undergoes secondary normalization and non-linear activation through the batch normalization layer and activation function layer of the second processing layer to further optimize the feature distribution and enhance feature representation capabilities. Finally, the features are fed into a 3×3 convolutional layer for deep spatial feature extraction, capturing contextual information within a larger receptive field.

[0101] Considering that the predictive ability of the inference network directly determines the efficiency and accuracy of speech recognition, supervised training is required to enable the inference network to accurately predict the number of future frame skips based on the current encoding and decoding states. This embodiment describes the training process of the inference network in detail.

[0102] Specifically, the training steps for an inference network include: Acquire sample speech data and extract the encoding status of each frame in the sample speech data and the label indicating whether the decoding result of each frame is invalid. The actual number of skipped frames for each frame is determined by the number of subsequent frames that are consecutive in time from each frame and whose labels are all invalid. The encoding state of each frame and the decoding state of the previous frame are used as inputs to the inference network to obtain the predicted number of skip frames for each frame. Based on the difference between the predicted number of skipped frames and the actual number of skipped frames in each frame, the inference network is iteratively trained until the preset convergence condition is met.

[0103] Considering that the quality of training data and the accuracy of annotation are the foundation of model training, the original sample speech data needs to be processed in order to construct a dataset containing real input and supervision signals.

[0104] Here, the sample speech data can be recording data from various scenarios. The invalid state label refers to a marker used to identify whether a frame corresponds to a blank state after decoding.

[0105] As an alternative implementation, tags can be extracted using a raw Encoder-Decoder model without frame skipping. Specifically, the correctly predicted sentences from the raw model (Encoder and Decoder) are extracted as the initial audio, and the encoding state of each frame of the audio is collected. Simultaneously, the encoding state of each frame and the corresponding frame's tag are manually labeled. For example, if a frame outputs "Blank" after decoding, its tag is invalid; if it outputs a valid character, its tag is valid.

[0106] Since the training objective of the inference network is to predict the number of frame skips, and the original labels only represent the state of a single frame, it is necessary to convert the single-frame labels into a supervisory signal for the number of frame skips at the current moment. Here, the actual number of frame skips refers to the number of consecutive invalid frames following the current frame.

[0107] This can be done by iterating through the tags of subsequent frames for each frame. For example, for frame i, if the tags of frame i and the subsequent frames i+1, ..., i+N-1 are all invalid, while frame i+N is valid, then the actual number of skipped frames corresponding to frame i is N.

[0108] Considering that the network needs to learn the mapping relationship between input and output during the training phase, the prepared input data needs to be fed into the inference network to be trained for forward propagation. Based on this, this embodiment uses the encoding state of each frame and the decoding state of the previous frame as the input to the inference network to obtain the predicted number of frame skips for each frame.

[0109] As an optional implementation, the above sample speech data can be divided into a training set, a test set, and a validation set, such as in an 8:1:1 ratio. The encoding and decoding states from the training set are input into the inference network, and after feature extraction and classification layer processing, the predicted number of frame skips for each sample is obtained.

[0110] Considering that training a neural network is essentially a process of minimizing prediction error, the network parameters can be continuously updated by calculating the differences and backpropagating the gradient, making its prediction results closer to the true values. To enable the inference network to accurately learn the nonlinear mapping relationship between the encoding state, historical decoding state, and the number of future invalid frames, thereby robustly predicting the optimal frame skipping strategy in practical applications, this embodiment iteratively trains the inference network based on the difference between the predicted number of skipped frames and the actual number of skipped frames for each frame, until a preset convergence condition is met.

[0111] Here, the difference can be measured using a loss function. Iterative training refers to updating parameters through multiple forward and backward propagations. Preset convergence conditions could be that the loss value is below a threshold, the number of iterations reaches an upper limit, or performance on the validation set no longer improves.

[0112] As an alternative implementation, the cross-entropy loss function can be used to measure the above differences. The loss function can be expressed as: in, This represents the value of the loss function. Indicates the batch size. This represents the dimension of the data, corresponding to the number of categories. One-hot encoding that represents the true distribution of data, i.e., the actual number of frame skips. This represents the predicted classification probability distribution.

[0113] in, Figure 7 This is a schematic diagram of the training process of the inference network provided by the present invention, as shown below. Figure 7 As shown, the dataset is first constructed and partitioned. Specifically, the sentences correctly predicted by the encoding and decoding networks are extracted as sample speech data. The encoding state of each frame of the sample speech data is collected, and the encoding state of each frame and the actual number of skipped frames are manually labeled. The above dataset is divided into training, testing, and validation sets according to a preset ratio (e.g., 8:1:1). Subsequently, the inference network is initialized, setting initial weights and biases for the convolutional and fully connected layers in the inference network.

[0114] Next, the training set is input into the inference network, and the training loss is calculated. Here, the cross-entropy loss function is used to measure the difference between the probability distribution predicted by the network and the true distribution of the data. After calculating the training loss, it is determined whether an iteration cycle has been completed. If not, it is further determined whether half of the iteration cycles have been completed.

[0115] Next, if less than half of the iterations have been completed, gradient backpropagation is performed directly to update the inference network parameters and return to continue calculating the training loss. If half of the iterations have been completed, the validation set is used to perform a phased evaluation of the inference network and determine whether overfitting has occurred. If overfitting has occurred, the process shifts to adjusting the hyperparameters and re-initializing the inference network; if no overfitting has occurred, gradient backpropagation continues.

[0116] Finally, upon completion of the iteration cycle, the test set is input for final evaluation, and the training loss is recalculated. At this point, it is determined whether the inference network performance meets the requirements. If it does not, the validation set is input again for analysis and hyperparameter adjustment, and training restarts; if it meets the requirements, the network parameters are determined, with the set of parameters that minimizes the loss value fixed as the final network parameters.

[0117] Considering that speech signals contain not only numerous silences or invalid segments but also valid speech segments carrying core semantic information, in order to ensure the integrity and accuracy of speech recognition, the system must not only have the ability to quickly skip invalid frames but also be able to accurately decode valid frames. Therefore, this embodiment supplements the processing logic when the prediction result indicates that frame skipping is not required, to form a complete closed-loop decoding process.

[0118] Specifically, in this embodiment, after predicting the number of frame skips based on the encoding state of the current frame and the decoding state of the previous frame, the method further includes: If there are no frames that do not require decoding calculation as indicated by the number of skipped frames, the decoding calculation is performed on the encoding state of the current frame to obtain the decoding state of the current frame; Take the next frame of the current frame as the current frame, return to perform the prediction step, and continue until the decoding status of all frames is obtained.

[0119] When the frame skipping count indicates the absence of frames that do not require decoding calculations, it means the current frame is likely to contain valid acoustic features, or it cannot be definitively identified as an invalid frame. In this case, skipping is not allowed; the decoding network must be invoked for normal computation to extract semantic information. A frame skipping count of 0 typically indicates the absence of frames that do not require decoding calculations.

[0120] Decoding the encoding state of the current frame means inputting the encoding state of the current frame and the decoding state of the previous frame into the decoding network, and outputting the hidden state at the current moment through the internal calculations of the decoding network.

[0121] As an optional embodiment, after receiving input, the decoding network updates its hidden state and cell state, and outputs the decoding state of the current frame. This decoding state is not only used to generate the predicted character at the current moment, but also serves as the input to the inference network and decoding network at the next moment, maintaining the contextual continuity of the decoding process.

[0122] Considering that speech recognition is a step-by-step processing of time sequences, after processing the current valid frame, it is necessary to continue processing subsequent audio frames until the entire speech segment is processed.

[0123] Based on this, after obtaining the decoding status of the current frame, this embodiment takes the next frame of the current frame as the current frame and returns to execute the prediction step until the decoding status of all frames is obtained.

[0124] Here, taking the next frame as the current frame means moving the time step pointer forward by one position. Returning to the prediction step means going back to step 120 and performing a new round of frame skipping prediction using the new current frame encoding state and the just-updated decoding state.

[0125] If the next frame is predicted to be invalid, the frame skipping process begins; if the next frame is still valid, normal decoding continues. This iterative process continues until the last frame of the input speech data has been processed, thus obtaining a complete decoding state sequence corresponding to all frames.

[0126] in, Figure 8 This is a flowchart illustrating another speech recognition method provided by the present invention, as shown below. Figure 8 As shown, firstly, the speech data to be recognized enters the coding network, and after coding processing, a fixed-length coded state sequence is obtained.

[0127] Then, the encoded state of the current frame and the decoded state of the previous frame output by the decoding network are concatenated and used as input to the inference network. Next, after feature extraction and classification prediction by the inference network, a predicted number of skipped frames N is obtained. This value indicates whether the next N frames are invalid blank states. Subsequently, based on the predicted N value, a judgment is made. If N=0, it means that the current frame is predicted to be a valid frame. At this time, the encoded state of the current frame and the decoded state of the previous frame are input to the decoding network. After decoding by the decoding network, the decoded state of the current frame is generated. The next N frames include the current frame and the N-1 frames following the current frame.

[0128] If N > 0, it means that no valid state will be decoded in the next N frames, i.e., all will be in a blank state, so no decoding calculation is performed. In this case, the decoding state of the previous frame is directly determined as the decoding state of these N frames, and the state of the decoding network is not updated; the processing of these N frames is skipped.

[0129] Finally, the Nth frame after the current frame is taken as the new current frame, and the above steps are repeated until the entire encoded state sequence is processed, thus obtaining the decoding state of the entire sequence output, which is the speech recognition result.

[0130] The speech recognition device provided by the present invention is described below. The speech recognition device described below and the speech recognition method described above can be referred to in correspondence.

[0131] Based on any of the above embodiments Figure 9 This is a schematic diagram of the structure of the voice recognition device provided by the present invention, as shown below.Figure 9 As shown, the device includes: The encoding unit 910 is used to encode the speech data to be recognized to obtain the encoding state corresponding to each frame in the speech data; The prediction unit 920 is used to predict the number of frame skips based on the encoding state of the current frame and the decoding state of the previous frame. The number of frame skips refers to the number of consecutive frames, including the current frame, from which no decoding calculation is required. The decoding unit 930 is used to, when the number of skipped frames indicates that there are frames that do not need to be decoded, take the frame corresponding to the number of skipped frames as a skipped frame, stop the decoding calculation of the skipped frame, and after determining the decoding state of the previous frame as the decoding state of the skipped frame, take the next frame of the frame sequence formed by the skipped frames as the current frame, return to execute the prediction step, until the decoding state of all frames is obtained. The recognition unit 940 is used to determine the speech recognition result of the speech data based on the decoding status of all frames.

[0132] Figure 10 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 10 As shown, the electronic device may include a processor 1010, a communications interface 1020, a memory 1030, and a communication bus 1040. The processor 1010, communications interface 1020, and memory 1030 communicate with each other via the communication bus 1040. The processor 1010 can call logical instructions from the memory 1030 to execute a speech recognition method.

[0133] Furthermore, the logical instructions in the aforementioned memory 1030 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0134] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program, the computer program being stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer is able to execute the speech recognition methods provided by the above methods.

[0135] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the speech recognition methods provided by the methods described above.

[0136] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0137] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0138] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A speech recognition method, characterized in that, include: The speech data to be recognized is encoded to obtain the encoding state corresponding to each frame in the speech data; Predict the number of skipped frames based on the encoding status of the current frame and the decoding status of the previous frame; When the number of skipped frames indicates that there are frames that do not require decoding calculation, the frame corresponding to the number of skipped frames is taken as a skipped frame, the decoding calculation of the skipped frame is stopped, and the decoding state of the previous frame is determined as the decoding state of the skipped frame. Then, the next frame of the frame sequence formed by the skipped frames is taken as the current frame, and the prediction step is returned to be executed until the decoding state of all frames is obtained. Based on the decoding status of all frames, the speech recognition result of the speech data is determined.

2. The speech recognition method according to claim 1, characterized in that, The step of predicting the number of skipped frames based on the encoding state of the current frame and the decoding state of the previous frame includes: The encoding state of the current frame is concatenated with the decoding state of the previous frame to obtain a joint feature; Based on the inference network, feature extraction and classification prediction are performed sequentially on the joint features to obtain the classification probability distribution of the number of candidate skip frames, and the number of skip frames is determined from the number of candidate skip frames according to the classification probability distribution.

3. The speech recognition method according to claim 2, characterized in that, The process of sequentially extracting features and classifying and predicting classifications from the joint features yields a classification probability distribution for the number of candidate skip frames, including: Feature extraction is performed on the joint features to obtain a high-dimensional feature representation; The probability of the classification is obtained by performing probability calculation on the high-dimensional feature representation.

4. The speech recognition method according to claim 2, characterized in that, The inference network includes a shallow feature extraction module, at least two dense connection modules, a transition module connecting two adjacent dense connection modules, and a classification module connected in sequence. The shallow feature extraction module is used to extract shallow features from the joint features; Each densely connected module is used to perform deep feature extraction on the received input features; The transition module is used to adjust the output feature dimension of the preceding dense module in two adjacent dense connection modules so that the adjusted output feature dimension is consistent with the input feature dimension of the following dense module in two adjacent dense connection modules. The classification module is used to perform probability calculations based on the high-dimensional feature representation output by the last densely connected module to obtain the classification probability distribution.

5. The speech recognition method according to claim 4, characterized in that, The dense connection module includes multiple processing layers. Each processing layer is used to perform deep feature extraction on the received input features. The input features of each layer include the output features of all layers before the current layer and the input features of the corresponding dense connection module.

6. The speech recognition method according to claim 2, characterized in that, The training steps for the inference network include: Acquire sample speech data, and extract the encoding status of each frame in the sample speech data and the label indicating whether the decoding result of each frame is invalid; The actual number of skipped frames for each frame is determined by the number of subsequent frames that are consecutive in time from each frame and whose labels are all invalid. The encoding state of each frame and the decoding state of the previous frame are used as inputs to the inference network to obtain the predicted number of skip frames for each frame. Based on the difference between the predicted number of skipped frames and the actual number of skipped frames in each frame, the inference network is iteratively trained until a preset convergence condition is met.

7. The speech recognition method according to any one of claims 1 to 6, characterized in that, After predicting the number of frame skips based on the encoding state of the current frame and the decoding state of the previous frame, the method further includes: If the frame skipping count indicates that there are no frames that do not require decoding calculation, the encoding state of the current frame is decoded to obtain the decoding state of the current frame; Take the next frame of the current frame as the current frame, return to the prediction step, and continue until the decoding status of all frames is obtained.

8. A voice recognition device, characterized in that, include: The encoding unit is used to encode the speech data to be recognized to obtain the encoding state corresponding to each frame in the speech data; The prediction unit is used to predict the number of frame skips based on the encoding state of the current frame and the decoding state of the previous frame. The decoding unit is configured to, when the number of skipped frames indicates that there are frames that do not require decoding calculation, take the frame corresponding to the number of skipped frames as a skipped frame, stop the decoding calculation of the skipped frame, and after determining the decoding state of the previous frame as the decoding state of the skipped frame, take the next frame of the frame sequence formed by the skipped frames as the current frame, and return to execute the prediction step until the decoding state of all frames is obtained. The recognition unit is used to determine the speech recognition result of the speech data based on the decoding status of all the frames.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the speech recognition method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the speech recognition method as described in any one of claims 1 to 7.

Images