A streaming speech keyword recognition method, device, equipment and storage medium

Abstract

The application provides a streaming speech keyword recognition method, device, equipment and storage medium, including training stage and inference stage; the training stage includes: constructing a convolutional neural network model, the convolutional neural network model includes multiple convolutional layers; the convolutional neural network model is trained, and the training includes: first edge processing is carried out on the input feature map of each convolutional layer in each convolutional layer, and the first edge processing is based on causality constraint; the inference stage includes: obtaining the streaming speech signal to be recognized, and the fixed length audio signal is intercepted each time; the audio signal intercepted each time is input into the trained convolutional neural network model, and second edge processing is carried out on the input feature map of each convolutional layer in each convolutional layer, and the second edge processing is based on historical state; according to the output of the convolutional neural network model, the speech keyword recognition result is obtained. By using the above method, the recognition accuracy can be improved while ensuring real-time.

Description

Technical Field

[0001] This application relates to the field of intelligent voice interaction technology, and more specifically, to a streaming voice keyword recognition method, apparatus, device, and storage medium. Background Technology

[0002] Intelligent voice interaction technology is applied to streaming voice keyword recognition scenarios in embedded IoT devices such as smart speakers and in-vehicle systems. Due to resource and power consumption limitations, real-time detection and accurate recognition of voice keywords have become a requirement for the technology to be implemented.

[0003] In existing technologies, basic streaming speech keyword recognition often adopts a simple convolutional neural network (CNN) scheme: the model is built only by basic convolutional layers and fully connected layers, long audio data is used in the training stage, and short audio segments are directly extracted and input in the inference stage. The convolutional layer padding adopts the simple method of uniformly filling zero values ​​in all dimensions (top, bottom, left, and right), without optimization design for streaming scenarios.

[0004] The scheme has obvious drawbacks: full-dimensional zero-value padding does not match the scenario where streaming recognition can only obtain limited context. Zero-value padding cannot truly reflect the distribution of speech endpoint features, causing significant boundary effects, resulting in a high false detection rate of keyword endpoints and an accuracy that is difficult to meet the needs of practical applications. Summary of the Invention

[0005] In view of this, the purpose of this application is to provide a streaming speech keyword recognition method, apparatus, device and storage medium that can improve recognition accuracy while ensuring real-time performance.

[0006] In a first aspect, embodiments of this application provide a streaming speech keyword recognition method, the method including a training phase and an inference phase;

[0007] The training phase includes:

[0008] Construct a convolutional neural network model, wherein the convolutional neural network model includes multiple convolutional layers;

[0009] The convolutional neural network model is trained, and the training includes: performing a first edge-padding process on the input feature map of each convolutional layer, wherein the first edge-padding process is based on causal constraints;

[0010] The reasoning phase includes:

[0011] Acquire the streaming speech signal to be recognized, and extract a fixed length of audio signal each time;

[0012] Each extracted audio signal is input into the trained convolutional neural network model, and a second edge-padding process is performed on the input feature map of each convolutional layer, which is based on historical states.

[0013] Based on the output of the convolutional neural network model, the speech keyword recognition result is obtained.

[0014] Optionally, the first edge-padding process performed on the input feature map of each convolutional layer includes:

[0015] Obtain the dimensions of the input feature map of the current convolutional layer in the height and width directions;

[0016] Based on the dimensions of the convolution kernel used by the current convolutional layer in the height direction and the width direction, determine the number of rows that need to be filled at the top edge of the input feature map in the height direction, and the number of columns that need to be filled at the left and right edges in the width direction, respectively.

[0017] The top, left, and right edges of the input feature map are filled according to the number of rows and columns determined above.

[0018] Optionally, when filling the top, left, and right edges of the input feature map, zero values ​​are used for filling.

[0019] Optionally, the second edge-padding process performed on the input feature map of each convolutional layer includes:

[0020] Obtain the input feature map of the current convolutional layer during the current inference;

[0021] Based on the kernel size of the current convolutional layer, determine the number of rows that need to be filled at the top edge of the input feature map in the height direction, and the number of columns that need to be filled at the left and right edges in the width direction, respectively.

[0022] At the top edge in the height direction, the feature values ​​of the input feature map of this convolutional layer during the last inference are used to fill the gaps at the end of the corresponding row in the height direction;

[0023] Fill the left and right edges in the width direction with zero values;

[0024] The input feature map retains its original feature values ​​at the lower edge in the height direction.

[0025] Optionally, the reasoning stage further includes:

[0026] For each convolutional layer, save the feature values ​​of the input feature map of that convolutional layer at the end of a specified number of rows in the height direction during the previous inference. The specified number of rows is the same as the number of rows that need to be filled at the top edge in the height direction.

[0027] During the current inference, the corresponding data is read from the saved feature values ​​and used for filling the upper edge in the height direction;

[0028] After completing the current inference, save the feature values ​​of the input feature map of the current inference at the same number of rows at the end in the height direction for use in the next inference.

[0029] Optionally, the training phase further includes:

[0030] Each training audio data is processed by frame segmentation and windowing to obtain multiple time-domain signal frames;

[0031] Perform a short-time Fourier transform on each time-domain signal frame to convert it into a frequency-domain signal;

[0032] The power spectrum is calculated based on the frequency domain signal, and the power spectrum is mapped onto the Mel frequency scale to obtain the filter bank energy characteristics of each frame;

[0033] The logarithm of the energy features of the filter bank is taken to obtain the log-Mel filter bank features, which are used as the input of the convolutional neural network model.

[0034] Optionally, the convolutional neural network model includes four convolutional layers and two fully connected layers;

[0035] Each convolutional layer includes, in sequence: edge padding, convolution operation, batch normalization, modified linear unit activation function, and average pooling;

[0036] Each convolutional layer has its own kernel size, stride, and number of output channels;

[0037] Each average pooling layer has a pooling window size and step size.

[0038] In the two fully connected layers, the last fully connected layer is equipped with output nodes. Each output node corresponds to a keyword category and is connected to a normalized exponential function to output the confidence level of each keyword category.

[0039] Secondly, embodiments of this application provide a streaming speech keyword recognition device, the device including a training module and an inference module;

[0040] The training module is used to construct a convolutional neural network model, which includes multiple convolutional layers; and to train the convolutional neural network model, the training including: performing a first edge-padding process on the input feature map of each convolutional layer, the first edge-padding process being based on causal constraints;

[0041] The inference module is used to acquire the streaming speech signal to be recognized, and to extract a fixed length of audio signal each time; to input the extracted audio signal into the trained convolutional neural network model, and to perform a second edge-padding process on the input feature map of each convolutional layer, the second edge-padding process being based on historical states; and to obtain the speech keyword recognition result based on the output of the convolutional neural network model.

[0042] Optionally, the first edge-padding process performed on the input feature map of each convolutional layer includes:

[0043] Obtain the dimensions of the input feature map of the current convolutional layer in the height and width directions;

[0044] Based on the dimensions of the convolution kernel used by the current convolutional layer in the height direction and the width direction, determine the number of rows that need to be filled at the top edge of the input feature map in the height direction, and the number of columns that need to be filled at the left and right edges in the width direction, respectively.

[0045] The top, left, and right edges of the input feature map are filled according to the number of rows and columns determined above.

[0046] Optionally, when filling the top, left, and right edges of the input feature map, zero values ​​are used for filling.

[0047] Optionally, the second edge-padding process performed on the input feature map of each convolutional layer includes:

[0048] Obtain the input feature map of the current convolutional layer during the current inference;

[0049] Based on the kernel size of the current convolutional layer, determine the number of rows that need to be filled at the top edge of the input feature map in the height direction, and the number of columns that need to be filled at the left and right edges in the width direction, respectively.

[0050] At the top edge in the height direction, the feature values ​​of the input feature map of this convolutional layer during the last inference are used to fill the gaps at the end of the corresponding row in the height direction;

[0051] Fill the left and right edges in the width direction with zero values;

[0052] The input feature map retains its original feature values ​​at the lower edge in the height direction.

[0053] Optionally, the inference module is further configured to:

[0054] For each convolutional layer, save the feature values ​​of the input feature map of that convolutional layer at the end of a specified number of rows in the height direction during the previous inference. The specified number of rows is the same as the number of rows that need to be filled at the top edge in the height direction.

[0055] During the current inference, the corresponding data is read from the saved feature values ​​and used for filling the upper edge in the height direction;

[0056] After completing the current inference, save the feature values ​​of the input feature map of the current inference at the same number of rows at the end in the height direction for use in the next inference.

[0057] Optionally, the training module is further configured to:

[0058] Each training audio data is processed by frame segmentation and windowing to obtain multiple time-domain signal frames;

[0059] Perform a short-time Fourier transform on each time-domain signal frame to convert it into a frequency-domain signal;

[0060] The power spectrum is calculated based on the frequency domain signal, and the power spectrum is mapped onto the Mel frequency scale to obtain the filter bank energy characteristics of each frame;

[0061] The logarithm of the energy features of the filter bank is taken to obtain the log-Mel filter bank features, which are used as the input of the convolutional neural network model.

[0062] Optionally, the convolutional neural network model includes four convolutional layers and two fully connected layers;

[0063] Each convolutional layer includes, in sequence: edge padding, convolution operation, batch normalization, modified linear unit activation function, and average pooling;

[0064] Each convolutional layer has its own kernel size, stride, and number of output channels;

[0065] Each average pooling layer has a pooling window size and step size.

[0066] In the two fully connected layers, the last fully connected layer is equipped with output nodes. Each output node corresponds to a keyword category and is connected to a normalized exponential function to output the confidence level of each keyword category.

[0067] Thirdly, embodiments of this application provide a computer device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, the steps of the streaming speech keyword recognition method described in any of the optional embodiments of the first aspect are performed.

[0068] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the streaming speech keyword recognition method described in any of the optional embodiments of the first aspect.

[0069] The technical solution provided in this application includes, but is not limited to, the following beneficial effects:

[0070] A convolutional neural network model with multiple convolutional layers was constructed, providing a suitable model carrier for streaming speech keyword recognition. The structure of multiple convolutional layers enables hierarchical extraction of speech features, endowing the model with basic keyword recognition capabilities. At the same time, it provides structural support for the edge-padding processing designed for the convolutional layers in the subsequent training and inference stages, which is the basic guarantee for realizing streaming speech keyword recognition.

[0071] The convolutional neural network model is trained by performing a first edge-padding process based on causal constraints on the input feature map at each convolutional layer. This first edge-padding process on the input feature map of the convolutional layer based on causal constraints aligns the feature processing logic during model training with the temporal characteristics of streaming recognition, making the feature processing during training more targeted. The trained model parameters can better adapt to the actual needs of streaming recognition, effectively improving the model's generalization ability and laying a solid foundation for accurate recognition during the inference stage.

[0072] It acquires the streaming speech signal to be recognized, and extracts a fixed length of audio signal each time. By extracting a fixed length of audio signal each time for the streaming speech signal, it adapts to the real-time processing requirements of streaming recognition, realizes the segment-by-segment streaming processing of the speech signal, avoids the recognition response delay caused by waiting for complete audio data, and ensures the real-time performance of streaming speech keyword recognition from the input stage, which is in line with the actual application scenarios of embedded IoT devices.

[0073] The extracted audio signal is input into the trained model, and a second edge-padding process based on historical states is performed on the input feature map at each convolutional layer. This second edge-padding process on the input feature map of the convolutional layer based on historical states makes the edge-padding operation in the inference stage conform to the characteristics of streaming segment-by-segment recognition, making full use of historical feature information to optimize the feature processing effect. This allows the input feature map of the convolutional layer to more realistically reflect the actual distribution of speech features, effectively improving the feature processing quality in the streaming recognition process and enhancing the accuracy of speech keyword recognition in the inference stage.

[0074] The speech keyword recognition results are obtained based on the output of the convolutional neural network model. The recognition results are obtained directly from the output of the trained and specifically optimized convolutional neural network model, which simplifies the process of obtaining the results of streaming speech keyword recognition. Combined with the dual optimization of model training and inference processing in the preceding steps, the accuracy and effectiveness of the recognition results are guaranteed, and it can provide a direct and reliable basis for keyword recognition for the voice interaction of embedded IoT devices.

[0075] This application establishes a complete technical process adapted to streaming speech keyword recognition through a layered design and progressive approach in the training and inference phases. A multi-convolutional model provides the foundational structure for recognition; first-stage padding based on causal constraints optimizes model training; fixed-length audio truncation ensures real-time recognition; second-stage padding based on historical states improves the accuracy of inference recognition; and obtaining results from the model output achieves efficient output of recognition results. These steps work synergistically, from model foundation, training optimization, input processing, inference optimization to result output, comprehensively optimizing the technology of streaming speech keyword recognition. This ensures real-time performance while improving recognition accuracy, making the entire recognition method more suitable for the streaming speech keyword recognition needs of embedded IoT devices.

[0076] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0077] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0078] Figure 1 A flowchart of a training method provided in Embodiment 1 of this application is shown;

[0079] Figure 2 A flowchart of a reasoning method provided in Embodiment 1 of this application is shown;

[0080] Figure 3 A flowchart of a first edge-padding processing method provided in Embodiment 1 of this application is shown;

[0081] Figure 4 A flowchart of a second edge-padding method provided in Embodiment 1 of this application is shown;

[0082] Figure 5A flowchart of the second reasoning method provided in Embodiment 1 of this application is shown;

[0083] Figure 6 A flowchart of the second training method provided in Embodiment 1 of this application is shown;

[0084] Figure 7 This paper shows a schematic diagram of the structure of a streaming speech keyword recognition device provided in Embodiment 2 of this application;

[0085] Figure 8 A schematic diagram of the structure of a computer device provided in Embodiment 3 of this application is shown. Detailed Implementation

[0086] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0087] Example 1

[0088] To facilitate understanding of this application, the following is combined with... Figure 1 The flowchart of the training method provided in Embodiment 1 of this application illustrates Embodiment 1 in detail.

[0089] The streaming speech keyword recognition method provided in Embodiment 1 of this application includes a training phase and an inference phase.

[0090] Specifically, this method addresses the problems of inconsistent training and actual deployment input distribution, low accuracy due to limited contextual information in streaming recognition, and unexpected / repeated text caused by overlapping temporal signal processing, which increases the difficulty of post-processing, in existing speech keyword recognition algorithms based on convolutional neural networks (CNNs). An improved CNN streaming speech keyword network is designed to specifically adapt to the resource and power consumption constraints of embedded IoT devices such as smart speakers, in-vehicle systems, and smart home devices. Through dedicated causal edge padding and standardized feature extraction during the training phase, and streaming short audio processing and historical state edge padding during the inference phase, it achieves high accuracy and low latency real-time speech keyword recognition. This solves the problems of poor time sensitivity, false endpoint detection, and difficulty in balancing real-time performance and accuracy in traditional algorithms. Furthermore, the model reduces parameter size through weight sharing, making it easier to deploy in embedded devices.

[0091] See Figure 1 As shown, Figure 1 A flowchart of a training method provided in Embodiment 1 of this application is shown, wherein the training phase includes steps S101-S102:

[0092] S101: Construct a convolutional neural network model, which includes multiple convolutional layers.

[0093] Specifically, this convolutional neural network is an improved CNN model optimized for streaming speech keyword recognition. The model consists of four convolutional layers and two fully connected layers. It is a lightweight network structure that can reduce the model size through weight sharing, maximizing its adaptability to the deployment requirements of embedded devices. The four convolutional layers are speech feature extraction layers. Each convolutional layer includes edge padding, convolution, batch normalization, ReLU (Rectified Linear Unit) activation function, and average pooling operation. The output layer is configured with Softmax (normalized exponential function) activation function to normalize the output results. The two fully connected layers are keyword classification output layers, realizing the mapping from high-dimensional speech features to specific keyword categories. The network as a whole balances feature extraction capability and lightweight requirements.

[0094] S102: Train the convolutional neural network model, the training including: performing a first edge-padding process on the input feature map of each convolutional layer, the first edge-padding process being based on causal constraints.

[0095] Specifically, during model training, the logarithmic Mel filter bank (Fbank) features obtained after feature extraction from longer audio segments of 3-20 seconds are used as input. To avoid introducing speech data from future directions during streaming processing in the inference stage, a first edge-padding process based on causal constraints is performed on the input feature map of each convolutional layer. This edge-padding process differs from the uniform top, bottom, left, and right edge-padding method of traditional CNNs. It only pads the upper edge in the height direction and the left and right edges in the width direction of the input feature map, and does not perform any edge-padding operation on the lower edge in the height direction. It only relies on current and historical feature information to complete the convolution operation, which fundamentally ensures the causality of the convolution operation and avoids the problem of poor model generalization ability caused by the inconsistency between the feature processing logic of training and inference.

[0096] See Figure 2 As shown, Figure 2 A flowchart of a reasoning method provided in Embodiment 1 of this application is shown, wherein the reasoning stage includes steps S201 to S203:

[0097] S201: Acquire the streaming speech signal to be recognized, and extract a fixed length of audio signal each time.

[0098] Specifically, in the inference phase, to achieve real-time detection and rapid response to keywords, a streaming method is used to process continuous speech signals. Each processing iteration extracts a fixed-length audio signal as input, with the extracted audio length being 0.2-0.5 seconds, much shorter than the audio length used in the training phase. To ensure that each input audio signal completely covers the network input feature frames, the required audio signal length for each iteration in generating the M×N two-dimensional feature matrix for the network input is... One sampling point.

[0099] Each character has a unique corresponding definition: This indicates the length of the network input features in the time dimension. This indicates the length of the sampling points in a frame. Indicates the frame shift length; and The value is matched with the audio sampling rate; when the sampling rate is 8000... When there are 256 sampling points and a sampling rate of 16000... There are 512 sampling points; For the corresponding Half of that, at a sampling rate of 8000 When there are 128 sampling points and a sampling rate of 16000... There are 256 sampling points.

[0100] If the actual audio length is less than the required audio length, zeros are added to achieve the required audio length; if the actual audio length is greater than the required audio length, the audio is split into two parts for truncation to ensure that the input data size for each inference meets the model requirements.

[0101] S202: Input the audio signal captured each time into the trained convolutional neural network model, and perform a second edge-padding process on the input feature map of each convolutional layer. The second edge-padding process is based on the historical state.

[0102] Specifically, during the inference phase, to adapt to the resource-constrained characteristics of embedded devices, the input dimension of the trained model is optimized by increasing the length of the network input features in the time dimension. Set to be much smaller than the time dimension length corresponding to the training phase. ,Right now This reduces the amount of data involved in calculations during a single inference process, improves the data access hit rate on embedded devices, shortens network inference time, and reduces model memory usage. At the same time, the 0.2-0.5s audio signal extracted each time is input into the model after feature extraction, and a second edge-padding process based on historical states is performed on the input feature map of each convolutional layer.

[0103] This edge-padding process differs from zero-value padding during the training phase. Zero-value padding is only used in the first frame. In subsequent inferences, zero-value padding is no longer used to fill the edges in the height direction. Instead, the feature values ​​of the input feature boundary positions of the convolutional layer from the previous inference are used for padding. The left and right edges in the width direction are still filled with zero values. This reduces the impact of boundary effects on keyword detection and lowers the probability of false detections at keyword endpoint positions. Furthermore, this edge-padding process requires saving the historical state separately for each convolutional layer to achieve state transfer across inferences.

[0104] S203: Based on the output of the convolutional neural network model, the speech keyword recognition result is obtained.

[0105] Specifically, the output layer of the model uses the Softmax activation function to normalize the network output, so that each output value represents the confidence level of the corresponding keyword, and the sum of all output values ​​is 1. In practical applications, the final speech keyword recognition result is determined based on the confidence level of the keyword corresponding to each output node. This method ensures the interpretability of the recognition result.

[0106] Meanwhile, because the edge-padding logic of model training and inference is consistent, and the method of overlapping time-domain signals is not required in the inference stage, the problems of unexpected and repeated text caused by overlapping time-domain signal processing in traditional algorithms are avoided, which greatly reduces the difficulty of post-processing of recognition results, while ensuring that the recognized keywords are consistent with the recognition results of the entire audio input.

[0107] This process completely solves the problems of low accuracy and false endpoint detection caused by limited input features in traditional streaming recognition. Furthermore, by optimizing the network input dimension and edge-padding method, it significantly reduces model running time and memory usage. At the same time, it eliminates the need to use overlapping time-domain signals to improve accuracy, avoids the generation of unexpected or duplicate text, and reduces the difficulty of post-processing the recognition results.

[0108] In an optional implementation, see Figure 3 As shown, Figure 3 The flowchart of a first edge-padding method provided in Embodiment 1 of this application is shown, wherein the first edge-padding process is performed on the input feature map of each convolutional layer, including steps S301 to S303:

[0109] S301: Obtain the dimensions of the input feature map of the current convolutional layer in the height direction and the width direction.

[0110] Specifically, the input feature map of the current convolutional layer is in tensor form, and its expression is: Each character has a unique corresponding definition: For channel index, For height direction index, Indexed in the width direction. The number of channels in the feature map. The input feature map has dimensions in the height direction. The input feature map is the dimension in the width direction. For the real number field, For the input feature tensor, the first Passage, height ,width Eigenvalues ​​at; This is the matrix transpose operator, indicating that the matrix is ​​transposed.

[0111] At the same time, the Input features of each channel ,in It is a row vector; this step requires accurately obtaining the input feature tensor. and The specific values ​​provided the basis for subsequent calculations of the number of edges to be added.

[0112] S302: Based on the dimensions of the convolution kernel used by the current convolutional layer in the height direction and the width direction, determine the number of rows to be filled at the top edge of the input feature map in the height direction, and the number of columns to be filled at the left and right edges in the width direction, respectively.

[0113] Specifically, let the first The kernel size of the convolutional layer is ,in For an odd number of rows, the stride of the convolution operation in this convolutional layer is uniformly set to 1×1. The number of padding rows is calculated based on the kernel size; padding is only applied to the top edge in the height direction, not the bottom edge. The number of rows to be padded at the top edge is... Fill the bottom edge with rows =0; the width direction is symmetrically padded left and right, and the left edge is filled with columns. Number of columns filled with the right edge They are equal, and the calculation formula is: .

[0114] Each character has a unique corresponding definition: This is the layer index for the convolutional layer, with values ​​of 0, 1, 2, and 3 corresponding to four convolutional layers respectively. For the first The size of the layer convolution kernel in the height direction, For the first The size of the convolution kernel in the width direction. Add the number of rows of padding along the height direction. Add the number of rows to the bottom edge in the height direction. Add the number of columns to the left edge in the width direction. This represents the number of columns used to fill the right edge in the width direction.

[0115] S303: Fill the top, left, and right edges of the input feature map according to the number of rows and columns determined above.

[0116] Specifically, calculated according to step S302 , , For input feature tensor Perform a directional edge patching operation; the output tensor after edge patching is: Each character has a unique corresponding definition: The output feature tensor after edge patching, For the first edge after filling Passage, height ,width eigenvalues ​​at that location This represents the dimension of the feature map in the height direction after edge patching. This represents the dimension of the feature map in the width direction after edge patching.

[0117] The formula for calculating the size of the feature map after edge patching is as follows: , This step only fills the upper edge of the input feature map. Row, left edge fill Column, right margin padding In this column, no padding is applied to the lower edge in the height direction, thus ensuring the causality of the convolution operation from the feature processing level.

[0118] This process is the main step in achieving causal constraint-based edge padding. By accurately obtaining the feature map size, calculating the number of edges to be padded, and performing directional edge padding, it ensures that the feature map size is consistent before and after the convolution operation, and fundamentally avoids introducing data from future directions. This edge padding process is applicable to four convolutional layers during the training phase. Each convolutional layer performs the edge padding operation independently according to this step, and the edge padding parameters are calculated separately based on the size of each convolutional kernel. The edge padding processes of each layer do not interfere with each other.

[0119] In an optional implementation, zero values ​​are used to fill the top, left, and right edges of the input feature map.

[0120] Specifically, the zero-value padding in the first edge-padding process during the training phase applies to the feature tensor after edge padding. Execute at the specified location, the first Output characteristics of each channel Each character has a unique corresponding definition: For the edge in the height direction after patching, Channel, First The eigenvalues ​​of the row The first edge of the original input feature map after edge patching Channel, First The characteristic value of the row.

[0121] and Let be a row vector, where A value of 0 represents the padding value for the left and right edges in the width direction; during the training phase... It is a dimension of The zero vector, that is, the edge padding positions in the height direction are also filled with the value 0, realizing the zero-value filling operation of all specified padding positions during the training phase.

[0122] In an optional implementation, see Figure 4 As shown, Figure 4 The flowchart of a second edge-padding method provided in Embodiment 1 of this application is shown, wherein the second edge-padding process is performed on the input feature map of each convolutional layer, including steps S401 to S405:

[0123] S401: Obtain the input feature map of the current convolutional layer during the current inference.

[0124] Specifically, the input feature map for the current inference is composed of logarithmic Mel filter bank (Fbank) features obtained after feature extraction from each 0.2-0.5s segment of the audio signal. Its time dimension length is... ,and ( (The input feature time dimension length is used during the training phase). This feature map is also represented in tensor form, with the tensor dimension defined consistent with that used during the training phase. This ensures that the input size matches the model's convolutional layers, eliminating the need for additional adjustments to the network structure.

[0125] S402: Based on the kernel size of the current convolutional layer, determine the number of rows to be filled at the top edge of the input feature map in the height direction, and the number of columns to be filled at the left and right edges in the width direction, respectively.

[0126] Specifically, the rules for determining the number of rows and columns for edge completion during the inference phase are completely consistent with those during the training phase, still based on the first... kernel size calculate The lower edge in the height direction remains unchanged. =0; The definitions of all characters are completely consistent with those in the training phase, without any adjustments, ensuring the consistency of convolution operations in the training and inference phases, and avoiding a decrease in recognition accuracy due to changes in the edge-padding rules.

[0127] S403: At the top edge in the height direction, fill with the feature values ​​of the input feature map of this convolutional layer at the end of the corresponding row in the height direction during the last inference.

[0128] Specifically, during the inference phase, except for the first frame which uses zero-value padding, the edge padding value in the height direction of the subsequent nth inference iteration is... Directly adopt the first The input feature map of this convolutional layer in the next inference step The feature values ​​are the lower data of the previous inference input features in the height direction; each character has a unique corresponding definition: Index for the number of inferences, For the first After the second reasoning and edge patching, the edge in the height direction is... Channel, First The eigenvalues ​​of the row For the first The input feature map of the next inference step Channel, width The eigenvalue at that location.

[0129] S404: Fill the left and right edges with zero values ​​in the width direction.

[0130] Specifically, during the inference phase, only the edge padding method in the height direction is optimized and adjusted; the left and right edges in the width direction continue the zero-value padding method from the training phase, and the padding feature value is... (Value 0) The definition of all characters is completely consistent with that in the training phase; this design ensures the consistency of the width-direction side-padding rules in the training and inference phases, avoids model inference confusion caused by changes in multi-dimensional side-padding rules, and simplifies the network computation logic.

[0131] S405: The input feature map retains its original feature values ​​at the lower edge in the height direction.

[0132] Specifically, the second edge-padding process in the inference phase also strictly follows the causal constraint. No edge-padding operation is performed on the lower edge in the height direction, completely preserving the original feature values ​​of the current input feature map and not introducing any speech data from the future direction. This design is highly consistent with the edge-padding logic in the training phase, ensuring that the input feature processing distribution of the model training and inference is consistent, and solving the problem of decreased temporal sensitivity and accuracy caused by the inconsistency between the training and actual use input distribution in traditional algorithms.

[0133] This process is designed for the characteristics of streaming processing in the inference stage. Unlike the zero-value edge padding in the training stage, it mainly introduces historical feature data from the previous inference to complete the edge padding in the height direction, thereby reducing the impact of boundary effects on keyword detection. This edge padding process is applicable to the four convolutional layers in the inference stage. Each convolutional layer is executed independently according to this step, and historical feature data must be maintained separately for each layer to ensure the state transfer across inference cycles.

[0134] In an optional implementation, see Figure 5 As shown, Figure 5 A flowchart of the second reasoning method provided in Embodiment 1 of this application is shown, wherein the reasoning stage further includes steps S501 to S503:

[0135] S501: For each convolutional layer, save the feature values ​​of the input feature map of the convolutional layer at the end of a specified number of rows in the height direction during the last inference. The specified number of rows is the same as the number of rows that need to be filled at the top edge in the height direction.

[0136] Specifically, each convolutional layer needs to allocate separate memory space to store historical feature values. The number of rows stored is equal to the number of rows calculated by the convolutional kernel of that layer. The values ​​must be exactly the same, meaning the edges in the height direction need to be filled. If so, then save the end of the previous inference input feature map in the height direction. Row feature values; different convolutional layers vary depending on the kernel size. different, The values ​​may differ, so the number of rows of historical feature values ​​saved in each layer may vary, and they need to be set and saved separately.

[0137] S502: During the current inference, read the corresponding data from the saved feature values ​​for filling the upper edge in the height direction.

[0138] Specifically, during the current inference process, for each convolutional layer, the last saved data from the previous inference is retrieved directly from the dedicated historical feature data storage space of that layer. The feature values ​​are directly filled into the corresponding positions of the upper edge in the height direction of the current input feature map without recalculation or using zero values, thus completing the edge filling operation. This method does not require additional feature calculations, which ensures the accuracy of edge filling and improves inference efficiency.

[0139] S503: After completing the current inference, save the feature values ​​of the input feature map of the current inference at the same number of rows at the end in the height direction for use in the next inference.

[0140] Specifically, after the current inference convolution operation and keyword recognition result are completed, the historical feature data storage space of that convolutional layer is updated in a timely manner, and the end of the feature map input in the current inference in the height direction is added. The updated data replaces the existing historical feature values ​​in the storage space. The updated data is used as the padding data for the next inference, thereby realizing the continuous transmission of historical feature states in streaming inference and ensuring the real-time performance and effectiveness of the padding data.

[0141] This process needs to be executed separately for each of the four convolutional layers. That is, each convolutional layer independently maintains a set of historical feature data, and the historical states of each layer do not interfere with each other, ensuring the accuracy of edge padding for each convolutional layer.

[0142] In an optional implementation, see Figure 6 As shown, Figure 6 A flowchart of the second training method provided in Embodiment 1 of this application is shown, wherein the training stage further includes steps S601 to S605:

[0143] S601: Perform frame-by-frame windowing processing on each training audio data to obtain multiple time-domain signal frames.

[0144] Specifically, the training audio data consists of longer audio clips ranging from 3 to 20 seconds, processed according to the formula. Calculate the number of frames and divide the entire audio file. The signal is divided into multiple time-domain signal frames of equal length; each character / symbol has a unique corresponding definition: This is the floor function symbol, indicating that the result within the parentheses is rounded down. Indicates the number of audio sampling points. This indicates the length of the sampling points in a frame. Indicates the frame shift length.

[0145] and The value of matches the audio sampling rate: when the sampling rate is 8000, There are 256 sampling points. With 128 sampling points and a sampling rate of 16000, There are 512 sampling points. There are 256 sampling points.

[0146] After framing, each frame of signal is multiplied by a window function (such as the Hanning window) to reduce spectral leakage during subsequent frequency domain conversion and ensure the accuracy of feature extraction.

[0147] S602: Perform a short-time Fourier transform on each time-domain signal frame to convert it into a frequency-domain signal.

[0148] Specifically, for each frame of time-domain signal after Hanning windowing, a short-time fourier transform (STFT) is performed separately to convert the time-domain signal to the frequency-domain signal. The short-time fourier transform can take into account both the time-domain and frequency-domain characteristics of the speech signal, adapt to the feature extraction requirements of non-stationary signals such as speech, and provide the basic frequency-domain data for subsequent power spectrum calculation and Mel band mapping.

[0149] S603: Calculate the power spectrum based on the frequency domain signal, and map the power spectrum onto the Mel frequency scale to obtain the filter bank energy characteristics of each frame.

[0150] Specifically, first, the positive frequency portion after the short-time Fourier transform (STFT) is taken, and its power spectrum is calculated. ,in The frequency index is used; then, a set of triangular filter banks is constructed based on the Mel (Mel Scale) frequency scale, with the number of filters... Matching the audio sampling rate: at a sampling rate of 8000 =20, sampling rate 16000 =40, the center frequency of each triangular filter is... , No. The expression for the triangular filter is:

[0151]

[0152] Each character has a unique corresponding definition: This is the index for the Mel filter, with values ​​of 0, 1, ... , For the first A triangular filter at frequency The value at that location, For the first The center frequency of the triangular filter, For the first The center frequency of the triangular filter, For the first The center frequency of the triangular filter, For frequency indexing.

[0153] The power spectrum is then mapped onto the Mel frequency band, and the energy characteristics of the filter bank are obtained by weighted summation of the spectral energy within each frequency band. The calculation formula is as follows:

[0154]

[0155] Each character / symbol has a unique corresponding definition: The subscript is the starting index value and the superscript is the ending index value, representing the summation operation on all values ​​within the index range; For the first Energy characteristics of filter banks in the Mel frequency band For frequency The power spectral value at that location, For the first A triangular filter at frequency The value at that location, The frequency dimension of the power spectrum is given by the following formula: , The length of the sampling points in one frame. This is the index for the Mel filter.

[0156] S604: Take the logarithm of the energy features of the filter bank to obtain the log-Mel filter bank features, which are used as the input of the convolutional neural network model.

[0157] Specifically, according to the formula , Energy characteristics of the filter bank Perform logarithmic operations to obtain the logarithmic Mel filter bank characteristics. Each character / symbol has a unique corresponding definition: This is the symbol for a logarithmic function, indicating that a logarithmic operation is performed on the value within the parentheses. For the first Characteristics of a logarithmic filter bank in one Mel frequency band. For the first Energy characteristics of filter banks in the Mel frequency band This is the index for the Mel filter.

[0158] The final result for The two-dimensional feature matrix, where For the number of frames, The number of Mel filters represents the feature matrix, which is directly used as input features for improving the CNN keyword recognition network model.

[0159] In one alternative implementation, the convolutional neural network model includes four convolutional layers and two fully connected layers.

[0160] Specifically, this network structure is a lightweight structure designed specifically for streaming speech keyword recognition. Four convolutional layers are responsible for extracting local and global acoustic features of speech layer by layer, from low-dimensional speech spectral features to high-dimensional semantic features, realizing hierarchical feature mining. Two fully connected layers are responsible for mapping the high-dimensional features extracted by the convolutional layers to specific keyword category spaces, realizing feature dimensionality reduction and classification. The overall network has few layers and small parameter scale. Through weight sharing, the model size is further reduced, enabling it to run efficiently on embedded devices while taking into account both feature extraction and classification recognition capabilities.

[0161] Each convolutional layer includes, in sequence: edge padding, convolution, batch normalization, modified linear unit activation function, and average pooling.

[0162] Specifically, the operations of each convolutional layer are executed in a fixed order. The function and execution logic of each operation are as follows: the edge padding operation is the first edge padding process during the training phase and the second edge padding process during the inference phase, ensuring that the feature map size is consistent before and after the convolution operation and meets the causal constraints; the stride of the convolution operation is uniformly set to 1×1, and local features are extracted through the convolution operation between the convolution kernel and the feature map; the batch normalization operation standardizes the output of the convolution operation, accelerates the model training convergence speed, and avoids gradient vanishing; the linear unit activation function, i.e., the ReLU function, is modified to introduce non-linear characteristics into the network and improve the non-linear fitting ability of the model; the average pooling operation downsamples the activated feature map, reduces the feature map dimension, and reduces the model's computational load and parameter size; the operations work together to achieve efficient speech feature extraction.

[0163] Each convolutional layer has its own kernel size, stride, and number of output channels.

[0164] Specifically, each convolutional layer is independently configured with kernel size, stride, and number of output channels, where the kernel size is... ,and For odd numbers, the convolution operation stride is uniformly set to 1×1, and the number of output channels can be flexibly configured according to feature extraction needs; different convolution kernel sizes in different convolutional layers enable the extraction of speech features at different scales, and the differentiated configuration of the number of output channels enables the dimensionality of features to be increased and the features to be mined in a diversified manner. The parameter configurations of the four convolutional layers work together to ensure the comprehensiveness and accuracy of feature extraction.

[0165] Each average pooling layer has its own pooling window size and step size.

[0166] Specifically, the average pooling layer following each convolutional layer has an independently configured pooling window size and stride. The values ​​of the pooling window size and stride are designed according to the temporal characteristics of speech features and the requirements for dimensionality compression. Average pooling downsamples the feature map by averaging the feature values ​​within the pooling window. This reduces the spatial dimensionality of the feature map while retaining the main information of the features and avoiding overfitting. Different pooling layers have different parameter configurations to achieve gradient dimensionality compression of the output features of the convolutional layers, adapting to the input size requirements of subsequent fully connected layers.

[0167] In the two fully connected layers, the last fully connected layer is equipped with output nodes. Each output node corresponds to a keyword category and is connected to a normalized exponential function to output the confidence level of each keyword category.

[0168] Specifically, the first fully connected layer is responsible for mapping the high-dimensional features extracted by the convolutional and pooling layers into medium-dimensional feature vectors, thereby achieving feature dimensionality reduction and fusion. The number of output nodes in the last fully connected layer is exactly the same as the number of keyword categories to be identified. Each output node uniquely corresponds to a keyword category, and the original output value of the output node is the predicted value of that keyword category.

[0169] The final fully connected layer uses the Softmax activation function as the normalized exponential function to normalize the original predicted values. The calculation formula is as follows: Each character / symbol has a unique corresponding definition: is a natural constant, with a value of approximately 2.71828; For exponential operations, it means using the natural constant. as the base For exponentiation; For the first The original predicted values ​​of each output node. This is the output node index for the fully connected layer, with values ​​of 1, 2, ... , The number of output nodes for the fully connected layer is consistent with the number of keyword categories to be identified.

[0170] After processing, the value of each output node is the confidence level of the corresponding keyword category, and the sum of the values ​​of all output nodes is 1, which makes it easy to directly determine the final keyword recognition result based on the confidence level.

[0171] Example 2

[0172] See Figure 7 As shown, Figure 7 A schematic diagram of the structure of a streaming speech keyword recognition device provided in Embodiment 2 of this application is shown, wherein the device includes a training module 701 and an inference module 702;

[0173] The training module is used to construct a convolutional neural network model, which includes multiple convolutional layers; and to train the convolutional neural network model, the training including: performing a first edge-padding process on the input feature map of each convolutional layer, the first edge-padding process being based on causal constraints;

[0174] The inference module is used to acquire the streaming speech signal to be recognized, and to extract a fixed length of audio signal each time; to input the extracted audio signal into the trained convolutional neural network model, and to perform a second edge-padding process on the input feature map of each convolutional layer, the second edge-padding process being based on historical states; and to obtain the speech keyword recognition result based on the output of the convolutional neural network model.

[0175] In an optional implementation, the first edge-padding process on the input feature map of each convolutional layer includes:

[0176] Obtain the dimensions of the input feature map of the current convolutional layer in the height and width directions;

[0177] Based on the dimensions of the convolution kernel used by the current convolutional layer in the height direction and the width direction, determine the number of rows that need to be filled at the top edge of the input feature map in the height direction, and the number of columns that need to be filled at the left and right edges in the width direction, respectively.

[0178] The top, left, and right edges of the input feature map are filled according to the number of rows and columns determined above.

[0179] In an optional implementation, zero values ​​are used to fill the top, left, and right edges of the input feature map.

[0180] In an optional implementation, the second edge-padding process on the input feature map of each convolutional layer includes:

[0181] Obtain the input feature map of the current convolutional layer during the current inference;

[0182] Based on the kernel size of the current convolutional layer, determine the number of rows that need to be filled at the top edge of the input feature map in the height direction, and the number of columns that need to be filled at the left and right edges in the width direction, respectively.

[0183] At the top edge in the height direction, the feature values ​​of the input feature map of this convolutional layer during the last inference are used to fill the gaps at the end of the corresponding row in the height direction;

[0184] Fill the left and right edges in the width direction with zero values;

[0185] The input feature map retains its original feature values ​​at the lower edge in the height direction.

[0186] In an optional implementation, the inference module is further configured to:

[0187] For each convolutional layer, save the feature values ​​of the input feature map of that convolutional layer at the end of a specified number of rows in the height direction during the previous inference. The specified number of rows is the same as the number of rows that need to be filled at the top edge in the height direction.

[0188] During the current inference, the corresponding data is read from the saved feature values ​​and used for filling the upper edge in the height direction;

[0189] After completing the current inference, save the feature values ​​of the input feature map of the current inference at the same number of rows at the end in the height direction for use in the next inference.

[0190] In an optional implementation, the training module is further configured to:

[0191] Each training audio data is processed by frame segmentation and windowing to obtain multiple time-domain signal frames;

[0192] Perform a short-time Fourier transform on each time-domain signal frame to convert it into a frequency-domain signal;

[0193] The power spectrum is calculated based on the frequency domain signal, and the power spectrum is mapped onto the Mel frequency scale to obtain the filter bank energy characteristics of each frame;

[0194] The logarithm of the energy features of the filter bank is taken to obtain the log-Mel filter bank features, which are used as the input of the convolutional neural network model.

[0195] In one optional implementation, the convolutional neural network model includes four convolutional layers and two fully connected layers;

[0196] Each convolutional layer includes, in sequence: edge padding, convolution operation, batch normalization, modified linear unit activation function, and average pooling;

[0197] Each convolutional layer has its own kernel size, stride, and number of output channels;

[0198] Each average pooling layer has a pooling window size and step size.

[0199] In the two fully connected layers, the last fully connected layer is equipped with output nodes. Each output node corresponds to a keyword category and is connected to a normalized exponential function to output the confidence level of each keyword category.

[0200] Example 3

[0201] Based on the same application concept, see [link / reference] Figure 8 As shown, Figure 8 This illustration shows a structural schematic diagram of a computer device provided in Embodiment 3 of this application, wherein, as shown... Figure 8 As shown, the computer device 800 provided in Embodiment 3 of this application includes:

[0202] The computer device 800 includes a processor 801, a memory 802, and a bus 803. The memory 802 stores machine-readable instructions that can be executed by the processor 801. When the computer device 800 is running, the processor 801 communicates with the memory 802 through the bus 803. When the machine-readable instructions are executed by the processor 801, they perform the steps of the streaming speech keyword recognition method shown in Embodiment 1 above.

[0203] Example 4

[0204] Based on the same concept, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the streaming speech keyword recognition method described in any of the above embodiments.

[0205] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system and apparatus described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0206] The computer program product for streaming speech keyword recognition provided in this application includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the preceding method embodiments. For specific implementation details, please refer to the method embodiments, which will not be repeated here.

[0207] The streaming speech keyword recognition device provided in this application embodiment can be specific hardware on a device or software or firmware installed on the device. The implementation principle and technical effects of the device provided in this application embodiment are the same as those in the foregoing method embodiments. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the foregoing method embodiments. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can all be referred to the corresponding processes in the above method embodiments, and will not be repeated here.

[0208] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0209] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0210] In addition, the functional units in the embodiments provided in this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0211] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0212] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0213] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The protection scope of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application. All should be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A method of streaming speech keyword recognition, the method comprising: Includes a training phase and an inference phase; The training phase includes: Construct a convolutional neural network model, wherein the convolutional neural network model includes multiple convolutional layers; The convolutional neural network model is trained, and the training includes: performing a first edge-padding process on the input feature map of each convolutional layer, wherein the first edge-padding process is based on causal constraints; The reasoning phase includes: Acquire the streaming speech signal to be recognized, and extract a fixed length of audio signal each time; Each extracted audio signal is input into the trained convolutional neural network model, and a second edge-padding process is performed on the input feature map of each convolutional layer, which is based on historical states. Based on the output of the convolutional neural network model, the speech keyword recognition result is obtained; The first edge-padding process performed on the input feature map of each convolutional layer includes: Obtain the dimensions of the input feature map of the current convolutional layer in the height and width directions; Based on the dimensions of the convolution kernel used by the current convolutional layer in the height direction and the width direction, determine the number of rows that need to be filled at the top edge of the input feature map in the height direction, and the number of columns that need to be filled at the left and right edges in the width direction, respectively. The top, left, and right edges of the input feature map are filled according to the number of rows and columns determined above.

2. The method of claim 1, wherein, When filling the top, left, and right edges of the input feature map, zero values ​​are used for filling.

3. The method of claim 1, wherein, The second edge-padding process performed on the input feature map of each convolutional layer includes: Obtain the input feature map of the current convolutional layer during the current inference; Based on the kernel size of the current convolutional layer, determine the number of rows that need to be filled at the top edge of the input feature map in the height direction, and the number of columns that need to be filled at the left and right edges in the width direction, respectively. At the top edge in the height direction, the feature values ​​of the input feature map of this convolutional layer during the last inference are used to fill the gaps at the end of the corresponding row in the height direction; Fill the left and right edges in the width direction with zero values; The input feature map retains its original feature values ​​at the lower edge in the height direction.

4. The method of claim 3, wherein, The reasoning phase also includes: For each convolutional layer, save the feature values ​​of the input feature map of that convolutional layer at the end of a specified number of rows in the height direction during the previous inference. The specified number of rows is the same as the number of rows that need to be filled at the top edge in the height direction. During the current inference, the corresponding data is read from the saved feature values ​​and used for filling the upper edge in the height direction; After completing the current inference, save the feature values ​​of the input feature map of the current inference at the same number of rows at the end in the height direction for use in the next inference.

5. The method of claim 1, wherein, The training phase also includes: Each training audio data is processed by frame segmentation and windowing to obtain multiple time-domain signal frames; Perform a short-time Fourier transform on each time-domain signal frame to convert it into a frequency-domain signal; The power spectrum is calculated based on the frequency domain signal, and the power spectrum is mapped onto the Mel frequency scale to obtain the filter bank energy characteristics of each frame; The logarithm of the energy features of the filter bank is taken to obtain the log-Mel filter bank features, which are used as the input of the convolutional neural network model.

6. The method of claim 1, wherein, The convolutional neural network model includes four convolutional layers and two fully connected layers; Each convolutional layer includes, in sequence: edge padding, convolution operation, batch normalization, modified linear unit activation function, and average pooling; Each convolutional layer has its own kernel size, stride, and number of output channels; Each average pooling layer has a pooling window size and step size. In the two fully connected layers, the last fully connected layer is equipped with output nodes. Each output node corresponds to a keyword category and is connected to a normalized exponential function to output the confidence level of each keyword category.

7. An apparatus for streaming voice keyword recognition, the apparatus comprising: a processor configured to: receive a voice input; determine a keyword in the voice input; and output a keyword result based on the keyword. The device includes a training module and an inference module; The training module is used to construct a convolutional neural network model, which includes multiple convolutional layers; and to train the convolutional neural network model, the training including: performing a first edge-padding process on the input feature map of each convolutional layer, the first edge-padding process being based on causal constraints; The inference module is used to acquire the streaming speech signal to be recognized, and to extract a fixed length of audio signal each time; to input the extracted audio signal into the trained convolutional neural network model, and to perform a second edge-padding process on the input feature map of each convolutional layer, the second edge-padding process being based on historical states; and to obtain the speech keyword recognition result based on the output of the convolutional neural network model. The first edge-padding process performed on the input feature map of each convolutional layer includes: Obtain the dimensions of the input feature map of the current convolutional layer in the height and width directions; Based on the dimensions of the convolution kernel used by the current convolutional layer in the height direction and the width direction, determine the number of rows that need to be filled at the top edge of the input feature map in the height direction, and the number of columns that need to be filled at the left and right edges in the width direction, respectively. The top, left, and right edges of the input feature map are filled according to the number of rows and columns determined above.

8. A computer device, comprising: include: The computer device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform the steps of the streaming speech keyword recognition method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the streaming speech keyword recognition method as described in any one of claims 1 to 6.

Images