A method and device for extracting speech signal features based on space and time series
By combining a transformer encoder with multi-scale convolution processing and a self-attention mechanism with a high-order feature encoder, the problem of low efficiency in speech data feature extraction in existing technologies is solved, achieving more complete feature fusion and information utilization, and improving the model's learning ability and prediction accuracy.
Patent Information
- Application Number
- CN202411218646.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-02
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-09-02
AI Technical Summary
Existing speech data feature extraction methods mainly rely on acousticians to manually construct and select features, which is inefficient and cannot fully utilize the inherent relationship between spatial and temporal features in speech data, resulting in insufficient feature fusion effect.
We employ a speech signal feature extraction method based on spatial and temporal series. Through multi-scale convolution processing, a transformer encoder with self-attention mechanism, and a high-order feature encoder, we combine local spatial features and global temporal features for feature fusion to achieve efficient feature learning.
The improved model performance enables better extraction of key information from speech data, enhances the model's learning ability and prediction accuracy, and achieves more comprehensive feature fusion and information utilization.
Smart Images

Figure CN118762716B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the technical field of speech data processing, and in particular to a speech signal feature extraction method and device based on space and time series. Background Art
[0002] With the rapid development of artificial intelligence technology, professional intelligent devices serving different fields and objects are constantly being developed, providing great help and convenience for industries, life, entertainment, education and other fields. As a hot topic in the field of artificial intelligence, speech recognition can effectively improve the interaction between people and machines and bring great convenience to people's production and life. As a part of speech recognition, speech data feature extraction has become an emerging field that is constantly being studied in the industry.
[0003] However, the existing speech data feature extraction methods mainly rely on acousticians to manually construct and screen features. This process requires a lot of time and is inefficient. Most methods only analyze spatial features or time series features. For example, some methods may only consider the spatial distribution in speech data and ignore the law of data changes over time. Or conversely, some methods may only focus on the time series features of speech data and ignore the spatial features. In this case, the extracted features may not fully reflect the complexity and regularity of speech, which makes it impossible to fully utilize all the information in the speech data. Moreover, in some models for extracting speech space and time series features, spatial features and time series features are often extracted separately and fused at a primary level. This method may ignore the deep correlation that may exist between these features, resulting in insufficient learning effect after feature fusion. Therefore, traditional feature extraction methods cannot fully adapt to the needs of different speech data and have limitations. Therefore, there is an urgent need for a method that can effectively mine the intrinsic connection between spatial and time series features. Summary of the invention
[0004] The present invention provides a method and device for extracting speech signal features based on space and time series, which solves the technical problem that the existing speech signal feature extraction method can only analyze the spatial features or time series features and cannot fully utilize all the information in the speech data.
[0005] According to a first aspect of the present disclosure, a method for extracting speech signal features based on space and time series is provided, comprising:
[0006] Collect voice data;
[0007] The speech data is sent to a plurality of convolution kernels of different sizes for convolution processing to obtain multi-scale features, and feature processing is performed on the multi-scale features to obtain local spatial features of the speech data;
[0008] The speech data is fed into a transformer encoder based on a self-attention mechanism to obtain primary time series features, and the primary time series features are subjected to dimensionality reduction processing to obtain global time series features of the speech data;
[0009] Mapping the local spatial features and the global temporal features to the same feature space for feature fusion to obtain low-order speech features;
[0010] The fused low-order speech features are sent to a high-order feature encoder for high-order feature learning to obtain high-order speech features.
[0011] According to the above aspect and any possible implementation manner, an implementation manner is further provided, wherein the performing feature processing on the multi-scale features to obtain the local spatial features of the speech data includes:
[0012] The multi-scale features are sequentially normalized and activated to obtain optimized features;
[0013] The optimized features are subjected to average pooling and down-sampling processing to obtain local spatial features of the speech signal.
[0014] According to the aspects described above and any possible implementation method, an implementation method is further provided, wherein the multiple convolution kernels of different sizes are three parallel convolution kernels of different sizes; and the scales of the three convolution kernels are 11x1, 1x9 and 3x3 respectively.
[0015] According to the above aspects and any possible implementation manner, an implementation manner is further provided, wherein the method further includes:
[0016] Inputting the speech data into the three parallel convolution kernels to extract the frequency domain, time domain and space domain features of the speech to obtain multi-scale features;
[0017] The multi-scale features are further normalized, activated and averaged-pooled to obtain optimized features;
[0018] The optimized features obtained in the frequency domain, time domain and space domain are added element by element according to the channel dimension to obtain the local spatial features of the speech signal.
[0019] According to the above aspects and any possible implementation, an implementation is further provided, wherein the step of feeding the speech data into a transformer encoder based on a self-attention mechanism to obtain primary time series features includes:
[0020] The speech data is firstly converted into a feature vector through an embedding layer;
[0021] Injecting time series information into the feature vector through position coding to obtain coding features;
[0022] Input the encoded features into a multi-head self-attention mechanism, and obtain an attention score by calculating the query and key of the speech data;
[0023] The softmax of the attention score is calculated to obtain the weight, and then the weight and the value are weighted to output the attention feature vector Attention(Q,K,V).
[0024] ;
[0025] Among them, Q, K, and V represent the query, key, and value of the voice data respectively. k refers to the dimension of the key vector;
[0026] The attention feature vector is regularized and then residually connected with the original speech data, and then subjected to layer normalization to obtain an enhanced feature vector;
[0027] Inputting the enhanced feature vector into a feedforward neural network consisting of two layers of full connection and one layer of activation function for processing;
[0028] The output of the feedforward neural network is regularized, and then a second residual is performed with the enhanced feature vector, and normalized again to obtain the primary time series feature.
[0029] According to the above aspects and any possible implementation manner, an implementation manner is further provided, wherein the performing dimensionality reduction processing on the primary time series features to obtain the global time series features of the speech signal includes:
[0030] The primary time series features are input into the pooling layer for maximum pooling processing, so that the dimension of the primary time series features is reduced and redundant information is reduced, thereby obtaining the global time series features of the speech signal.
[0031] According to the above aspects and any possible implementation manner, an implementation manner is further provided, wherein mapping the local spatial features and the global temporal features to the same feature space for feature fusion to obtain low-order speech features includes:
[0032] The local spatial features and the global temporal features are mapped to the same feature space, and are fused by adding them element by element according to the channel dimension to obtain low-order speech features.
[0033] According to the above aspects and any possible implementation, an implementation is further provided, wherein the step of sending the fused low-order speech features to a high-order feature encoder for high-order feature learning to obtain high-order speech features comprises:
[0034] The low-order speech features obtained by the fusion are input into a 3×3 convolution for convolution processing, and converted into high-level features that can be used by the model;
[0035] The high-level features are sequentially subjected to layer normalization, activation function and average pooling processing to obtain enhanced features;
[0036] The enhanced features are input into the squeeze and stimulate attention to emphasize the features that are helpful for the target task, and obtain efficient high-order features, including:
[0037] Performing global average pooling on the enhanced features, so that the features of each channel are represented by a vector to represent the global receptive field of the channel, thereby obtaining compressed features;
[0038] The compressed features are passed through two fully connected layers to excite the feature map to obtain channel attention weights; wherein the activation function of the first fully connected layer is ReLU, and the activation function of the second fully connected layer is Sigmoid;
[0039] Using the channel attention weights to assign weights to the enhanced features to obtain efficient high-order features;
[0040] The efficient high-order features are input into a multi-layer combined high-order feature encoder for multiple times for learning, so as to obtain high-order speech features.
[0041] According to the above aspects and any possible implementation, an implementation is further provided, wherein the step of inputting the efficient high-order features into a multi-layer combined high-order feature encoder for learning multiple times to obtain high-order speech features comprises: extracting richer hierarchical features in the data by gradually increasing the network depth, specifically: extracting richer hierarchical features in the high-order efficient features by increasing the convolution kernel depth in the high-order feature learning encoder layer by layer, selecting a suitable pooling layer size, and fully learning effective features;
[0042] The last layer of high-order feature encoder uses global average pooling to filter and integrate the feature information of all levels of features, and averages the feature values of each channel of the entire feature map to obtain a high-order speech feature represented by a global feature.
[0043] According to a second aspect of the present disclosure, a speech signal feature extraction device based on space and time series is provided, comprising:
[0044] A voice data collection unit, used for collecting voice data;
[0045] A spatial feature extraction unit, used for sending the speech data to a plurality of convolution kernels of different sizes for convolution processing to obtain multi-scale features, and performing feature processing on the multi-scale features to obtain local spatial features of the speech data;
[0046] A time series feature extraction unit, used for feeding the speech data into a transformer encoder based on a self-attention mechanism to obtain primary time series features, and performing dimensionality reduction processing on the primary time series features to obtain global time series features of the speech data;
[0047] A feature fusion unit, used for mapping the local spatial feature and the global temporal feature to the same feature space for feature fusion to obtain low-order speech features;
[0048] The high-order feature learning unit is used to send the fused low-order speech features to a high-order feature encoder for high-order feature learning to obtain high-order speech features.
[0049] The present invention discloses the following technical effects:
[0050] The present invention proposes a more flexible feature extraction method, which can better extract richer feature information from a speech data set. This flexible and specific feature extraction method can capture more key information and improve model performance. This method is a more comprehensive learning method, which interactively integrates the high-level features of the two, improves the efficiency of feature fusion, and better explores the intrinsic connection between spatial and temporal features. The spatial features and temporal features of speech data extracted by the present invention jointly determine the nature and pattern of speech data to a large extent. Extracting both at the same time can reflect the spatial layout and temporal changes of speech data, which can help the model capture the complete representation of speech data. The complementarity of spatial features and temporal features greatly enhances the learning ability of the model, thereby achieving more accurate model prediction. Learning after feature fusion can better combine different feature information, use one feature to complement another feature, convert it into a new and better feature representation, and enhance the discrimination ability of the model. For space-time features, the extraction of spatial features can make up for the spatial position information that the temporal features cannot pay attention to, and the temporal features can reflect the dynamic change information that the spatial features cannot reveal. This complementarity greatly enhances the learning ability of the model, thereby achieving more accurate model prediction.
[0051] It should be understood that the contents described in the summary of the invention are not intended to limit the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, among which:
[0053] Figure 1 A schematic flow chart of a method for extracting speech signal features based on space and time series according to an embodiment of the present disclosure is shown;
[0054] Figure 2 A schematic diagram of a speech signal feature extraction device based on space and time series according to an embodiment of the present disclosure;
[0055] Figure 3 A schematic diagram of the structure of a speech data spatial feature extraction module and a speech signal temporal feature extraction module according to an embodiment of the present disclosure is shown;
[0056] Figure 4 A schematic diagram of the structure of a high-order feature learning module according to an embodiment of the present disclosure is shown;
[0057] Figure 5 A schematic diagram of the structure of the squeeze and excitation attention layers in a high-order feature encoder according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION
[0058] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.
[0059] The original speech data contains rich but redundant and irrelevant information, so the key of feature extraction is particularly important. Feature extraction aims to extract and transform the original data from high-dimensional to low-dimensional features that retain key information through accurate and representative selection, making it easier to process, and ultimately improving the performance and accuracy of the model.
[0060] Spatial features and temporal features jointly determine the nature and pattern of speech data to a large extent. Spatial features contain the spatial layout and spatial pattern of data, and reflect the relative position relationship and spatial structural characteristics between data elements. Temporal features reveal the evolution trend of speech data over time, reflecting the dynamic and sequential relationship of data. The present invention simultaneously extracts the spatial features and temporal features of speech data to form a complete speech feature processing process, which means that the spatial layout and temporal changes of speech data are taken into account at the same time. These two types of features can reveal the complexity and integrity of speech data from different dimensions and angles, and can help the model capture the complete representation of speech data.
[0061] Among them, the main goal of extracting spatial features is to understand the distribution and organization of speech data in spatial structure; the purpose of extracting temporal features is to understand the dynamic change process and periodic laws of speech data. By combining the spatial features and temporal features of the original speech data, we can understand the data more comprehensively and deeply, and then discover deeper laws and patterns from the data.
[0062] After that, these deep-mined and screened effective features are input into machine learning and deep learning models, which can greatly improve the accuracy of the model and provide stable and reliable data support for specific application tasks. The present invention comprehensively extracts local spatial features and global temporal features in speech data, and deeply integrates these two types of features for effective learning, providing an effective means for more accurate and efficient speech signal analysis.
[0063] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.
[0064] Please refer to the attached Figure 1 This embodiment provides a method for extracting speech signal features based on space and time series, comprising the following steps:
[0065] S101, collect voice data. This embodiment collects voice data in multiple scenarios, such as offices, cafes, shopping malls, homes, and outdoors. At the same time, by performing pre-collection before formal voice collection, the working state of the voice collection device is evaluated through the pre-collected voice data collected during the pre-collection process, so that the voice collection device is adjusted in time when the working state of the voice collection device does not meet the collection requirements, and the formal voice collection is started only when the voice collection device meets the collection requirements, thereby ensuring the reliability of voice data collection and further improving the quality and reliability of the constructed voice data set.
[0066] S102, sending the speech data to a plurality of convolution kernels of different sizes for convolution processing to obtain multi-scale features, performing feature processing on the multi-scale features to extract spatial information of different spatial granularities, and performing data processing on the spatial information of different spatial granularities to obtain local spatial features of the speech data;
[0067] This step S102 extracts features at different spatial granularities, so that the information of each spatial position is mapped into a higher-dimensional feature space, thereby increasing the expressive power of the model.
[0068] In this step S102, feature processing is performed on the multi-scale features to obtain local spatial features of the speech data, including:
[0069] S1021: performing normalization (BN) processing and activation function (ReLU) processing on the multi-scale features in sequence to obtain optimized features;
[0070] S1022: Performing average pooling (Ave-Pool) down-sampling processing on the optimized features to obtain local spatial features of the speech signal.
[0071] Preferably, in an embodiment of the present invention, three parallel convolution kernels (Conv) of different sizes are selected, and the scales of the three convolution kernels are 11x1, 1x9 and 3x3 respectively.
[0072] Specifically, in this embodiment, step S102 specifically includes:
[0073] S1021: Input the speech data into three parallel convolutions (Conv) with scales of 11x1, 1x9 and 3x3 to extract the frequency domain, time domain and space domain features of the speech, and obtain multi-scale features; this embodiment extracts features from different spatial granularities, so that the information of each spatial position is mapped to a higher-dimensional feature space, thereby increasing the expression ability of the model;
[0074] S1022: The multi-scale features are processed by a normalization layer (BN), an activation function (ReLU) and an average pooling (Ave-Pool) to obtain optimized features. This step can enhance the expressiveness of the model and prevent the problem of gradient disappearance or gradient explosion during model training. The optimized features are input into the average pooling (Ave-pool) to perform feature downsampling to obtain optimized features. The purpose of pooling is to reduce the number of model parameters and reduce the model complexity, thereby preventing the model from overfitting.
[0075] S1023: Add the optimized features obtained in the time domain, frequency domain and space domain element by element according to the channel dimension to obtain the local spatial features of the speech signal.
[0076] S103: Send the speech data to the transformer encoder based on the self-attention mechanism to obtain primary time series features, and perform dimensionality reduction processing on the primary time series features to obtain global time series features of the speech data.
[0077] Specifically, the speech data is first converted into a feature vector through an embedding layer;
[0078] By injecting time series information into the feature vector through positional encoding, a coding feature is obtained, so that the model can distinguish the order relationship in the data sequence;
[0079] The encoded features are input into the multi-head self-attention mechanism, and the attention score is obtained by calculating the query and key of the speech data;
[0080] The softmax of the attention score is calculated to get the weight, and then the weight is weighted with the value (V) to output the attention feature vector Attention (Q, K, V).
[0081]
[0082] Among them, Q, K, and V represent the query, key, and value of the voice data respectively. k refers to the dimension of the key vector;
[0083] The attention feature vector is regularized and then residually connected (Add) with the original speech data, and then subjected to layer normalization (Norm) processing to obtain an enhanced feature vector; the enhanced feature vector is input into a feed-forward neural network consisting of two layers of full connection and one layer of activation function for processing; the output of the feed-forward neural network is regularized and then residually connected (Add) with the enhanced feature vector for the second time, and then subjected to layer normalization (Norm) processing again to obtain primary time series features.
[0084] The primary time series features are input into the pooling layer for maximum pooling processing, so that the dimension of the primary time series features is reduced and redundant information is reduced, and the global time series features of the speech signal are obtained.
[0085] S104: Mapping local spatial features and global temporal features to the same feature space for feature fusion to obtain low-order speech features.
[0086] Specifically, in this step, local spatial features and global temporal features are mapped to the same feature space, and are fused element by element according to the channel dimension to obtain low-order speech features.
[0087] S105, the fused low-order speech features are sent to a high-order feature encoder (such as Figure 5 As shown in the figure, high-order feature learning is performed to obtain high-order speech features. The specific process includes:
[0088] S1051: Input the fused low-order speech features into a 3X3 convolution (Conv) for convolution processing and convert them into high-level features that can be used by the model;
[0089] S1052: The high-level features are processed in sequence by normalization (BN), activation function (ReLU) and average pooling (Ave-Pool) to obtain enhanced features; these processes can generalize the model and reduce the dimension of feature data to reduce the complexity of the model during training;
[0090] S1053: Perform global average pooling on the enhanced features so that the features of each channel are represented by a vector representing the global receptive field of the channel, thereby obtaining compressed features;
[0091] S1054: The compressed features are passed through two fully connected layers (FC) to excite the feature map to obtain the channel attention weight; wherein the activation function of the first fully connected layer is ReLU, and the activation function of the second fully connected layer is Sigmoid;
[0092] S1055: Use the channel attention weight to assign weights to the enhanced features to obtain efficient high-order features, i.e., new feature maps;
[0093] S1056: Inputting the efficient high-order features into the multi-layer combined high-order feature encoder for multiple times to learn and obtain high-order speech features; specifically, this step also includes the following steps:
[0094] S10561: First, by gradually increasing the network depth, richer hierarchical features in the data are extracted. Specifically, by gradually increasing the convolution kernel depth in the high-order feature learning encoder, selecting a suitable pooling layer size, and fully learning effective features, richer hierarchical features in the high-order and efficient features are extracted.
[0095] S10562: Then, use global average pooling to filter and integrate the feature information of all hierarchical features, and average the feature values of each channel of the entire feature map to obtain a high-order speech feature represented by a global feature. Select appropriate pooling to filter effective features from complex features, while reducing the complexity of the model during training; in order to enable the model to be trained on data sets of different lengths and have stronger generalization capabilities. At the end of the feature learning model, use global average pooling to filter and integrate all feature information, and average the feature values of each channel of the entire feature map to obtain a global feature representation.
[0096] like Figure 2 As shown, this embodiment also provides a speech signal feature extraction device 200 based on space and time series, including:
[0097] A voice data collection unit 201, used for collecting voice data;
[0098] The spatial feature extraction unit 202 is used to send the speech data to a plurality of convolution kernels of different sizes for convolution processing to obtain multi-scale features, and perform feature processing on the multi-scale features to obtain local spatial features of the speech data;
[0099] The time series feature extraction unit 203 is used to input the speech data into the transformer encoder based on the self-attention mechanism to obtain primary time series features, and perform dimensionality reduction processing on the primary time series features to obtain global time series features of the speech data;
[0100] The specific structure of the spatial feature and temporal feature extraction unit in this embodiment is as follows: Figure 3 As shown;
[0101] A feature fusion unit 204 is used to map the local spatial features and the global temporal features to the same feature space for feature fusion to obtain low-order speech features;
[0102] The high-order feature learning unit 205 is used to send the fused low-order speech features to the high-order feature encoder for high-order feature learning to obtain high-order speech features.
[0103] like Figure 4 and Figure 5As shown, in this embodiment, the multi-layer combined high-order feature encoder includes a five-layer structure. Specifically, the first to fourth layers include four convolution kernels of different sizes, a normalization layer, a ReLU activation function, an average pooling layer, and a squeeze and excitation attention layer, respectively, wherein the squeeze and excitation attention layer is composed of a global average pooling layer, a fully connected layer, a ReLU activation function, a fully connected layer, a Sigmoid activation function and an attention weight connected in sequence.
[0104] In the fifth layer of high-order feature learning, the depth of the convolution kernel is increased to extract more complex features. After normalization and activation function processing, in order to facilitate the training of speech data of different dimensions, the processed features are unified in spatial dimensions through global average pooling, and the high-order speech features extracted by the high-order feature learning encoder are output.
[0105] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present disclosure is not limited by the described action sequence, because according to the present disclosure, certain steps can be performed in other sequences or simultaneously. In addition, those skilled in the art should also know that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present disclosure.
[0106] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.
[0107] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.
Claims
1. A speech signal feature extraction method based on space and time series, characterized in that: include: Collect voice data; The speech data is sent to a plurality of convolution kernels of different sizes for convolution processing to obtain multi-scale features, and feature processing is performed on the multi-scale features to obtain local spatial features of the speech data; The step of sending the speech data to a plurality of convolution kernels of different sizes for convolution processing to obtain multi-scale features, and performing feature processing on the multi-scale features to obtain local spatial features of the speech data includes: Inputting the speech data into a plurality of parallel convolution kernels of different sizes to extract the frequency domain, time domain and space domain features of the speech to obtain multi-scale features; The multi-scale features are further normalized, activated and averaged-pooled to obtain optimized features; The optimized features obtained in the frequency domain, time domain and space domain are added element by element according to the channel dimension to obtain the local spatial features of the speech signal; The speech data is fed into a transformer encoder based on a self-attention mechanism to obtain primary time series features, and the primary time series features are subjected to dimensionality reduction processing to obtain global time series features of the speech data; The method of feeding the speech data into a transformer encoder based on a self-attention mechanism to obtain primary time series features includes: The speech data is firstly converted into a feature vector through an embedding layer; Injecting time series information into the feature vector through position coding to obtain coding features; Input the encoded features into a multi-head self-attention mechanism, and obtain an attention score by calculating the query and key of the speech data; The softmax of the attention score is calculated to obtain the weight, and then the weight and the value are weighted to output the attention feature vector Attention(Q,K,V). ; Among them, Q, K, and V represent the query, key, and value of the voice data respectively. k refers to the dimension of the key vector; The attention feature vector is regularized and then residually connected with the original speech data, and then subjected to layer normalization to obtain an enhanced feature vector; Inputting the enhanced feature vector into a feedforward neural network consisting of two layers of full connection and one layer of activation function for processing; Regularizing the output of the feedforward neural network, performing a second residual with the enhanced feature vector, and normalizing it again to obtain a primary time series feature; Mapping the local spatial features and the global temporal features to the same feature space for feature fusion to obtain low-order speech features; The fused low-order speech features are sent to a high-order feature encoder for high-order feature learning to obtain high-order speech features; The step of sending the fused low-order speech features to a high-order feature encoder for high-order feature learning to obtain high-order speech features includes: The low-order speech features obtained by the fusion are input into a 3×3 convolution for convolution processing, and converted into high-level features that can be used by the model; The high-level features are sequentially subjected to layer normalization, activation function and average pooling processing to obtain enhanced features; The enhanced features are input into the squeeze and stimulate attention to emphasize the features that are helpful for the target task and obtain efficient high-order features; The efficient high-order features are input into a multi-layer combined high-order feature encoder for multiple times for learning, so as to obtain high-order speech features.
2. The method for extracting speech signal features based on space and time series according to claim 1, characterized in that: in, The number of the multiple parallel convolution kernels of different sizes is three; and the scales of the three convolution kernels are 11x1, 1x9 and 3x3 respectively.
3. The method for extracting speech signal features based on space and time series according to claim 1, characterized in that: in, The step of performing dimensionality reduction processing on the primary time series features to obtain the global time series features of the speech signal includes: The primary time series features are input into the pooling layer for maximum pooling processing, so that the dimension of the primary time series features is reduced and redundant information is reduced, thereby obtaining the global time series features of the speech signal.
4. The method for extracting speech signal features based on space and time series according to claim 1, characterized in that: in, Mapping the local spatial features and the global temporal features to the same feature space for feature fusion to obtain low-order speech features includes: The local spatial features and the global temporal features are mapped to the same feature space, and are fused by adding them element by element according to the channel dimension to obtain low-order speech features.
5. The method for extracting speech signal features based on space and time series according to claim 1, characterized in that: in, The enhanced features are input into the squeeze and stimulate attention to emphasize the features that are helpful for the target task, and obtain efficient high-order features, including: Performing global average pooling on the enhanced features, so that the features of each channel are represented by a vector to represent the global receptive field of the channel, thereby obtaining compressed features; The compressed features are passed through two fully connected layers to excite the feature map to obtain channel attention weights; wherein the activation function of the first fully connected layer is ReLU, and the activation function of the second fully connected layer is Sigmoid; The channel attention weight is used to assign weights to the enhanced features to obtain efficient high-order features.
6. The method for extracting speech signal features based on space and time series according to claim 5, characterized in that: in, The step of inputting the efficient high-order features into a multi-layer combined high-order feature encoder for multiple times to learn and obtain the high-order speech features comprises: extracting richer hierarchical features in the data by gradually increasing the network depth, specifically: extracting richer hierarchical features in the high-order efficient features by increasing the convolution kernel depth in the high-order feature learning encoder layer by layer, selecting a suitable pooling layer size, and fully learning effective features; The last layer of high-order feature encoder uses global average pooling to filter and integrate the feature information of all levels of features, and averages the feature values of each channel of the entire feature map to obtain a high-order speech feature represented by a global feature.
7. A speech signal feature extraction device based on space and time series, used to implement the speech signal feature extraction method based on space and time series as described in any one of claims 1 to 6, characterized in that: include: A voice data collection unit (201), used for collecting voice data; A spatial feature extraction unit (202) is used to send the speech data to a plurality of convolution kernels of different sizes for convolution processing to obtain multi-scale features, and perform feature processing on the multi-scale features to obtain local spatial features of the speech data; A time series feature extraction unit (203) is used to input the speech data into a transformer encoder based on a self-attention mechanism to obtain primary time series features, and perform dimensionality reduction processing on the primary time series features to obtain global time series features of the speech data; A feature fusion unit (204) is used to map the local spatial feature and the global temporal feature to the same feature space for feature fusion to obtain a low-order speech feature; The high-order feature learning unit (205) is used to send the fused low-order speech features to a high-order feature encoder for high-order feature learning to obtain high-order speech features.
Citation Information
Patent Citations
Speech recognition method and device for multi-modal data, household appliance and medium
CN117198278A
Voice signal processing system and method based on layered inattention model
CN117894342A
