Speech recognition method based on inertial measurement unit feature fusion

Abstract

The invention provides a speech recognition method based on inertial measurement unit feature fusion, which comprises the following steps of: merging six-axis data of an accelerometer and a gyroscope into an RGB (Red, Green and Blue) feature map to obtain a time-frequency feature map of the accelerometer and the gyroscope; respectively obtaining high-level features through the multi-layer CNN; the high-level features of the accelerometer and the gyroscope are respectively enhanced through a channel attention module, and mutual calibration is carried out on the features of the accelerometer and the gyroscope through a space attention weight; and performing feature fusion on the features after space attention calibration by using a gating fusion module, performing multi-scale fusion on the fused features through an FPN module, and finally performing classification and recognition through a classifier module. According to the invention, by fusing the multi-axis data of the accelerometer and the gyroscope, the information can be complemented, the effective sensing range can be expanded, the sound characteristics can be restored together, and the recognition of the human voice of the wireless earphone wearer and the sound of the earphone loudspeaker under the ultralow sampling rate can be realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of wireless sensing and identification technology, in particular to a voice recognition method based on fusion of inertial measurement unit features. BACKGROUND

[0002] With the popularity and continuous improvement of wireless earphones, the inertial measurement unit (IMU) built into the earphones has become an important part of wireless earphones. These sensors provide users with a wealth of features, enabling a more intelligent user experience. The inertial measurement unit (IMU) is usually composed of an accelerometer and a gyroscope, which can sense the motion state and position change of the device in real time. The accelerometer is used to measure the acceleration of the device in three-dimensional space, while the gyroscope is used to measure the angular velocity of the device. The high sensitivity and real-time nature of these sensors make them indispensable components of smartphones.

[0003] IMU sensors can capture voice signals propagating through solid media. When the wearer of a wireless earphone speaks and when the earphone's built-in speaker plays voice, it will cause special oscillation of the IMU accelerometer and gyroscope. In the low sampling rate scenario, voice recognition based on the inertial measurement unit faces multiple challenges such as signal overlap interference and information loss. Traditional single-sensor or single-modal processing methods cannot balance feature integrity and semantic recognition accuracy. SUMMARY

[0004] To address the above problems, the present application proposes a voice recognition method based on fusion of inertial measurement unit features, which focuses on collaborative use of sensor data, network architecture optimization, and cross-modal feature fusion. The present application provides a technical solution as follows:

[0005] A voice recognition method based on fusion of inertial measurement unit features, comprising the following steps:

[0006] S1, merge the six-axis data of the accelerometer and gyroscope as the red, green and blue channels respectively to form 2 RGB feature maps, obtaining the time-frequency feature maps of the accelerometer and gyroscope;

[0007] S2, the accelerometer and gyroscope feature maps pass through multiple layers of CNN to obtain high-level features and respectively, each layer of CNN contains convolution, batch normalization, activation and average pooling;

[0008] S3, the high-level features of the accelerometer and gyroscope are enhanced through a channel attention module respectively, obtaining the features after channel attention enhancement and ;

[0009] S4, mutual calibration of the features of the accelerometer and the gyroscope through spatial attention weights;

[0010] S5, feature fusion of the features of the accelerometer and the gyroscope after spatial attention calibration by using a gated fusion module to obtain fused features ;

[0011] S6, fused features are obtained through an FPN module for multi-scale fusion to obtain multi-scale fused features ;

[0012] S7, classification and recognition of the multi-scale fused features by using a classifier module.

[0013] Preferably, the specific steps of S2 are as follows:

[0014] S21, local features are extracted from the features of the accelerometer and the gyroscope through convolution operation respectively;

[0015] S22, normalization processing is performed on the features after convolution to make the mean value 0 and the variance 1, accelerate the training process and improve the stability of the model;

[0016] S23, LeakyReLU activation function is used;

[0017] S24, average pooling is used on the features after activation to down-sample, reduce the spatial size, reduce the computational complexity, and retain important features.

[0018] Preferably, the specific steps of S3 are as follows:

[0019] S31, global average pooling and global maximum pooling are used simultaneously to capture the average information and maximum information of the high-level features of the accelerometer and the gyroscope respectively, and then the average information and the maximum information are added;

[0020] S32, Linear linear transformation is performed to adaptively adjust the weights of each channel to improve the attention of the network to important features;

[0021] S33, Sigmoid function is used to calculate the channel attention weights of the accelerometer and the gyroscope respectively and ;

[0022] S34, the attention weights and are multiplied by the high-level features , respectively, to obtain the features after channel attention enhancement and .

[0023] Preferably, the specific steps of S4 are as follows:

[0024] S41, using global average pooling and global max pooling simultaneously to capture the average information and maximum information of the accelerometer and gyroscope feature channels respectively, and then concatenating the results in the channel dimension;

[0025] S42, performing convolution on the concatenated features;

[0026] S43, using the Sigmoid function to calculate the spatial attention weights of the accelerometer and gyroscope features respectively ;

[0027] S44, multiplying the spatial attention weights of the accelerometer and gyroscope with the features of the other after channel attention enhancement to mutually calibrate the features.

[0028] Preferably, the specific steps of S5 are as follows:

[0029] S51, concatenating the mutually calibrated accelerometer and gyroscope features in the channel dimension;

[0030] S52, performing linear transformation on the concatenated features;

[0031] S53, using the Sigmoid function to calculate the gating signal ;

[0032] S54, using the generated gating signal to weight and combine the mutually calibrated accelerometer and gyroscope features to obtain the fused features .

[0033] Preferably, the specific steps of S6 are as follows:

[0034] S61, performing convolution on the fused features ;

[0035] S62, using the bilinear interpolation method to upsample the convolved data by a factor of 2 to obtain the features ;

[0036] S63, performing convolution on the features to obtain ;

[0037] S64, comparing and​​​​​ The size, right Upsample to the same size as equal;

[0038] S65, and resized Add together to get the features of the two scales fusion .

[0039] Preferably, the specific steps of S7 are:

[0040] S71, flattening the multi-scale fusion features Get two-dimensional tensor features ;

[0041] S72, flattened two-dimensional tensor features After Linear transformation, the dimension is mapped to the hidden layer;

[0042] S73, batch normalization of features after the hidden layer;

[0043] S74, applying an activation function to the batch normalized features;

[0044] S75 uses Linear transformation to map the activated features to the category dimension.

[0045] Compared with the prior art, the beneficial effects achieved by the present invention are:

[0046] By simultaneously utilizing six-axis data from the accelerometer and gyroscope, and using their RGB spectrograms as input to the recognition network, this approach breaks through the limitations of traditional single-modality approaches that rely solely on a single sensor. This multimodal fusion approach not only preserves the unique characteristics of each axis but also effectively reduces information loss in a single modality by integrating complementary information from different sensor modalities, providing a more comprehensive feature representation for subsequent recognition tasks.

[0047] A five-layer convolutional neural network is used as the backbone model to process the RGB spectrograms of the accelerometer and gyroscope separately, enabling parallel feature extraction from both sensor data. This architecture achieves hierarchical feature extraction of sensor data through deep convolution operations, abstracting time-domain correlation features from the raw signals layer by layer. Compared to traditional single-stream networks, this improves the representation of cross-modal signals and provides richer feature input for the subsequent attention mechanism.

[0048] By combining local and spatial attention, it adaptively focuses on "important frequencies" and "key signal regions," improving feature representation capabilities at low sampling rates. It emphasizes the interdependence between channels and, by calculating the attention weight for each channel, highlights important channels and suppresses irrelevant ones, allowing the network to focus more on speech-related feature dimensions. By analyzing channel statistics at each spatial location and generating a spatial attention map, it identifies key spatial regions within the feature map, improving the ability to capture the spatial distribution of speech features.

[0049] By cross-utilizing the channel attention and spatial attention information of the two sub-networks, the attention weights are calibrated to enhance the relevance and complementarity of cross-modal features, rather than a simple parallel branch structure. By cross-referencing the attention weights of the accelerometer and gyroscope (for example, using the gyroscope's channel attention to adjust the accelerometer's spatial attention), the correlation between the two is strengthened.

[0050] The gated fusion model is used to dynamically adjust the feature weights of the accelerometer and gyroscope, and the feature pyramid network (FPN) is used to integrate multi-scale information. The features of the accelerometer and gyroscope are projected into a unified space to achieve multi-scale information complementarity and improve the semantic recognition accuracy at low sampling rates. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0052] Figure 1 It is the overall process structure diagram of the method of the present invention;

[0053] Figure 2 This is a structural diagram of one layer of the multi-layer CNN of the present invention;

[0054] Figure 3 It is the channel attention structure diagram;

[0055] Figure 4 Compute a structure graph for spatial attention weights;

[0056] Figure 5 Characteristic structure diagram for mutual calibration between accelerometer and gyroscope;

[0057] Figure 6 This is a diagram of the gated fusion structure;

[0058] Figure 7 This is the structure diagram of the FPN module;

[0059] Figure 8 This is the structural diagram of the classifier. DETAILED DESCRIPTION

[0060] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0061] In order to make the above-mentioned objects, features and effects of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0062] Example 1: A speech recognition method based on inertial measurement unit feature fusion, such as Figure 1 As shown, the following steps are included:

[0063] S1 generates time-frequency feature maps of the accelerometer and gyroscope corresponding to the audio. This is done by combining the six-axis data of the accelerometer and gyroscope into two RGB feature maps: the red channel (x-axis), the green channel (y-axis), and the blue channel (z-axis).

[0064] Generating a time-frequency feature map, the feature input for the entire system, involves the following steps: Data preprocessing: A mean filter is applied to the accelerometer (or gyroscope)'s (X, Y, and Z) six-axis data to remove DC offsets and then normalize it. Time-frequency spectrum generation: Short-time Fourier transforms are performed on the preprocessed X, Y, and Z-axis data, mapping them into red, green, and blue time-frequency spectrograms, respectively. Feature map synthesis: The X-axis time-frequency spectrum is used as the red channel, the Y-axis time-frequency spectrum as the green channel, and the Z-axis time-frequency spectrum as the blue channel, and these are combined to generate a complete RGB feature map. The accelerometer and gyroscope six-axis data are processed through these steps to generate RGB feature maps for each. This RGB feature map, which serves as input for the subsequent system, contains the time-frequency feature information for the six axes of the accelerometer and gyroscope.

[0065] Accelerometers sense the physical vibrations generated by sound, whether it's facial muscle movement or bone vibrations caused by the wearer's voice, or vibrations transmitted through the device's structure by sound emitted by the headphone speaker. Accelerometers convert these vibrations into electrical signals. The response along three axes (X, Y, and Z) provides information about the direction and intensity of the sound vibrations. For example, when a headphone speaker emits sound, accelerometer data along different axes can reflect the propagation characteristics of the sound wave through the device.

[0066] The gyroscope can perceive angular motion and rotational vibration caused by sound, whether it is the rotational signal generated by the movement of the lower jaw, lips and other parts of the earphone wearer when speaking, or the angular displacement change caused by the vibration of the device shell and internal structure caused by the sound of the earphone speaker. The response of the three axes (X, Y, Z axis) can provide the rotational direction and angular velocity information of the sound vibration. For example, when the earphone plays sound, the gyroscope data in different axes can reflect the rotational vibration characteristics of the device structure in the sound wave propagation, and its signal-to-noise ratio for sound is usually higher than that of the accelerometer, and it is easier to capture the speaker signal at a low sampling rate.

[0067] S2, the accelerometer and gyroscope feature maps are respectively obtained by a multi-layer CNN (convolutional neural network), and each layer of the CNN includes convolution, batch normalization, activation and average pooling, as shown in Figure 2 .

[0068] CNN is a deep learning model specially used for processing data with grid structure (such as the RGB map of the accelerometer and gyroscope), which can extract features in grid data through convolution, activation, batch normalization layer, pooling and full connection, etc. Convolution is the core operation of CNN, which extracts data by sliding the convolution kernel on the input data; batch normalization is performed on the input to make its mean value 0 and variance 1, which accelerates the training process and improves the stability of the model; the activation function introduces nonlinear expression ability, so that the network can learn more complex features and decision boundaries. The traditional ReLU has a gradient of 0 at negative input, which may cause some neurons to never activate. In the present application, the Leaky ReLU activation function is used, which retains the small gradient in the negative interval, so that the parameters still have the opportunity to update; after the activation layer, the features are down-sampled by average pooling to reduce the spatial size, reduce the computational complexity, and retain important features to obtain high-level features of the accelerometer and gyroscope. The specific expression is as follows.

[0069] S21, the accelerometer and gyroscope feature data are respectively subjected to convolution operation to extract local features;

[0070]

[0071] wherein is the accelerometer feature, is the convolution kernel, and the bias term (Bias) is a learnable scalar parameter. Similarly, the convolution features of the gyroscope can be obtained by the same method .

[0072] S22, the data after convolution is subjected to normalization processing to make its mean value 0 and variance 1, which accelerates the training process and improves the stability of the model;

[0073]

[0074] where and are the mean and variance of the feature along the channel dimension for each batch of data, is the feature of the accelerometer or gyroscope after passing through the convolution layer, i.e. or , is a small constant to prevent division by zero.

[0075] S23, using LeakyReLU activation function, ensures that all neurons can participate in training;

[0076]

[0077] where is the feature of the accelerometer or gyroscope after passing through the batch normalization layer, i.e. , is a small positive slope (such as 0.01), and the negative interval gradient is .

[0078] S24, using average pooling on the activated feature for down-sampling, reducing spatial size, reducing computational complexity, while preserving important features;

[0079]

[0080] where is the activated feature, i.e. ; and are the height and width of , respectively; are the channel index, vertical position and horizontal position of , respectively.

[0081] The above process can be represented as:

[0082]

[0083]

[0084] where and are the accelerometer and gyroscope features, , , and are the convolution, batch normalization, activation and average pooling operations, respectively. After the above steps, the high-level features of the accelerometer and gyroscope and can be extracted.

[0085] S3, high-level features of the accelerometer and gyroscope are enhanced by channel attention modules respectively, and the structure model is as shown in Figure 3 .

[0086] Channel attention is an attention mechanism that focuses on adjusting the importance of different channels in the feature map. By dynamically assigning weights to each channel, the model can pay more attention to important channels, thereby improving the expression ability of features and the performance of the model.

[0087] Max pooling selects the maximum value within the pooling window, which usually corresponds to the most prominent features in the image, such as edges, textures, etc. in the image. In this way, the most important features can be focused on, while some relatively less important details can be ignored. The calculation formula of is as follows: , where is the feature of the accelerometer or gyroscope, is the size of the pooling window, is the step size, are the channel index, vertical position and horizontal position of respectively.

[0088] Linear transformation is a basic operation structure in neural networks. Its main function is to perform linear transformation on input data, so as to learn the linear relationship of input data. In this invention, the multi-dimensional accelerometer and gyroscope input features are flattened into long vectors through Linear fully connected layer: , where is the input feature, is the bias vector, is the weight matrix, the row and column of which depend on the dimension of the feature before and after flattening.

[0089] Sigmoid function is an activation function that maps any real number in the range to the interval (0, 1): , where is the input feature. The specific expression of S3 is as follows.

[0090] S31, global average pooling and global maximum pooling are used simultaneously to capture the average information and maximum information of the high-level features of the accelerometer and gyroscope respectively, and then the average information and maximum information results are added to enhance the perception ability of the model to channel information;

[0091]

[0092]

[0093] wherein and are high-level features of the accelerometer and gyroscope through a multi-layer CNN network, is an average pooling operation on the input, is a max pooling operation on the input.

[0094] S32, a Linear transformation is performed to adaptively adjust the weights of each channel to improve the attention of the network to important features;

[0095]

[0096]

[0097] wherein and are features of the accelerometer and gyroscope, respectively, is a feature obtained through step S31, is a Linear transformation operation on the input.

[0098] S33, a Sigmoid function is used to calculate the channel attention weights of the accelerometer and gyroscope, respectively and ;

[0099]

[0100]

[0101] wherein and are Linear transformed accelerometer and gyroscope features, respectively; is a Sigmoid function, and the channel attention weights and with values between 0 and 1 can be obtained.

[0102] S34, the attention weights and are multiplied by the high-level features , to obtain the features after channel attention enhancement and ;

[0103]

[0104]

[0105] wherein and are the channel attention weights of the accelerometer features and the gyroscope features, respectively, wherein and Accelerometer and gyroscope features through multi-layer CNN network.

[0106] S4, mutual calibration of accelerometer and gyroscope features through spatial attention weight, enhancing expression ability and consistency of features, structure model as shown in Figure 4 and Figure 5 , specific expression as follows.

[0107] S41, simultaneously using global average pooling and global maximum pooling, respectively capturing average information and maximum information of accelerometer and gyroscope feature channels and , then splicing the results in channel dimension to enhance spatial features;

[0108]

[0109]

[0110] wherein and are accelerometer and gyroscope features after channel attention enhancement, is average pooling operation on input, is maximum pooling operation on input, is splicing and in channel dimension.

[0111] S42, convolution on spliced features;

[0112] S43, using Sigmoid function, respectively calculating spatial attention weights of accelerometer and gyroscope features and .

[0113]

[0114]

[0115] wherein and are maximum pooled and average pooled features of accelerometer and gyroscope after splicing, is convolution operation on input, is Sigmoid activation operation on input, obtaining spatial attention weights and between 0 and 1.

[0116] S44, spatial attention weights of accelerometer and gyroscope and Multiply each other's channel attention enhanced features and , to calibrate the features with each other, the structure diagram is as follows Figure 5 shown.

[0117]

[0118]

[0119] in and They are the features after attention enhancement of the accelerometer and gyroscope channels, and is the spatial attention weight of the accelerometer feature and the gyroscope feature, and are the accelerometer and gyroscope characteristics after mutual calibration.

[0120] S5, the features of the accelerometer and gyroscope after spatial attention calibration are fused using the gated fusion module. The structural model is as follows Figure 6 As shown, gated fusion dynamically controls the fusion ratio of multi-branch features through a gating mechanism, linearly transforms the input features, and generates a gating value of 0~1. , and finally use the gate value Fusion of accelerometer and gyroscope features. The specific expression is as follows.

[0121] S51, concatenates the accelerometer and gyroscope features in the channel dimension;

[0122]

[0123] in are the accelerometer and gyroscope features calibrated against each other through channel attention, It concatenates the two inputs in the channel dimension.

[0124] S52, the concatenated features undergo linear transformation;

[0125]

[0126] in is the accelerometer and gyroscope features stitched in the channel dimension, Performs a Linear transformation on the input.

[0127] S53, calculate the gating signal using the Sigmoid function ;

[0128]

[0129] in is the feature after Linear transformation, Perform Sigmoid activation on the input.

[0130] S54, using the generated gating signal Perform weighted combination of accelerometer and gyroscope features to obtain fusion features ;

[0131]

[0132] in is the gating signal, and are the accelerometer and gyroscope features calibrated with each other through the spatial attention module, respectively. is the first parameter of the gated fusion module, It is the second parameter of the gated fusion module. is the fused feature.

[0133] S6, the fused features are fused at multiple scales through the FPN (Feature Pyramid Network) module. The structural model is as follows Figure 7 As shown in Figure 1, FPN fuses high-level semantic information with low-level detail information through upsampling, forming a bottom-up, top-down, and horizontally connected structure. This generates a feature pyramid with rich semantics and details, effectively improving the fusion of accelerometer and gyroscope features. The specific expression is as follows.

[0134] S61, for the fused features Perform convolution;

[0135] S62, upsample the convolved data using bilinear interpolation times, get the features ;

[0136]

[0137] in, is the upsampling function, is the convolution function, is the fusion feature processed by gated fusion technology, Indicates the magnification factor, which expands the original space size times; Indicates that the height and width of the final feature map are Corresponding dimensions times.

[0138] S63, features The convolution operation is performed to obtain ;

[0139]

[0140] in is the convolution function, It is the fusion feature of the accelerometer and gyroscope after being processed by gated fusion technology.

[0141] S64, comparison and Due to the size of The size is small and it is upsampled to make its size equal;

[0142] S65, and resized Add together to get the features of the two scales fusion .

[0143] S7, through the classifier module, the fused features are classified and identified. The model is as follows Figure 8 The specific expression is as follows.

[0144] S71, flattening the multi-scale fusion features Get a two-dimensional tensor of dimension [BatchSize, Channel×Width×High] ;

[0145]

[0146] in is the output of the FPM module, Flatten the input, that is, the original 4D tensor [BatchSize, Channel, Width, High] is flattened into a 2D tensor [BatchSize, Channel×Width×High].

[0147] S72: The flattened features are transformed by Linear to map the dimension Channel×Width×High to the dimension Channel×Width×High / / N (hidden layer, N is a positive number).

[0148]

[0149] in is the feature after flattening, Perform Linear transformation on the input features;

[0150] S73, batch normalization of features after the hidden layer;

[0151] S74, applying an activation function to the batch-normalized features;

[0152]

[0153] wherein is the feature after linear transformation, is the batch normalization to the input, is the ReLu activation function.

[0154] S75, applying a Linear transformation to map the input features from Channel x Width x Height / / N dimension to class dimension (number of classes of words or numbers to be recognized).

[0155]

[0156] wherein is the feature after ReLu function activation.

[0157] Embodiment 2: a computer readable storage medium of the present embodiment, which stores a computer program, the program being executed by a processor to implement the steps of the speech recognition method based on fusion of inertial measurement unit features in embodiment 1.

[0158] The computer readable storage medium of the present embodiment can be an internal storage unit of the terminal, such as a hard disk or a memory of the terminal; the computer readable storage medium of the present embodiment can also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card, a secure digital card, a flash memory card, etc. equipped on the terminal; further, the computer readable storage medium can include both the internal storage unit and the external storage device of the terminal.

[0159] The computer readable storage medium of the present embodiment is used to store the computer program and other programs and data required by the terminal, and the computer readable storage medium can also be used to temporarily store data that has been output or will be output.

[0160] Embodiment 3: a computer device of the present embodiment, comprising a processor, a memory, and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to implement the steps of the speech recognition method based on fusion of inertial measurement unit features in embodiment 1.

[0161] In this embodiment, the processor can be a central processing unit, and can also be other general-purpose processors, digital signal processors, application-specific integrated circuits, ready programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or can also be any conventional processor, etc. The memory can include read-only memory and random access memory, and provide instructions and data to the processor. Part of the memory can also include non-volatile random access memory, for example, the memory can also store device type information.

[0162] Those skilled in the art can clearly understand that each embodiment can be realized by means of software and necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of software product, and the computer software product can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the method described in each embodiment or some parts of the embodiment.

[0163] The above only describes the preferred embodiments of the present application and is not used to limit the present application. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can modify the technical solutions described in the foregoing embodiments or make equivalent replacements to some technical features. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A speech recognition method based on inertial measurement unit feature fusion, characterized in that: The following steps are involved: S1: Combine the six-axis data of the accelerometer and gyroscope as the red channel, green channel, and blue channel into two RGB feature maps to obtain the time-frequency feature maps of the accelerometer and gyroscope; S2, accelerometer and gyroscope feature maps are obtained through multi-layer CNN respectively. and ,Each layer of CNN contains convolution, batch normalization, activation and average pooling; S3, the high-level features of the accelerometer and gyroscope are enhanced by the channel attention module respectively to obtain the features after channel attention enhancement and ; S4, calibrates the accelerometer and gyroscope features to each other through spatial attention weights; S5, the features of the accelerometer and gyroscope after spatial attention calibration are fused using the gated fusion module to obtain the fused features ; S6, fusion features Through the FPN module, multi-scale fusion is performed to obtain multi-scale fusion features ; S7, through the classifier module, multi-scale fusion features Classify and identify.

2. The speech recognition method based on inertial measurement unit feature fusion according to claim 1, characterized in that: The specific steps of S2 are: S21, the accelerometer and gyroscope feature data are subjected to convolution operations to extract local features; S22, normalizes the convolutional features to make their mean 0 and variance 1, which speeds up the training process and improves the stability of the model; S23, using LeakyReLU activation function; S24, uses average pooling to downsample the activated features to reduce the spatial size and computational complexity while retaining important features.

3. The method for speech recognition based on inertial measurement unit feature fusion according to claim 2, characterized in that: The specific steps of S3 are: S31, uses global average pooling and global maximum pooling at the same time to capture the average information and maximum information of the accelerometer and gyroscope high-level features respectively, and then adds the average information and maximum information results; S32, after Linear transformation, adaptively adjusts the weight of each channel to increase the network's attention to important features; S33, using the Sigmoid function, calculates the channel attention weights of the accelerometer and gyroscope respectively and ; S34, attention weight and Multiply by high-level features , get the features after channel attention enhancement and .

4. The method for speech recognition based on inertial measurement unit feature fusion according to claim 3, characterized in that: The specific steps of S4 are: S41, using global average pooling and global maximum pooling to capture accelerometer and gyroscope feature channels respectively and The average information and maximum information of the data are then concatenated in the channel dimension. S42, convolution is performed on the concatenated features; S43, using the Sigmoid function, calculates the spatial attention weights of the accelerometer and gyroscope features respectively and ; S44, the spatial attention weights of the accelerometer and gyroscope and Multiply each other's channel attention enhanced features and , to calibrate the features against each other.

5. The method for speech recognition based on inertial measurement unit feature fusion according to claim 4, characterized in that: The specific steps of S5 are: S51, stitching the mutually calibrated accelerometer and gyroscope features in the channel dimension; S52, the concatenated features undergo linear transformation; S53, calculate the gating signal using the Sigmoid function ; S54, using the generated gating signal The accelerometer and gyroscope features that have been calibrated with each other are weighted and combined to obtain the fusion feature .

6. The method for speech recognition based on inertial measurement unit feature fusion according to claim 5, characterized in that: The specific steps of S6 are: S61, for the fused features Perform convolution; S62, upsample the convolved data using bilinear interpolation times, get the features ; S63, features The convolution operation is performed to obtain ; S64, comparison and The size, right Upsample to the same size as equal; S65, and resized Add together to get the features after fusion of the two scales .

7. The method for speech recognition based on inertial measurement unit feature fusion according to claim 6, characterized in that: The specific steps of S7 are: S71, flattening the multi-scale fusion features Get two-dimensional tensor features ; S72, flattened two-dimensional tensor features After Linear transformation, the dimension is mapped to the hidden layer; S73, batch normalization of features after the hidden layer; S74, applying an activation function to the batch normalized features; S75 uses Linear transformation to map the activated features to the category dimension.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the speech recognition method based on inertial measurement unit feature fusion as described in any one of claims 1 to 7 are implemented.

9. A computer device comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the speech recognition method based on inertial measurement unit feature fusion as described in any one of claims 1 to 7 are implemented.

Images