A cough recognition method based on multi-channel Conformer

Abstract

The present invention relates to a cough recognition method based on a multi-channel Conformer, comprising the following steps: S1, obtaining different acceleration data signals generated by different movements of an object, namely a first acceleration data signal and a second acceleration data signal, wherein the first acceleration data signal is three-axis acceleration data including angular motion; the second acceleration data signal is three-axis acceleration data including gravity; S2, labeling the acceleration data signals obtained in step S1 according to different movements; S3, preprocessing the labeled data and dividing the data into small segments to construct training and test sets; S4, constructing a multi-channel Conformer network, inputting different acceleration data signals through different channels, and training the network model based on the multi-channel Conformer. The cough recognition method of the present invention improves recognition accuracy and model classification performance, and the trained model can accurately and quickly recognize human movements such as coughs.

Description

Technical Field

[0001] The present invention belongs to the field of health monitoring, and in particular relates to a cough recognition method based on a multi-channel Conformer. Technical Background

[0002] In the current health monitoring field, cough monitoring mainly includes devices based on sound recognition and recognition devices based on the combination of three-axis accelerometers and neural networks. Devices based on sound recognition are easily affected by surrounding noise, have low recognition accuracy, poor reliability, are time-consuming, and have high requirements for analysts. Recognition devices based on the combination of three-axis accelerometers and neural networks can effectively identify coughs and abnormal breathing.

[0003] The patent application number 202110103360.4 discloses a three-axis acceleration activity recognition method based on CNN+Transformer, which can recognize human motion through a sequential network of 1D CNN and Transformer. However, it still has the problem of high memory usage in the Transformer network resulting in slow recognition speed and weak interactivity between local features and global features.

[0004] Patent application number 202210153909.5 discloses an end-to-end bone-conducted and air-conducted speech joint recognition method, which realizes speech recognition based on the built Conformer and multimodal Transducer network. However, since its branch network is a Conformer architecture, it will inevitably cause the Transformer network inside the Confomer to have large memory usage and be insensitive to the characteristics of long sequence data.

[0005] Due to the limitations of algorithm development, existing network models have difficulty in extracting local features while retaining global features as much as possible while training and converging quickly, thus affecting the accuracy and speed of recognition.

[0006] Therefore, at this stage, there is still a lack of a network that has fast recognition speed, takes into account both local and global features, and can effectively extract long sequence features. Summary of the Invention

[0007] In order to solve the above problems, the present invention proposes a cough recognition method based on a multi-channel Conformer with fast recognition speed, taking into account both local and global features, and strong generalization ability.

[0008] In order to solve the above problems, the technical solution adopted by the present invention is:

[0009] A cough recognition method based on a multi-channel Conformer comprises the following steps:

[0010] S1. Obtain different acceleration data signals generated by different actions of the object, namely a first acceleration data signal and a second acceleration data signal, wherein:

[0011] The first acceleration data signal is three-axis acceleration data including angular motion;

[0012] The second acceleration data signal is three-axis acceleration data including gravity;

[0013] S2, marking the acceleration data signal obtained in step S1 according to different actions;

[0014] S3. Preprocess the labeled data and divide the data into small segments to construct training and test sets;

[0015] S4. Construct a multi-channel Conformer network. Different acceleration data signals are input through different channels. The multi-channel Conformer network extracts local and global features and performs feature fusion on all channels. A network model based on the multi-channel Conformer is trained to obtain a trained multi-channel Conformer network. The model is evaluated and the optimal network model obtained during the current training process is saved. The classification result output by the multi-channel Conformer network is the probability that the action is a cough.

[0016] Furthermore, in step S4, the specific formula of the multi-channel Conformer network is as follows:

[0017]

[0018] Where x i is the input variable, FFN refers to the feedforward module, Q, K, and V are three matrices of the same size obtained by linear transformation of the input feature variables.

[0019] It is obtained by Q being sparse according to probability,

[0020] d k is the dimension of a Q or K vector,

[0021] Softmax is the activation function;

[0022] Inf refers to the multi-head sparse self-attention module;

[0023] Conv refers to the convolution module;

[0024] Layernorm means layer normalization,

[0025] y i is the output variable;

[0026] The first acceleration data signal and the second acceleration data signal are respectively connected to the stem module through a channel for input. Each channel performs a time series processing, and the feature extraction of each channel is independent. The three-axis acceleration data including angular motion uses a single-layer multi-channel Conformer, and the three-axis acceleration data including gravity uses a three-layer multi-channel Conformer.

[0027] The multi-channel Conformer adopts the parallel coupling structure of CNN and Informer, merges multiple channels through the Concatenate layer, connects to the fully connected layer, and finally outputs through Softmax.

[0028] Furthermore, in step S4, the training process of the multi-channel Conformer network model is visualized in real time using Visdom, and a confusion matrix is ​​used to obtain the optimal network model, and the optimal model is saved.

[0029] Furthermore, in step S1, the subject performs four designated actions according to the instructions, namely coughing, speaking, swallowing and clearing the throat.

[0030] Furthermore, in step S2, the method used for data labeling is: let the subject start to perform different output actions, and another person records the video on the side to record the timestamps of different actions.

[0031] Furthermore, in step S3, the specific method of preprocessing and dividing the data set is:

[0032] The missing values ​​in the acquired acceleration data are filled with linear interpolation to reduce the impact of missing data during sensor acquisition on the results;

[0033] The acquired acceleration data is subjected to denoising processing using a Butterworth filter to reduce noise interference;

[0034] Standardize the denoised data to improve the speed of network model solution;

[0035] The standardized data is divided into small segments using a sliding window method, and the data set is divided into 70% as training samples and 30% as test samples.

[0036] Furthermore, the fixed length of the sliding window is 64, the time signal in the sliding window is sampled every 3.2 seconds, and there is a 50% overlap rate between two adjacent sliding windows to prevent leakage of valid data information and obtain more feature information.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] First, the present invention fully combines the advantages of the multi-channel Conformer network. The multi-channel Conformer used in the present invention is a parallel network of CNN and Informer. The Informer network replaces the Transformer structure, and a parallel coupling structure of CNN and Informer is adopted. The advantages of CNN are used to extract local features, and the advantages of Informer are used to extract global features. This can solve the problems of high memory usage and insensitivity to long sequence feature information in the original Conformer.

[0039] Second, the present invention leverages the advantages of multiple channels to train the model by using each sensor data as a separate input. This allows for full extraction of features from each channel, learning features from each time series, and leveraging the advantages of a multi-channel Conformer to enable interaction between local and global features. Compared with traditional machine learning for human cough recognition, the multi-channel structure enables the acquisition of more effective features for the same action, improving recognition accuracy and model classification performance. The trained model can accurately and quickly recognize actions such as human coughs. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a flow chart of the main steps of the present invention;

[0041] Figure 2 This is a network model diagram in an example of the present invention. DETAILED DESCRIPTION

[0042] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0043] A cough recognition method based on multi-channel Conformer, such as Figure 1 As shown, the following steps are included:

[0044] (1) having multiple subjects wear sensors at their suprasternal incisions, wherein the sensors include a three-axis accelerometer, a gravity accelerometer, and a gyroscope, wherein the three-axis accelerometer is used to obtain acceleration data generated by different movements of the subjects;

[0045] The gravity accelerometer is used to obtain three-axis acceleration data including gravity generated by different movements of the acquisition object;

[0046] The gyroscope is used to obtain gyroscope sensor data generated by different actions of the acquisition object;

[0047] (2) Label the data collected by the sensor according to different actions;

[0048] (3) Preprocess the labeled data and divide the data into small segments by sliding windows to construct training and test sets;

[0049] (4) Construct a multi-channel Conformer network. Different sensor signals are input through different channels. Local features and global features are extracted through the multi-channel Conformer network and features are fused for all channels. A network model based on the multi-channel Conformer is trained to obtain a trained network model. The model is evaluated and the optimal network model is saved. The classification result output by the network model is the probability that the action is a cough.

[0050] Furthermore, in step (1) data collection, 10 data collection subjects were fitted with sensors on their suprasternal incisions. The sensors included a three-axis accelerometer, a gravity accelerometer, and a gyroscope. The three-axis accelerometer collected acceleration data in the X (left and right), Y (up and down), and Z (front and back) directions, and recorded time series data. The subjects were instructed to perform four designated actions, namely, coughing, speaking, swallowing, and clearing the throat. The coughing, swallowing, and clearing the throat were each performed 10 times, and the speaking action lasted for 20 seconds. Five sets of experiments were repeated, and raw data were collected for subsequent processing.

[0051] Furthermore, in step (2), the method used for data labeling is: attaching the sensor to the suprasternal incision and starting it, starting different output actions, another person recording on one side to mark the actions, recording the timestamps of different actions, and labeling different data for subsequent training.

[0052] Furthermore, in step (3), the specific method of preprocessing and dividing the data set is as follows: if the data collected by the sensor contains noise and data missing, the missing values ​​in the collected sensor data are filled by linear interpolation; the Butterworth filter is used to reduce the noise of the data; and the data is processed by standard deviation normalization (Z-Score), the formula is as follows:

[0053]

[0054] In the formula, σ is the standard deviation of the original data, x is the original data, is the mean of the original data, and z is the standardized data. The standardized data is divided into small segments using a sliding window method, which expands the number of training samples. The processed data set is divided into 70% as training samples and 30% as test samples.

[0055] Furthermore, in the sliding window processing described in the previous step, the fixed length of the sliding window is 64, and the time signal in the sliding window is sampled every 3.2 seconds, thereby dividing each input sequence into a separate data graph. There is a 50% overlap rate between two adjacent windows to obtain more feature information, which can effectively reduce data leakage.

[0056] Furthermore, in step (4), in the model construction, the model used is a multi-channel Conformer network model, and the network model is a Conformer network with multi-channel input. Compared with the original Conformer network, the Transformer structure in the original network is improved, and the Transformer structure is replaced by the Informer network. At the same time, a CNN and Informer parallel coupling structure is adopted, and multiple channels are used to input multiple features. Each layer of the Conformer network belongs to a CNN and Informer parallel network, and feature interaction is performed through a feature coupling unit (FCU). The CNN part provides fine local features, and the Informer part provides global features, which greatly improves the speed of long sequence reasoning. The multi-channel Conformer of the present invention can retain global features while extracting local features.

[0057]

[0058]

[0059] Where x i is the input variable, FFN refers to the feedforward module, Q, K, and V are three matrices of the same size obtained by linear transformation of the input feature variables. It is obtained by probabilistically sparsifying Q, dk is the dimension of a Q or K vector, and Softmax is the activation function. Inf refers to the multi-head sparse self-attention module, Conv refers to the convolution module, Layernorm represents layer normalization, and y i is the output variable.

[0060] Furthermore, the specific structure of the model described in the previous step is as follows: Figure 2As shown, three-axis acceleration data (Accel), gyroscope sensor data (Gyro), and three-axis acceleration data including gravity (Total-Accel) are connected to the stem module through three channels for input. The stem module is used to extract initial shallow features. A single-layer Conformer is used for the three-axis acceleration data (Accel) and gyroscope sensor data (Gyro), while a three-layer Conformer is used for the three-axis acceleration data including gravity (Total-Accel). The multiple channels are merged through the Concatenate layer and then connected to the fully connected layer. Finally, the probabilities of different actions are output through Softmax. Each channel processes a time series, and the feature extraction of each channel is independent, resulting in an independent feature map for each time series. Multi-channel input extracts rich features while accelerating model training.

[0061] Furthermore, in step (4), during model training, the training process of the multi-channel Conformer network model is visualized in real time using Visdom, and a confusion matrix is ​​used to obtain the optimal network model, and the model is saved.

[0062] The present invention fully combines the advantages of the multi-channel Conformer network. The multi-channel Conformer used in the present invention is a parallel network of CNN and Informer. It uses the advantages of CNN to extract local features and uses the advantages of Informer to extract global features, solving the problem of high memory usage in the original Conformer. It uses the advantages of multi-channel to train the model with each sensor data as a separate input. Features can be learned from each long time series. The advantages of the multi-channel Conformer are used to enable local features to interact with global features. The multi-channel Conformer fully combines the advantages of the two, reduces memory usage, and solves the problem of the original Conformer network being insensitive to long time series feature information. The model obtained after training can accurately identify human actions such as coughing.

[0063] Example

[0064] The inventive method is specifically described below by way of examples:

[0065] Step 1: Multiple subjects are fitted with sensors on their upper sternal incisions. The sensors include a three-axis accelerometer, a gravity accelerometer, and a gyroscope. The three-axis accelerometer is used to obtain acceleration data generated by different movements of the subjects.

[0066] The gravity accelerometer is used to obtain three-axis acceleration data including gravity generated by different movements of the acquisition object;

[0067] The gyroscope is used to obtain gyroscope sensor data generated by different actions of the acquisition object;

[0068] Step 2: Attach the accelerometer containing the gyroscope to the suprasternal incision and activate it, starting different output actions. Another person records the video on one side to record the timestamps of different actions, and labels the data collected by the sensor according to different actions.

[0069] Step 3: Fill missing values ​​in the labeled data using linear interpolation, and perform noise reduction and normalization on the data. The processed data is then divided into small segments using a sliding window with a fixed length of 64 and an overlap rate of 50% to construct training and test sets.

[0070] Step 4: Construct a multi-channel Conformer network. Different sensor signals are input after shallow feature processing by Stem through different channels. Local and global features are extracted through the multi-channel Conformer network and feature fusion is performed on all channels. The network model based on the multi-channel Conformer is trained. The training process uses Visdom real-time visualization training and confusion matrix analysis of training results to obtain the optimal network model, evaluate the model, and save the optimal network model.

[0071] This method replaces the Transformer architecture with an Informer network, employing a parallel coupling structure between a CNN and an Informer. The CNN and Informer networks interact via a Feature Coupling Unit (FCU). Multi-channel inputs are used for multiple features, and a multi-channel Conformer network is used to learn the features of sensor data from different channels. This effectively extracts both local and global features from long sequences of different motion data, reduces memory usage, and improves recognition speed, achieving accurate cough recognition.

[0072] This paper addresses the existing Conformer network's high memory usage and poor feature extraction on long sequences of data, which limits cough recognition speed and accuracy. Instead, it proposes a cough recognition method based on a multi-channel Conformer. This method replaces the Transformer architecture with an Informer network, employing a parallel coupling structure between a CNN and an Informer. The CNN and Informer networks interact via a Feature Coupling Unit (FCU), using multiple channels for inputting multiple features. The multi-channel Conformer network learns the features of sensor data from different channels, effectively extracting both local and global features from different long sequences of motion data. This reduces memory usage, improves recognition speed, and enables accurate cough recognition.

[0073] The above only describes in detail the preferred embodiments of the present invention, but the present invention is not limited to the above embodiments. Various changes can be made within the knowledge of ordinary technicians in this field without departing from the purpose of the present invention, and various changes should be included in the scope of protection of the present invention.

Claims

1. A cough recognition method based on multi-channel Conformer, characterized in that: The following steps are involved: S1. Obtain different acceleration data signals generated by different actions of the object, which are respectively a first acceleration data signal and a second acceleration data signal, wherein: The first acceleration data signal is three-axis acceleration data including angular motion; The second acceleration data signal is three-axis acceleration data including gravity; S2, marking the acceleration data signal obtained in step S1 according to different actions; S3. Preprocess the labeled data and divide the data into small segments to construct training and test sets; S4. Construct a multi-channel Conformer network. Different acceleration data signals are input through different channels. The multi-channel Conformer network extracts local and global features and performs feature fusion on all channels. A network model based on the multi-channel Conformer is trained to obtain a trained multi-channel Conformer network. The model is evaluated and the optimal network model obtained during the current training process is saved. The classification result output by the multi-channel Conformer network is the probability that the action is a cough. In step S4, the specific formula of the multi-channel Conformer network is as follows: Where x i is the input variable, FFN refers to the feedforward module, Q, K, and V are three matrices of the same size obtained by linear transformation of the input feature variables. It is obtained by Q being sparse according to probability, d k is the dimension of a Q or K vector, Softmax is the activation function; Inf refers to the multi-head sparse self-attention module; Conv refers to the convolution module; Layernorm means layer normalization, y i is the output variable; The first acceleration data signal and the second acceleration data signal are respectively connected to the stem module through a channel for input. Each channel performs a time series processing, and the feature extraction of each channel is independent. The three-axis acceleration data including angular motion uses a single-layer multi-channel Conformer, and the three-axis acceleration data including gravity uses a three-layer multi-channel Conformer. The multi-channel Conformer adopts the parallel coupling structure of CNN and Informer, merges multiple channels through the Concatenate layer, connects to the fully connected layer, and finally outputs through Softmax.

2. The cough recognition method based on multi-channel Conformer according to claim 1, characterized in that: In step S4, the training process of the multi-channel Conformer network model is visualized in real time using Visdom, and a confusion matrix is ​​used to obtain the optimal network model, and the optimal model is saved.

3. The cough recognition method based on multi-channel Conformer according to claim 1, characterized in that: In step S1, the subject performs four designated actions according to the instructions, namely coughing, speaking, swallowing and clearing the throat.

4. The cough recognition method based on multi-channel Conformer according to claim 1, characterized in that: In step S2, the data marking method is as follows: let the subject start to perform different output actions, and another person records the video on the side to record the timestamps of different actions.

5. The cough recognition method based on multi-channel Conformer according to claim 1, characterized in that: In step S3, the specific method of preprocessing and dividing the data set is: The missing values ​​in the acquired acceleration data are filled with linear interpolation to reduce the impact of missing data during sensor acquisition on the results; The acquired acceleration data is subjected to denoising processing using a Butterworth filter to reduce noise interference; Standardize the denoised data to improve the speed of network model solution; The standardized data is divided into small segments using a sliding window method, and the data set is divided into 70% as training samples and 30% as test samples.

6. The cough recognition method based on multi-channel Conformer according to claim 5, characterized in that: The fixed length of the sliding window is 64, and the time signal in the sliding window is sampled every 3.2 seconds. There is a 50% overlap rate between two adjacent sliding windows to prevent leakage of valid data information and obtain more feature information.

Images