Cough sound recognition method based on pre-training model and graph neural network
By combining pre-trained models and graph neural networks in cough sound recognition, the graph data of audio clips is constructed, which solves the problem that traditional methods are difficult to deal with complex cough sound information, and achieves higher-precision cough sound detection.
Patent Information
- Application Number
- CN202510001150.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-02
- Publication Date
- 2025-04-25
AI Technical Summary
Traditional deep convolutional neural networks are difficult to effectively characterize the complex time and frequency information in cough sounds, resulting in insufficient accuracy in cough sound detection.
Using a method based on pre-trained model and graph neural network, audio features are extracted through pre-trained models, and graph data of audio clips is constructed using graph neural network, node feature matrix and adjacency matrix are established, feature fusion and relationship construction are carried out, thereby improving the accuracy of cough sound recognition.
Through the feature fusion and relationship construction of graph neural networks, complex information in cough sounds can be processed more effectively and the accuracy of cough sound detection can be improved.
Smart Images
Figure CN119889366A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of artificial intelligence-assisted diagnosis, and specifically relates to a cough sound recognition method based on a pre-trained model and a graph neural network. Background Art
[0002] With the continuous development of industrial technology, the technology of audio classification through deep learning has become more mature, and the use of body sound to detect diseases has become a current research hotspot. Researchers have found that the coughs of patients with different diseases have different potential characteristics, so using coughs to detect lung diseases is a feasible and rapid detection method.
[0003] However, existing studies have shown that there is correlation between different time periods and frequency bands of a patient's cough, and traditional deep convolutional neural networks are difficult to characterize their complex relationships. Graph neural networks can effectively process non-Euclidean data, establish connections between nodes in different frames, perform feature extraction, and flexibly process complex relationships between different frames of audio signals. Summary of the invention
[0004] In response to the problems described in the background technology, the present invention proposes a cough sound recognition method based on a pre-trained model and a graph neural network to achieve higher-precision cough sound detection.
[0005] To achieve the above object, the present invention adopts the following technical solutions:
[0006] Design a cough sound recognition method based on a pre-trained model and graph neural network, including the following steps:
[0007] S1, collect sound data to build a dataset containing cough and non-cough sounds;
[0008] S2, removes silence from the audio data, and then segments it into fixed-length segments to construct cough segment and non-cough segment data;
[0009] S3, divide each audio clip into frames according to a certain frame length and frame shift, and use the pre-trained model to extract the features of each frame of audio;
[0010] S4, using the features extracted by the pre-trained model to generate the node feature matrix and adjacency matrix of the graph data, construct the graph data, and divide it into a training set and a test set;
[0011] S5, dividing the cough and non-cough segment graph data into training and test sets according to a certain ratio, inputting the graph data into the graph neural network model, training the graph neural network model, and using the trained graph neural network model to recognize cough sounds.
[0012] Furthermore, a microphone is used in S1 to record a cough database of different scenarios, including coughs, breathing sounds and voices. These include breathing sounds (fast and slow), coughs (deep and shallow), continuously pronounced vowel phonemes, and voice audio counting at slow and fast rhythms. At the same time, non-coughing data of different scenarios are recorded, including common scenes in the ESC-50 dataset and the Urbansound8K dataset, such as natural sounds, animal sounds, urban noise, etc. In addition, the collected cough sound data and non-cough data are mixed in a certain proportion to synthesize new sound data, and they are classified as cough sound datasets.
[0013] Furthermore, in S2, the cough audio segment is silenced and then the audio data is segmented, all the audio data is divided into 1-second audio segments, and the cough audio segments and non-cough audio segments are sorted.
[0014] Furthermore, in S3, the audio segment is divided into frames according to a certain frame length and frame shift, and then a pre-trained model (such as VGGish, wav2vec, etc.) is used to output an embedded feature with a shape of [1, M] for each frame of audio data.
[0015] Furthermore, in S4, the N frames of audio formed by each audio segment are sequentially fed into the pre-trained model to generate N embedded features of shape [1, M]. The feature generated by each frame of audio is defined as a node, and the node feature matrix X∈R is generated by all nodes. N×M The cosine similarity is calculated for each pair of nodes generated by each audio, and the formula is:
[0016]
[0017] where u i ,u j Represents the node feature vector. If the similarity value is greater than the set threshold δ, it is considered that there is a relationship between the two nodes, and an edge is established between them. The connection relationship between all nodes is represented by the adjacency matrix. The adjacency matrix A∈R N×N is a symmetric matrix, where:
[0018]
[0019] The graph data G = (X, A) is jointly represented by the node feature matrix and the adjacency matrix, and finally the training set and test set are divided according to a certain ratio.
[0020] Furthermore, in S5, the graph neural network model may adopt various mainstream graph network models in the field of graph neural networks. For example, taking the classic graph convolutional network GCN as an example, the definition is as follows:
[0021]
[0022] Among them, H (l) is the node feature matrix of the l-th layer, and the initial value of H (0) = X, W (l) is the learnable parameter weight matrix of the l-th layer. is the matrix obtained by adding the node self-connection to the adjacency matrix A of the graph. is the identity matrix. The degree matrix D ∈ R N×N is a diagonal matrix, and its diagonal elements represent the degrees of each node in the graph: D = diag(d1, d2, d3…, d N ), and the degree of a node refers to the number of edges directly connected to the node. is the degree matrix after adding node self-connection. is the normalized matrix of the degree matrix, which is used to eliminate the influence of node degrees. The graph pooling method is Sort Aggregation Pooling (SortAggregation(K)), and the sorting index of each node is defined as:
[0023]
[0024] Among them, h i ∈ R M is the feature vector of the i-th node, and h ij is the j-th component of the i-th node. The result is a sorted score vector S = [s1, s2, s3...., s N ∈ R N (N is the number of nodes). According to the node scores S, the node feature matrix H is sorted in descending order to obtain the sorted feature matrix H' = sort(H, S), H' ∈ R N×M . Set the target node K. If N > K, the first K node features are intercepted to obtain the global representation H” of the graph. If N < K, the remaining positions are filled with zero vectors to obtain the global representation H” of the graph.
[0025]
[0026] The training task of the cough sound recognition model is a binary classification task. The optimization goal of model training is to minimize the loss function. The loss function of the model adopts the typical binary cross-entropy loss function, and its formula is as follows:
[0027]
[0028] Among them, y is the binary label 0 or 1, and p(y) is the probability that the output belongs to the y label. The model dynamically adjusts the parameters in the model by minimizing the loss function using backpropagation to improve the model performance.
[0029] Compared with the prior art, the present invention has the following beneficial effects:
[0030] For the coughing sound of patients, there is correlation between the information in different time periods and frequency segments, and traditional deep convolutional networks are difficult to represent their complex relationships. Graph neural networks can effectively process non-Euclidean data. By establishing connections between different nodes and performing feature fusion, they can flexibly handle various complex relationships, thereby improving performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] Figure 1 is a flow chart of the method of the present invention. DETAILED DESCRIPTION
[0032] The specific implementation modes of the present invention are described below in conjunction with the accompanying drawings and examples. However, the following examples are only used to illustrate the present invention in detail and do not limit the scope of the present invention in any way.
[0033] Example 1: Figure 1 As shown, the cough sound recognition method based on the pre-trained model and the graph neural network of the present invention comprises the following steps:
[0034] S1, collect sound data to build a dataset containing cough and non-cough sounds;
[0035] Cough sound data can be recorded in the mp3, wav and other formats that can be recognized by computers. We use the Coswara dataset as the cough sound data source for the cough detection algorithm. The audio categories include cough, breath sounds, and speech. These include breath sounds (fast and slow), cough sounds (deep and shallow), continuously pronounced vowel phonemes, and speech audio with slow and fast rhythms. The entire dataset provides about 9,000 cough segments, 9,000 breath segments, and 9,000 speech segments. At the same time, we use the ESC-50 dataset and the Urbansound8K dataset as the source of non-cough sound segment data for the cough detection algorithm.
[0036] S2, removes silence from the audio data, and then segments it into fixed-length segments to construct cough segment and non-cough segment data;
[0037] The audio data is silenced, and then all the audio data is divided into 1-second audio segments, and cough audio segments and non-cough audio segments are sorted.
[0038] S3, uses the pre-trained model to extract the features of each frame of audio;
[0039] The audio segment is divided into frames according to a certain frame length and frame shift, and each frame of audio data is input into a pre-trained model such as VGGish, and the output embedding feature is of shape [1,M].
[0040] S4, generating a feature node matrix with the features of each frame of the audio clip as nodes in the graph, constructing an adjacency matrix with the relationship between different frames, and generating audio clip graph data;
[0041] The N frames of audio formed by each audio segment are sequentially sent to the pre-trained model to generate N embedded features of shape [1, M]. The feature generated by each frame of audio is defined as a node, and the node feature matrix X∈R is generated by all nodes. N×M ,The cosine similarity is calculated for each pair of nodes generated by each audio, and the formula is:
[0042]
[0043] where u i ,u j Represents the node feature vector. If the similarity value is greater than the set threshold δ, it is considered that there is a relationship between the two nodes, and an edge is established between them. The connection relationship between all nodes is represented by the adjacency matrix. The adjacency matrix A∈R N×N is a symmetric matrix, where:
[0044]
[0045] The graph data G = (X, A) is jointly represented by the node feature matrix and the adjacency matrix.
[0046] S5, divide the cough and non-cough segment graph data into training and test sets according to a certain ratio, input the graph data into the graph neural network model, train the graph neural network model, and use the trained graph neural network model to recognize cough sounds.
[0047] The graph neural network model is GCN, which is defined as follows:
[0048]
[0049] Among them, H (l) is the feature matrix of the l-th layer node, and the initial value H (0) =X,W (l) is the learnable parameter weight matrix of the lth layer, The adjacency matrix A of the graph plus the matrix after node self-connection, is the identity matrix. The degree matrix D∈R N×N It is a diagonal matrix whose diagonal elements represent the degree of each node in the graph D = diag (d1, d2, d3…, d N ), the degree of a node refers to the number of edges directly connected to the node. Add the degree matrix of node self-connection. The normalized matrix of the degree matrix is used to eliminate the influence of node degree. The graph pooling method is SortAggregation(K), and the sorting metric for each node is defined as:
[0050]
[0051] where h i ∈R M is the feature vector of the i-th node, and h ij is the j-th component of the i-th node. The result is a sorted score vector S = [s1, s2, s3...., s N ∈R N (N is the number of nodes). According to the node scores S, the node feature matrix H is sorted in descending order to obtain the sorted feature matrix H' = sort(H, S), and H'∈R N×M . Set the target node K. If N > K, the first K node features are intercepted to obtain the global representation H” of the graph. If N < K, the remaining positions are filled with zero vectors to obtain the global representation H” of the graph
[0052]
[0053] Input the graph data into the graph neural network model and train the graph neural network model. The training task of the cough sound recognition model is a binary classification task. The optimization objective of model training is to minimize the loss function. The loss function of the model uses a typical binary cross-entropy loss function, and its formula is as follows:
[0054]
[0055] where y is the binary label 0 or 1, and p(y) is the probability that the output belongs to the y label. The model dynamically adjusts the parameters in the model by minimizing the loss function to improve the model performance.
[0056] Although the foregoing implementation method takes cough sound recognition as an example, the method of the present invention is not limited to detecting coughs from sound signals. Other different diseases and different types of cough sound recognition based on sound are still applicable, such as using cough sound recognition to identify dry coughs and wet coughs, and using cough sound recognition to identify different diseases.
[0057] The above has described the present invention in detail with reference to the drawings and embodiments; however, those skilled in the art can understand that without departing from the concept of the present invention, various specific parameters in the above embodiments can be changed, or equivalent replacements can be made to related components and structures, thereby forming multiple specific embodiments, all of which are within the common variation range of the present invention and will not be elaborated one by one.
Claims
1. A cough sound recognition method based on a pre-trained model and a graph neural network, characterized in that: The following steps are involved: S1, collect sound data to build a dataset containing cough and non-cough sounds; S2, removes silence from the audio data, and then segments it into fixed-length segments to construct cough segment and non-cough segment data; S3, divide each audio clip into frames according to a certain frame length and frame shift, and use the pre-trained model to extract the features of each frame of audio; S4, generating a node feature matrix with the features of each frame of the audio clip as nodes in the graph, constructing an adjacency matrix with the relationship between different frames, and generating audio clip graph data; S5, dividing the cough and non-cough segment graph data into training and test sets according to a certain ratio, inputting the graph data into the graph neural network model, training the graph neural network model, and using the trained graph neural network model to recognize cough sounds.
2. The cough sound recognition method based on a pre-trained model and a graph neural network according to claim 1, characterized in that: In step S1: the cough sound dataset uses a microphone to collect cough sound data from different scenes and different groups of people. The non-cough sound data uses a microphone to collect sound data from different scenes, such as the classic natural sound recognition dataset ESC-50 dataset and Urbansound8K different scene sounds, including natural sounds, animal sounds, urban noise, etc. In addition, the collected cough sound data and non-cough data are mixed in a certain proportion to synthesize new sound data, and are classified as a cough sound dataset.
3. The cough sound recognition method based on a pre-trained model and a graph neural network according to claim 1, characterized in that: In step S2: silence is removed from audio data of different lengths, and then segmented according to a fixed length of 1 second to construct cough audio segments and non-cough audio segments.
4. The cough sound recognition method based on a pre-trained model and a graph neural network according to claim 1, characterized in that: In step S3: the 1-second audio segment is divided into frames according to a certain frame length and frame shift, and a classic audio pre-training model is used to extract features for each audio frame to generate an embedded feature of shape [1, M]. The pre-training model includes but is not limited to classic pre-training models such as VGGish, wav2vec, etc.
5. The cough sound recognition method based on a pre-trained model and a graph neural network according to claim 1, characterized in that: In step S4: N frames of audio formed by each audio segment are sequentially fed into the pre-trained model to generate N embedded features of shape [1, M]. The feature generated by each frame of audio is defined as a node, and the node feature matrix X∈R is generated by all nodes. N×M ,The cosine similarity is calculated for each pair of nodes generated by each audio segment, and the formula is: where u i ,u j Represents the node feature vector. If the similarity value is greater than the set threshold δ, it is considered that there is a relationship between the two nodes, and an edge is established between them. The connection relationship between all nodes is represented by the adjacency matrix. The adjacency matrix A∈R N×N is a symmetric matrix, where: The graph data G = (X, A) is jointly represented by the node feature matrix and the adjacency matrix, and finally the training set and test set are divided according to a certain ratio.
6. The cough sound recognition method based on a pre-trained model and a graph neural network according to claim 1, characterized in that: In step S5: the graph neural network model can be any graph neural network model, including but not limited to the classic graph convolutional network (Graph Convolutional Network), graph attention network (Graph Attention Networks), etc.