Breathing sound classification method and device, storage medium and computer program product

Abstract

The invention discloses a breath sound classification method and device, a storage medium and a computer program product, and relates to the technical field of acoustic signal processing. The breath sound classification method is applied to the head-mounted equipment, and comprises the following steps: acquiring a bone conduction audio stream collected by a bone conduction sensor in the head-mounted equipment, and extracting frequency domain features of the bone conduction audio stream; a breathing sound classification model obtained through pre-training is adopted to process the frequency domain features, a breathing sound classification result is obtained, the breathing sound classification result is at least used for representing whether the bone conduction audio stream contains breathing sound or not, and the breathing sound classification model comprises a backbone network and a classifier; the backbone network comprises at least two layers of time sequence convolutional network structures which are connected in sequence, and each layer of time sequence convolutional network structure comprises a time sequence one-dimensional convolutional network module and a time sequence two-dimensional convolutional network module which are connected in sequence. According to the method, a set of complete bone conduction breath sound analysis process is realized, and the breath sound classification accuracy in a complex environment is improved.

Description

Technical Field

[0001] This application relates to the field of acoustic signal processing technology, and in particular to a method for classifying respiratory sounds, a head-mounted device, a storage medium, and a computer program product. Background Technology

[0002] There is a direct and close correlation between physiological health status and respiratory sound characteristics; abnormal changes in respiratory sounds are often important indicators of respiratory diseases or physiological dysfunction. Therefore, accurate and reliable respiratory sound detection is of great significance for health monitoring and disease early warning. Currently, respiratory sound detection technology based on air conduction microphones faces significant challenges. Although air conduction microphones can record audio data across the entire frequency band, their resistance to environmental interference is poor, and they are easily affected by ambient noise in practical applications. Especially in everyday life scenarios, environmental noise, other people's voices, and other interference factors can significantly reduce the accuracy and reliability of respiratory sound detection. In contrast, bone conduction sensors collect sound signals by detecting bone vibrations, which can effectively weaken interference from non-wearers and ambient sounds, giving them a natural advantage in noise resistance. However, the audio data collected by bone conduction sensors differs significantly from that of air conduction microphones, mainly in terms of bandwidth limitation and signal distortion, making it difficult to directly apply traditional respiratory sound detection algorithms based on air conduction audio. Especially in wearable devices such as augmented reality devices, how to fully utilize the characteristics of bone conduction sensors, overcome their signal acquisition limitations, and achieve high-precision, high-robustness breath sound detection has become a key challenge that urgently needs to be overcome in current technological development. Existing technologies have not yet effectively solved the core problems of signal processing, feature extraction, and classification accuracy in bone conduction breath sound detection, and there is an urgent need to develop innovative solutions specifically for the characteristics of bone conduction audio.

[0003] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0004] The main purpose of this application is to provide a method for classifying breath sounds, a head-mounted device, a storage medium, and a computer program product. It aims to realize a complete bone conduction breath sound analysis process, improve the accuracy of breath sound classification in complex environments, and effectively solve the technical problems of signal processing difficulties and insufficient classification reliability in bone conduction breath sound detection.

[0005] To achieve the above objectives, this application proposes a breath sound classification method, which is applied to a head-mounted device. The breath sound classification method includes: The bone conduction audio stream collected by the bone conduction sensor in the head-mounted device is acquired, and the frequency domain features of the bone conduction audio stream are extracted. The frequency domain features are processed using a pre-trained respiratory sound classification model to obtain a respiratory sound classification result. The respiratory sound classification result is used to characterize whether the bone conduction audio stream contains respiratory sounds. The respiratory sound classification model includes a backbone network and a classifier. The backbone network includes at least two sequentially connected temporal convolutional network structures. Each temporal convolutional network structure includes a sequentially connected one-dimensional temporal convolutional network module and a two-dimensional temporal convolutional network module.

[0006] Optionally, the respiratory sound classification model further includes a two-dimensional convolutional module disposed before the backbone network and a temporal recurrent attention module disposed between the backbone network and the classifier; The step of processing the frequency domain features using a pre-trained breath sound classification model to obtain the breath sound classification result includes: The frequency domain features are processed using the two-dimensional convolution module to obtain a first feature map; The first feature map is processed using the backbone network to obtain the second feature map; The second feature map is processed using the temporal cyclic attention module to obtain the third feature map; The classifier is used to classify the breath sounds based on the third feature map to obtain the classification result.

[0007] Optionally, the temporal one-dimensional convolutional network module includes a first size conversion unit, a dilated one-dimensional convolutional layer, a batch normalization layer, a one-dimensional convolutional layer, a normalization activation module, a one-dimensional convolutional layer, a normalization activation module, and a second size conversion unit connected in sequence. The first size conversion unit is used to merge the channel dimension and feature dimension in the input feature map, and the second size conversion unit is used to split the feature dimension in the input feature map into the channel dimension and feature dimension. The dilation rate of the dilated one-dimensional convolutional layer increases exponentially with the number of layers in the temporal convolutional network structure.

[0008] Optionally, a residual connection is provided between the input and output of the one-dimensional temporal convolutional network module, a residual connection is provided between the input and output of the two-dimensional temporal convolutional network module, and a residual connection is provided between the output of the one-dimensional temporal convolutional network module and the output of the two-dimensional temporal convolutional network module in the same layer of the temporal convolutional network structure.

[0009] Optionally, the temporal two-dimensional convolutional network module includes a dilated two-dimensional convolutional layer, a batch normalization layer, a two-dimensional convolutional layer, a normalization activation module, a two-dimensional convolutional layer, and a normalization activation module connected in sequence. The dilation rate of the dilated two-dimensional convolutional layer increases exponentially with the number of layers in the temporal convolutional network structure.

[0010] Optionally, the temporal recurrent attention module includes a gated recurrent unit, a fully connected layer, and an activation function. The step of processing the second feature map using the temporal recurrent attention module to obtain the third feature map includes: The second feature map is subjected to squared average pooling in the frequency dimension to obtain the time energy representation; The temporal energy representation is processed sequentially through the gated recurrent unit, the fully connected layer, and the activation function to obtain a one-dimensional attention mask; The one-dimensional attention mask is copied along the frequency dimension to obtain a two-dimensional attention mask; The second feature map is weighted using the two-dimensional attention mask to obtain the third feature map.

[0011] Optionally, the breath sound classification result is also used to characterize whether the breath sounds are abnormal when the bone conduction audio stream contains breath sounds; the breath sound classification method further includes: Acquire the collected training data, wherein the training data includes multiple first-type audio streams, multiple second-type audio streams, multiple third-type audio streams, and noise data, wherein the first-type audio stream is an audio stream containing pure normal breathing sounds collected by a bone conduction sensor, the second-type audio stream is an audio stream containing pure abnormal breathing sounds collected by a bone conduction sensor, and the third-type audio stream is an audio stream that does not contain breathing sounds collected by a bone conduction sensor; Each of the first type of audio streams is mixed with the noise data to obtain each of the first type of training samples; each of the second type of audio streams is mixed with the noise data to obtain each of the second type of training samples; and each of the third type of audio streams is mixed with the noise data to obtain each of the third type of training samples. The respiratory sound classification model is obtained by training each of the first type of training samples, each of the second type of training samples, and each of the third type of training samples.

[0012] In addition, to achieve the above objectives, this application also proposes a head-mounted device, which further includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the respiratory sound classification method described above.

[0013] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and which, when executed by a processor, implements the steps of the breath sound classification method described above.

[0014] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the respiratory sound classification method described above.

[0015] One or more technical solutions proposed in this application have at least the following technical effects: By acquiring the bone conduction audio stream collected by the bone conduction sensor in the head-mounted device and extracting its frequency domain features, this paper fully utilizes the inherent anti-environmental noise interference characteristics of the bone conduction sensor to convert the bone vibration signal into a frequency domain representation suitable for analysis, effectively overcoming the shortcomings of traditional air conduction microphones in detecting breath sounds in noisy environments. The frequency domain features are processed using a pre-trained breath sound classification model, leveraging the hierarchical feature extraction capabilities of the multi-layer temporal convolutional network structure in the backbone network. The one-dimensional temporal convolutional network module specifically captures the continuous features of breath sounds along the time axis, while the two-dimensional temporal convolutional network module jointly analyzes the time-frequency distribution pattern. This dual-module collaborative architecture can effectively learn the unique signal features of bone conduction breath sounds, overcoming the problem of poor adaptability to bone conduction signals in traditional methods. Therefore, this application implements a complete bone conduction breath sound analysis workflow. By extracting discriminative features from the interference-resistant bone conduction signal and performing deep pattern recognition, it significantly improves the accuracy of breath sound classification in complex environments, effectively solving the technical challenges of signal processing difficulties and insufficient classification reliability in bone conduction breath sound detection. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the first embodiment of the breath sound classification method of this application; Figure 2 This is a schematic diagram of a respiratory classification model involved in an embodiment of the respiratory sound classification method of this application; Figure 3 This is a schematic diagram of a temporal one-dimensional convolutional network module involved in an embodiment of the respiratory sound classification method of this application; Figure 4 This is a schematic diagram of the backbone network involved in an embodiment of the respiratory sound classification method of this application; Figure 5 This application provides an embodiment of a respiratory sound classification method that involves a temporal two-dimensional convolutional network module. Figure 6 This application provides an embodiment of the respiratory sound classification method involving a temporal loop attention module; Figure 7 This is a schematic diagram of the device structure of the hardware operating environment involved in the respiratory sound classification method in the embodiments of this application.

[0019] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0020] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0021] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0022] Existing technologies have not yet effectively solved the core problems of signal processing, feature extraction, and classification accuracy in bone conduction breath sound detection, and there is an urgent need to develop innovative solutions specifically for the characteristics of bone conduction audio.

[0023] This application provides a solution that acquires bone conduction audio streams collected by bone conduction sensors in a head-mounted device and extracts their frequency domain features. It fully utilizes the inherent anti-environmental noise interference characteristics of bone conduction sensors, converting bone vibration signals into a frequency domain representation suitable for analysis. This effectively overcomes the shortcomings of traditional air conduction microphones in detecting breath sounds in noisy environments. By employing a pre-trained breath sound classification model to process the frequency domain features, and utilizing the hierarchical feature extraction capabilities of the multi-layer temporal convolutional network structure in the backbone network, the application achieves this. The one-dimensional temporal convolutional network module specifically captures the continuous features of breath sounds along the time axis, while the two-dimensional temporal convolutional network module jointly analyzes the time-frequency distribution pattern. This dual-module collaborative architecture effectively learns the unique signal characteristics of bone conduction breath sounds, overcoming the poor adaptability of traditional methods to bone conduction signals. Therefore, this application implements a complete bone conduction breath sound analysis workflow. By extracting discriminative features from interference-resistant bone conduction signals and performing deep pattern recognition, it significantly improves the accuracy of breath sound classification in complex environments, effectively solving the technical challenges of signal processing difficulties and insufficient classification reliability in bone conduction breath sound detection.

[0024] The following presents a first embodiment of the respiratory sound classification method of this application. In this embodiment, the respiratory sound classification method is implemented by a head-mounted device. It is understood that there are many types of head-mounted devices, such as smart glasses, VR (Virtual Reality) / AR (Augmented Reality) devices, etc. Each type of head-mounted device has various hardware architectures and software system implementations. The embodiments of this application do not limit the type of head-mounted device, hardware architecture, or software system implementation of the respiratory sound classification method. A bone conduction sensor is installed in the head-mounted device. A bone conduction sensor is a device that collects sound signals by detecting bone vibrations. It does not rely on air conduction and can directly capture vibration signals from the skull or facial bones, thereby reducing interference from environmental noise. The location of the bone conduction sensor in the head-mounted device is not limited in this embodiment; it can be placed in a location conducive to collecting respiratory sounds, such as the nose pad or the inner side of the temples of the head-mounted device, so as to closely contact the user's facial bone area. In this embodiment, refer to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the breath sound classification method of this application. In this embodiment, the breath sound classification method includes steps S10 to S20: Step S10: Obtain the bone conduction audio stream collected by the bone conduction sensor in the head-mounted device, and extract the frequency domain features of the bone conduction audio stream.

[0025] Bone conduction audio stream refers to a sequence of audio signals continuously acquired by bone conduction sensors and transmitted through bone vibration. It contains vibration information generated during the user's breathing process, and this information is represented in the form of a time series.

[0026] Extracting the frequency domain features of bone conduction audio streams can be achieved by converting the time-domain bone conduction audio stream into a frequency domain representation using Short Time Fourier Transform (STFT). Since the audio stream is a single-channel signal and only amplitude information is used, the frequency domain features can be represented as a (B×1×T×F') dimensional vector, where B represents the batch number, 1 indicates that the number of channels is 1, T represents the number of time frames, and F' represents the number of frequency bins. It should be noted that during the training phase of the breath sound classification model, B is generally greater than 1; however, during the inference phase, B can be set to 1, meaning that a single audio stream is processed in real time.

[0027] Step S20: The frequency domain features are processed using a pre-trained respiratory sound classification model to obtain a respiratory sound classification result. The respiratory sound classification result is used to characterize whether the bone conduction audio stream contains respiratory sounds. The respiratory sound classification model includes a backbone network and a classifier. The backbone network includes at least two sequentially connected temporal convolutional network structures. Each temporal convolutional network structure includes a sequentially connected one-dimensional temporal convolutional network module and a two-dimensional temporal convolutional network module.

[0028] The breath sound classification result can be a binary result, for example, 1 indicates that the bone conduction audio stream contains breath sounds, and 0 indicates that the bone conduction audio stream does not contain breath sounds; or, the breath sound classification result can also be a tri-class result, for example, 0 indicates that the bone conduction audio stream does not contain breath sounds, 1 indicates that the bone conduction audio stream contains normal breath sounds, and 2 indicates that the bone conduction audio stream contains abnormal breath sounds.

[0029] The temporal 1d-convolutional network (TCN1D) module is a convolutional neural network specifically designed for processing time-series data. It captures temporal dependencies in the sequence, such as the rhythm and phase changes of breath sounds, by sliding a 1D convolutional kernel along the time dimension. The temporal 2d-convolutional network (TCN2D) module processes both time and frequency dimensions simultaneously. It operates on the time-frequency feature map using 2D convolutional kernels to jointly model local patterns of time and frequency, such as the spectral features of respiratory abnormalities. These two modules are connected sequentially. First, the 1D TCN1D is used to extract temporal features, and then the 2D TCN2D is used to extract joint time-frequency features, thereby gradually abstracting and enhancing the representation of breath sounds.

[0030] The backbone network consists of at least two (K layers, K > 2) temporal convolutional network structures. Each layer contains a sequentially connected one-dimensional TCN1D and a two-dimensional TCN2D. This design addresses the characteristics of bone conduction breath sound signals, such as long duration and slow pattern changes. It can construct multi-scale receptive fields through multi-layer stacking. The shallow network captures short-term respiratory rhythms, while the deep network models the complete respiratory cycle, thus achieving accurate modeling of the complete life cycle of breath sounds. The one-dimensional TCN1D module specifically models continuous features on the time axis, capturing key temporal patterns such as respiratory rhythm and respiratory phase transitions through one-dimensional temporal convolution, adapting to the periodic characteristics of breath sounds. The two-dimensional TCN2D module jointly analyzes the time-frequency distribution to identify specific abnormal respiratory patterns. This hierarchical processing fully utilizes the multi-scale characteristics of bone conduction signals, improving classification accuracy.

[0031] The classifier may include a fully connected layer that receives feature maps extracted by the backbone network and outputs the breath sound classification result. After the fully connected layer, for binary classification, the classifier can use the sigmoid activation function to output a probability value, and for tri-class classification, it can use softmax to output the probability distribution of the three classes, thereby mapping high-level features to specific class labels.

[0032] In one feasible implementation, processing the frequency domain features using a breath sound classification model to obtain a breath sound classification result may include: inputting the frequency domain features into a backbone network for feature extraction to obtain a feature map, and inputting the feature map into a classifier for classification to obtain a breath sound classification result.

[0033] In this embodiment, by acquiring the bone conduction audio stream collected by the bone conduction sensor in the head-mounted device and extracting its frequency domain features, the inherent anti-environmental noise interference characteristics of the bone conduction sensor are fully utilized. The bone vibration signal is converted into a frequency domain representation suitable for analysis, effectively overcoming the shortcomings of traditional air conduction microphones in detecting breath sounds in noisy environments. A pre-trained breath sound classification model is used to process the frequency domain features, leveraging the hierarchical feature extraction capabilities of the multi-layer temporal convolutional network structure in the backbone network. The one-dimensional temporal convolutional network module specifically captures the continuous features of breath sounds along the time axis, while the two-dimensional temporal convolutional network module jointly analyzes the time-frequency distribution pattern. This dual-module collaborative architecture can effectively learn the unique signal features of bone conduction breath sounds, overcoming the problem of poor adaptability to bone conduction signals in traditional methods. Therefore, this embodiment implements a complete bone conduction breath sound analysis workflow. By extracting discriminative features from the interference-resistant bone conduction signal and performing deep pattern recognition, the accuracy of breath sound classification in complex environments is significantly improved, effectively solving the technical challenges of signal processing difficulties and insufficient classification reliability in bone conduction breath sound detection.

[0034] Based on the first embodiment described above, a second embodiment of the breath sound classification method of this application is proposed. In this embodiment, content that is the same as or similar to that in the first embodiment can be referred to the above description and will not be repeated hereafter. In this embodiment, as... Figure 2As shown, the respiratory sound classification model further includes a two-dimensional convolutional module placed before the backbone network and a temporal recurrent attention module placed between the backbone network and the classifier. The two-dimensional convolutional module takes frequency domain features (B×1×T×F') as input and outputs (B×C×T×F). Its function is to address the relatively flat spectral characteristics and low signal-to-noise ratio of bone conduction audio by performing local feature enhancement in the time-frequency domain through two-dimensional convolution, effectively extracting weak respiratory sound features masked by environmental noise, thus laying the foundation for subsequent deep feature extraction. The temporal recurrent attention module has an input size of (B×C×T×F) and an output size of (B×C×T×F). Its function is to address the sparse distribution of respiratory sound events in bone conduction audio by dynamically focusing on key time segments containing effective respiratory sounds through an attention mechanism, suppressing interference noise such as muscle movement and swallowing, and significantly improving classification robustness in complex bone conduction environments.

[0035] In one feasible embodiment, step S20 includes S201 to S204: Step S201: The frequency domain features are processed using the two-dimensional convolution module to obtain the first feature map.

[0036] Step S202: The first feature map is processed using the backbone network to obtain the second feature map.

[0037] Step S203: The second feature map is processed using the temporal cyclic attention module to obtain the third feature map.

[0038] Step S204: The classifier is used to classify the breath sounds based on the third feature map to obtain the breath sound classification result.

[0039] In this embodiment, preliminary feature enhancement is performed through a two-dimensional convolutional module to extract key time-frequency patterns; deep feature extraction is performed through a backbone network to capture multi-scale respiratory sound features; feature recalibration is performed through a temporal recurrent attention module to focus on important information; and final classification is achieved through a classifier. Each step progressively refines the features, the two-dimensional convolutional module improves feature quality, the backbone network constructs hierarchical representations, the attention mechanism suppresses noise, and the classifier ensures accurate decision-making. Overall, these steps form an end-to-end processing flow, significantly improving the accuracy and robustness of respiratory sound classification.

[0040] In one feasible implementation, such as Figure 3The temporal one-dimensional convolutional network module includes a first size conversion unit, a dilated one-dimensional convolutional layer, a batch normalization layer, a one-dimensional convolutional layer, a normalization activation module, a one-dimensional convolutional layer, a normalization activation module, and a second size conversion unit connected in sequence. The first size conversion unit is used to merge the channel dimension and feature dimension in the input feature map, and the second size conversion unit is used to split the feature dimension in the input feature map into the channel dimension and feature dimension. The dilation rate of the dilated one-dimensional convolutional layer increases exponentially with the number of layers in the temporal convolutional network structure.

[0041] The first size transformation unit has an input size of (B×C×T×F) and an output size of (B×(C*F)×T). Its function is to merge the channel dimension C and the feature dimension F into a new feature dimension C*F by dimensional transformation (achieved through the Permute and Reshape operations), converting the four-dimensional feature map into a three-dimensional tensor. This allows subsequent one-dimensional convolution operations to effectively process the comprehensive feature vector containing multi-channel and frequency information in each time frame on the time dimension T, thereby better capturing the continuous pattern of breathing sounds on the time axis.

[0042] A dilated one-dimensional convolutional layer is a one-dimensional convolutional operation that introduces intervals in the time dimension. Its input size is B×(C*F)×T, and its output size is (B×C'×T), where C' is the number of output channels. This layer inserts intervals between convolutional kernel elements by setting a dilation rate greater than 1, thereby expanding the receptive field in the time dimension without increasing the number of parameters. This allows it to capture a longer range of temporal dependencies, such as changes in breathing rhythm and patterns of complete breathing cycles.

[0043] The batch normalization layer (BN) is used to normalize the feature distribution, stabilize the data distribution of bone conduction features during training, alleviate feature fluctuations caused by changes in sensor contact pressure or individual differences, and improve the training stability and convergence speed of the model.

[0044] The two one-dimensional convolutional layers in the temporal one-dimensional convolutional network module are both one-dimensional convolutions with a kernel size and stride of 1. The two normalized activation modules each include a BN layer and a ReLU activation function. This structure of stacking two 1×1 one-dimensional convolutions with normalization and activation can achieve information interaction and dimensional transformation between feature channels without changing the size of the time dimension. This completes the refinement and dimensional adaptation of bone conduction features, and enhances the model's ability to express key features of respiratory sounds and its nonlinear modeling ability.

[0045] The second size transformation unit has an input size of (B×(C*F)×T) and an output size of (B×C×T×F). Its function is to re-split the merged feature dimension C*F into independent channel dimension C and feature dimension F through inverse dimension transformation (achieved through Permute and Reshape operations), restore the three-dimensional tensor to a four-dimensional feature map, ensure that the output features match the input requirements of subsequent temporal two-dimensional convolutional network modules, and maintain the dimensional consistency of the entire network data flow.

[0046] In one feasible implementation, such as Figure 3 As shown, residual connections are provided between the input and output of the temporal one-dimensional convolutional network module. Figure 5 As shown, residual connections are provided between the input and output of the temporal two-dimensional convolutional network module. Figure 4 As shown, a residual connection is provided between the output of the one-dimensional temporal convolutional network module and the output of the two-dimensional temporal convolutional network module in the same layer of the temporal convolutional network structure.

[0047] Setting residual connections between the inputs and outputs of one-dimensional temporal convolutional network modules ensures the complete transmission of respiratory sound temporal features during module processing. Cross-layer connections preserve the original temporal information, preventing the loss of subtle but crucial respiratory phase information in deep networks during multiple transformations, thus improving the model's ability to retain respiratory temporal details. Setting residual connections between the inputs and outputs of two-dimensional temporal convolutional network modules helps maintain the richness of bone conduction spectral features. Identity mapping preserves the original time-frequency distribution features, preventing excessive smoothing or distortion of time-frequency details in the multi-layer nonlinear transformations of deep networks, ensuring the integrity of spectral features. Setting residual connections between the outputs of one-dimensional and two-dimensional temporal convolutional network modules within the same layer of the temporal convolutional network structure allows for the construction of deep network structures suitable for bone conduction audio analysis. Cross-module information fusion enhances feature diversity while ensuring effective backpropagation of gradients even with limited training data, preventing gradient vanishing or exploding, and improving the stability and convergence of model training.

[0048] In one feasible implementation, such as Figure 5 As shown, the temporal two-dimensional convolutional network module includes a dilated two-dimensional convolutional layer, a batch normalization layer, a two-dimensional convolutional layer, a normalization activation module, a two-dimensional convolutional layer, and a normalization activation module connected in sequence. The dilation rate of the dilated two-dimensional convolutional layer increases exponentially with the number of layers in the temporal convolutional network structure.

[0049] A dilated two-dimensional convolutional layer is a convolutional operation that introduces intervals in a two-dimensional time-frequency space. Its input size is (B×C×T×F) and its output size is (B×C'×T×F), where C' is the number of output channels. This layer expands the receptive field in the time-frequency domain by inserting intervals along the time and frequency dimensions between the convolutional kernel elements while keeping the number of parameters constant. This allows it to simultaneously capture local details and global trend patterns in the bone conduction spectrum, enabling multi-scale analysis of the time-frequency distribution of breath sounds.

[0050] Batch normalization (BN) layers are used to normalize the feature maps output by two-dimensional convolution, thereby stabilizing the distribution of bone conduction spectrum features, reducing internal covariate bias, improving the model's adaptability to differences in bone conduction characteristics among individuals and changes in acquisition conditions of various devices, and enhancing the model's generalization ability.

[0051] The two 2D convolutional layers in the temporal 2D convolutional network module are both 2D convolutions with kernel_size and stride of 1. The two normalized activation modules each include a BN layer and a ReLU activation function. This stacked 1×1 convolutional structure can achieve information recalibration and nonlinear transformation between feature channels while keeping the spatiotemporal dimension of the feature map unchanged. It optimizes the information flow of bone conduction features in the channel dimension, enhances the model's ability to extract discriminative features of breath sounds, and improves the quality of feature representation.

[0052] The specific method for the exponential increase in the hole ratio with the number of layers is as follows: for a dilated convolutional layer in the k-th layer of the temporal convolutional network structure, its hole ratio is set to 2^(k-1), where k is the layer number starting from 1, and K is the total number of layers in the temporal convolutional network structure. This hole ratio setting allows shallow networks to use a smaller hole ratio (e.g., the hole ratio of the first layer is 1, i.e., standard convolution), focusing on capturing local details and short-term patterns of breath sounds, such as the transition features between the inspiratory and expiratory phases; deep networks use a larger hole ratio (e.g., the hole ratio of the third layer is 4), which can cover a wider time-frequency region and model the complete respiratory cycle regularity and long-term dependence. This design is specifically designed for the multi-timescale characteristics of bone conduction breath sounds. By constructing a multi-scale feature extraction pyramid through an exponentially increasing hole ratio, it perfectly adapts to the characteristics of bone conduction breath sound signals, which contain both subtle instantaneous events and long-term periodic patterns, significantly improving the model's ability to model the complete life cycle of breath sounds.

[0053] In one feasible implementation, such as Figure 6 As shown, the temporal recurrent attention module includes a gated recurrent unit, a fully connected layer, and an activation function. Step S203 includes steps S2031 to S2034: Step S2031: Perform squared average pooling on the second feature map in the frequency dimension to obtain the time energy representation.

[0054] The temporal energy representation is obtained by calculating the average of the squared values ​​of each frequency point in the frequency dimension of the second feature map. Specifically, the calculation method is as follows: for each batch b, channel c, and time point t... Where f ranges from 0 to F-1, the resulting time energy representation Z has a size of (B×C×T). This operation can extract the time energy profile of bone conduction breath sounds, reflecting the trend of breathing intensity over time, providing a reference signal that conforms to the physiological characteristics of breathing for subsequent attention mechanisms, and enabling attention weight calculation to be based on the physical energy characteristics of breath sounds.

[0055] In step S2032, the time energy representation is processed sequentially through the gated recurrent unit, the fully connected layer, and the activation function to obtain a one-dimensional attention mask.

[0056] The size of the one-dimensional attention mask is (B×C×T). A gated recurrent unit is used to perform temporal modeling of the temporal energy representation, learning the distribution patterns and dependencies of respiratory sound events in the temporal dimension. Then, a fully connected layer transforms the learned temporal patterns into attention weights. Finally, a sigmoid activation function is used to normalize the weights to between 0 and 1, thus obtaining the one-dimensional attention mask. This process enables adaptive attention to respiratory phases such as inspiration and expiration, allowing the model to dynamically adjust the importance assessment of different time points based on respiratory physiological characteristics.

[0057] Step S2033: Copy the one-dimensional attention mask along the frequency dimension to obtain a two-dimensional attention mask.

[0058] By copying and extending the one-dimensional attention mask in the frequency dimension, specifically by repeating the one-dimensional attention mask of size (B×C×T) F times in the frequency dimension, the attention weights at each time point are applied to all frequency points, thus obtaining a two-dimensional attention mask of size B×C×T×F. This copying operation ensures that different frequency positions at the same time point in the time-frequency feature map share the same importance weights, maintaining the consistency of attention in the time dimension.

[0059] Step S2034: The second feature map is weighted using the two-dimensional attention mask to obtain the third feature map.

[0060] By performing an element-wise dot product operation between the two-dimensional attention mask A and the second feature map V, i.e., Y=V A, where Y is the third feature map. This means that the corresponding elements are multiplied, which can recalibrate the second feature map based on the learned attention weights, strengthen the features corresponding to key time segments containing effective breathing sounds, and suppress noise features caused by interference factors such as muscle movement and swallowing, thereby significantly improving the classification accuracy of breathing sounds in real bone conduction application scenarios.

[0061] Based on the first embodiment described above, a second embodiment of the breath sound classification method of this application is proposed. In this embodiment, content that is the same as or similar to that in the first embodiment can be referred to the above description and will not be repeated hereafter. In this embodiment, the breath sound classification result is also used to characterize whether the breath sounds are abnormal when the bone conduction audio stream contains breath sounds. The breath sound classification method further includes S30~S50: Step S30: Acquire the collected training data, wherein the training data includes multiple first-type audio streams, multiple second-type audio streams, multiple third-type audio streams, and noise data. The first-type audio stream is an audio stream containing pure normal breathing sounds collected by a bone conduction sensor, the second-type audio stream is an audio stream containing pure abnormal breathing sounds collected by a bone conduction sensor, and the third-type audio stream is an audio stream that does not contain breathing sounds collected by a bone conduction sensor.

[0062] A training dataset was constructed by systematically collecting bone conduction audio data from a large population. Normal bone conduction breath sounds were collected from at least 1000 speakers, randomly distributed according to factors such as gender and age, with at least 30 breath sounds (Type 1 audio stream) and 200 random speech sounds (Type 3 audio stream) collected from each speaker. Abnormal bone conduction breath sounds were collected from at least 1000 speakers, randomly distributed according to factors such as gender and age, with at least 30 breath sounds collected from each speaker (Type 2 audio stream). Simultaneously, at least 6000 audio streams without breath sounds or speech sounds from different real-world scenarios were collected (Type 3 audio stream). This large-scale, multi-scenario data collection method ensured the diversity, representativeness, and authenticity of the training samples, providing a sufficient data foundation for the model to learn to distinguish between normal, abnormal, and non-breath sounds.

[0063] Step S40: Mix each of the first type of audio streams with the noise data to obtain each of the first type of training samples; mix each of the second type of audio streams with the noise data to obtain each of the second type of training samples; and mix each of the third type of audio streams with the noise data to obtain each of the third type of training samples.

[0064] Data augmentation is performed by dynamically and randomly mixing clean audio samples with real-world environmental noise. Specifically, this involves dynamically and randomly selecting normal bone conduction breath sounds (Category 1 audio stream) and randomly mixing them with noise data as Category 1 training data (corresponding to Category 1 training samples); dynamically and randomly selecting abnormal bone conduction breath sounds (Category 2 audio stream) and randomly mixing them with noise data as Category 2 training data (corresponding to Category 2 training samples); and dynamically and randomly selecting non-breath sounds in bone conduction (Category 3 audio stream) and randomly mixing them with noise data as Category 0 training data (corresponding to Category 3 training samples). This data augmentation method effectively simulates the various complex acoustic environments encountered by head-mounted devices in actual use, allowing the training samples to cover more noise types and signal-to-noise ratio conditions, significantly improving the diversity and realism of the training samples.

[0065] Step S50: The respiratory sound classification model is trained based on each of the first type of training samples, each of the second type of training samples, and each of the third type of training samples.

[0066] An end-to-end training model for respiratory sound classification was developed using augmented training samples of three classes. By optimizing the classification loss function, the model learned to accurately distinguish normal respiratory sounds, abnormal respiratory sounds, and non-respiratory sounds from noisy bone conduction audio streams. During training, the model calculated prediction results through forward propagation and adjusted network parameters through backpropagation, gradually improving its ability to distinguish between the three classes of samples. This training strategy based on large-scale real-world simulated data, combined with a dynamic randomized data augmentation method, enables the model to adapt to various real-world usage environments, significantly improving the accuracy and robustness of respiratory sound classification under complex acoustic conditions, particularly enhancing its ability to detect abnormal respiratory sounds and its adaptability to real-world user scenarios.

[0067] This application provides a head-mounted device. The head-mounted device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the breath sound classification method described in the above embodiments.

[0068] The following is for reference. Figure 7 It shows a schematic diagram of a head-mounted device suitable for implementing the embodiments of this application. Figure 7 The head-mounted device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of this application.

[0069] like Figure 7As shown, the head-mounted device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the head-mounted device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the head-mounted device to communicate wirelessly or wiredly with other devices to exchange data. While the figures show head-mounted devices with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0070] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0071] Compared with the prior art, the beneficial effects of the head-mounted device provided in this application embodiment are the same as the beneficial effects of the respiratory sound classification method provided in the above embodiment, and other technical features of the head-mounted device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0072] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0073] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the respiratory sound classification method in the above embodiments.

[0074] The computer-readable storage medium provided in this application embodiment may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0075] The aforementioned computer-readable storage medium may be included in the head-mounted device or may exist independently and not assembled into the head-mounted device.

[0076] The aforementioned computer-readable storage medium carries one or more programs that, when executed by a head-mounted device, cause the head-mounted device to perform the functions defined in the methods of the embodiments disclosed in this application.

[0077] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0078] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0079] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0080] The readable storage medium provided in this application embodiment is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-described breath sound classification method. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application embodiment are the same as the beneficial effects of the breath sound classification method provided in the above-described embodiments, and will not be repeated here.

[0081] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the respiratory sound classification method described above.

[0082] Compared with the prior art, the beneficial effects of the computer program product provided in this application embodiment are the same as the beneficial effects of the respiratory sound classification method provided in the above embodiment, and will not be repeated here.

[0083] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A method for classifying breath sounds, characterized in that, The breath sound classification method is applied to a head-mounted device, and the breath sound classification method includes: The bone conduction audio stream collected by the bone conduction sensor in the head-mounted device is acquired, and the frequency domain features of the bone conduction audio stream are extracted. The frequency domain features are processed using a pre-trained respiratory sound classification model to obtain a respiratory sound classification result. The respiratory sound classification result is used to characterize whether the bone conduction audio stream contains respiratory sounds. The respiratory sound classification model includes a backbone network and a classifier. The backbone network includes at least two sequentially connected temporal convolutional network structures. Each temporal convolutional network structure includes a sequentially connected one-dimensional temporal convolutional network module and a two-dimensional temporal convolutional network module.

2. The breath sound classification method as described in claim 1, characterized in that, The respiratory sound classification model further includes a two-dimensional convolutional module placed before the backbone network and a temporal recurrent attention module placed between the backbone network and the classifier; The step of processing the frequency domain features using a pre-trained breath sound classification model to obtain the breath sound classification result includes: The frequency domain features are processed using the two-dimensional convolution module to obtain a first feature map; The first feature map is processed using the backbone network to obtain the second feature map; The second feature map is processed using the temporal cyclic attention module to obtain the third feature map; The classifier is used to classify the breath sounds based on the third feature map to obtain the classification result.

3. The breath sound classification method as described in claim 2, characterized in that, The temporal one-dimensional convolutional network module includes a first size conversion unit, a dilated one-dimensional convolutional layer, a batch normalization layer, a one-dimensional convolutional layer, a normalization activation module, a one-dimensional convolutional layer, a normalization activation module, and a second size conversion unit connected in sequence. The first size conversion unit is used to merge the channel dimension and feature dimension in the input feature map, and the second size conversion unit is used to split the feature dimension in the input feature map into the channel dimension and feature dimension. The dilation rate of the dilated one-dimensional convolutional layer increases exponentially with the number of layers in the temporal convolutional network structure.

4. The breath sound classification method as described in claim 3, characterized in that, A residual connection is provided between the input and output of the one-dimensional temporal convolutional network module, a residual connection is provided between the input and output of the two-dimensional temporal convolutional network module, and a residual connection is provided between the output of the one-dimensional temporal convolutional network module and the output of the two-dimensional temporal convolutional network module in the same layer of the temporal convolutional network structure.

5. The breath sound classification method as described in claim 2, characterized in that, The temporal two-dimensional convolutional network module includes a dilated two-dimensional convolutional layer, a batch normalization layer, a two-dimensional convolutional layer, a normalization activation module, a two-dimensional convolutional layer, and a normalization activation module connected in sequence. The dilation rate of the dilated two-dimensional convolutional layer increases exponentially with the number of layers in the temporal convolutional network structure.

6. The breath sound classification method as described in claim 2, characterized in that, The temporal recurrent attention module includes a gated recurrent unit, a fully connected layer, and an activation function. The step of processing the second feature map using the temporal recurrent attention module to obtain the third feature map includes: The second feature map is subjected to squared average pooling in the frequency dimension to obtain the time energy representation; The temporal energy representation is processed sequentially through the gated recurrent unit, the fully connected layer, and the activation function to obtain a one-dimensional attention mask; The one-dimensional attention mask is copied along the frequency dimension to obtain a two-dimensional attention mask; The second feature map is weighted using the two-dimensional attention mask to obtain the third feature map.

7. The breath sound classification method according to any one of claims 1 to 6, characterized in that, The breath sound classification result is also used to characterize whether the breath sounds are abnormal when the bone conduction audio stream contains breath sounds; the breath sound classification method further includes: Acquire the collected training data, wherein the training data includes multiple first-type audio streams, multiple second-type audio streams, multiple third-type audio streams, and noise data, wherein the first-type audio stream is an audio stream containing pure normal breathing sounds collected by a bone conduction sensor, the second-type audio stream is an audio stream containing pure abnormal breathing sounds collected by a bone conduction sensor, and the third-type audio stream is an audio stream that does not contain breathing sounds collected by a bone conduction sensor; Each of the first type of audio streams is mixed with the noise data to obtain each of the first type of training samples; each of the second type of audio streams is mixed with the noise data to obtain each of the second type of training samples; and each of the third type of audio streams is mixed with the noise data to obtain each of the third type of training samples. The respiratory sound classification model is obtained by training each of the first type of training samples, each of the second type of training samples, and each of the third type of training samples.

8. A head-mounted device, characterized in that, The head-mounted device further includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the breath sound classification method as described in any one of claims 1 to 7.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the respiratory sound classification method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the respiratory sound classification method as described in any one of claims 1 to 7.

Images