Semi-automatic annotation method, system and device for respiratory sounds

Abstract

The present invention provides a semi-automatic respiratory sound labeling method, system, and device. The method comprises acquiring respiratory sound audio data; manually labeling a portion of the respiratory sound audio data into a plurality of respiratory sound cycles; calculating the division positions of the remaining respiratory sound cycles in the respiratory sound audio data based on the manually labeled respiratory sound cycles to obtain all respiratory sound cycles in the respiratory sound audio data; classifying each respiratory sound cycle by pathological characteristics using an automatic recognition model to obtain a confidence probability for the corresponding category; directly labeling the respiratory sound cycle as the corresponding category when the confidence probability of the classification is greater than a set threshold; otherwise, multiple independent manual judgments are performed; if all manual judgments are consistent, the respiratory sound cycle is directly labeled as the corresponding category; otherwise, the respiratory sound cycle is discarded. The present invention achieves automatic labeling of respiratory sound categories, saving the cost and time of manual labeling, with a short cycle time, high accuracy, and good economic benefits.

Description

Technical Field

[0001] The present invention relates to a physiological sound processing technology in the field of sound processing technology, and in particular to a method, system and device for semi-automatic annotation of respiratory sounds. Background Art

[0002] Breath sounds, also known as lung sounds, reflect the acoustic properties of the lung tissue, trachea, chest wall, and other transmission media. The frequency range of breath sounds and other physiological sounds is approximately 50-3000Hz, while the human ear is sensitive to frequencies between approximately 1000-2000Hz. Mechanical stethoscopes have poor low-frequency response, making it difficult to detect faint sounds during auscultation, potentially leading to misidentification of this information. Therefore, it is crucial to collect and quantitatively analyze breath sounds, extract their typical features, and establish appropriate models for their classification and identification to obtain accurate data.

[0003] Breathing sound data annotation is a crucial step in respiratory sound classification and recognition. Currently, respiratory sound data annotation is accomplished by first collecting a long segment of respiratory sound data and manually segmenting and annotating it. As the amount of collected respiratory sound data increases, manually segmenting and annotating respiratory sounds will be a huge waste of manpower and resources, making it impractical. Furthermore, due to the subjectivity and varying judgment levels of different annotators, the question of how to choose between the annotation results of multiple people becomes a critical issue.

[0004] After searching, we found:

[0005] The Chinese invention patent application with publication number CN111753883A, "A Method for Establishing a Physiological Sound Training Set and Test Set," calculates the accuracy, sensitivity, and specificity of each physiological sound in the various categories of abnormal physiological sounds and the automatic identification of normal physiological sounds as optimization parameters of the training model, and improves the training model according to the optimization parameters. The physiological sounds provided by the present invention include a variety of physiological sounds, namely heart sounds, respiratory sounds, cough sounds, bowel sounds, vascular sounds, and joint movement sounds, which basically cover all sounds produced by the human body. Through reasonable classification of data, manual labeling by professionals and analysis of the training model in the test set and training set, the optimization parameters are improved to achieve the efficacy of artificial intelligence automatic labeling. In this method, all respiratory sound cycles are manually divided, the manual operation workload is large, the work efficiency is low, the work cycle is long, the cost is high, and the reliability of the results is poor.

[0006] Therefore, the existing respiratory sound data annotation technology that uses a fully manual annotation method has problems such as long cycle, high cost and poor result reliability. Currently, no description or report of similar technology to the present invention has been found, and similar information at home and abroad has not yet been collected. Summary of the Invention

[0007] In view of the above-mentioned deficiencies in the prior art, the present invention provides a method, system and device for semi-automatic annotation of respiratory sounds.

[0008] According to one aspect of the present invention, a method for semi-automatic annotation of respiratory sounds is provided, comprising:

[0009] Obtaining respiratory sound audio data;

[0010] Dividing a portion of the respiratory sound audio data into a plurality of respiratory sound cycles by manual labeling;

[0011] Based on the manually marked respiratory sound cycles, calculating the division positions of the remaining respiratory sound cycles of the respiratory sound audio data to obtain all respiratory sound cycles of the respiratory sound audio data;

[0012] Utilizing an automatic recognition model, each respiratory sound cycle is classified according to its pathological characteristics to obtain a confidence probability for the corresponding category. When the confidence probability of the classification is greater than a set threshold, the respiratory sound cycle is directly marked as the corresponding category. Otherwise, multiple manual independent judgments are performed.

[0013] If all manual judgment results are consistent, the respiratory sound cycle is directly marked as the corresponding category; otherwise, the respiratory sound cycle is discarded, and the labeling of the respiratory sound audio data is completed.

[0014] Preferably, the acquiring of respiratory sound audio data comprises:

[0015] The respiratory sounds are collected using sound collection equipment to obtain respiratory sound audio data.

[0016] Preferably, the number of respiratory sound cycles divided by manual annotation is at least two, and there is no overlapping area between two adjacent respiratory sound cycles.

[0017] Preferably, the calculating of the division positions of the remaining respiratory sound cycles of the respiratory sound audio data based on the manually marked respiratory sound cycles includes:

[0018] Obtaining a cycle length and a cycle interval of the respiratory sound cycle according to the manually marked respiratory sound cycle;

[0019] An average value is taken for the cycle length and the cycle interval of the respiratory sound cycle, and the remaining respiratory sound cycle positions of the respiratory sound audio data are divided according to the average value.

[0020] Preferably, the automatic recognition model includes: a K-nearest neighbor model, a support vector machine model, a Gaussian mixture model and / or a deep neural network model.

[0021] Preferably, the method further comprises:

[0022] Manual correction is performed on all respiratory sound cycles of the acquired respiratory sound audio data.

[0023] According to another aspect of the present invention, a semi-automatic respiratory sound annotation system is provided, comprising:

[0024] A data acquisition module, which is used to acquire respiratory sound audio data;

[0025] a period manual division module, which divides a portion of the respiratory sound audio data into a number of respiratory sound periods by manual annotation;

[0026] an automatic period division module, which calculates the division positions of the remaining respiratory sound periods of the respiratory sound audio data based on the manually marked respiratory sound periods, and obtains all respiratory sound periods of the respiratory sound audio data;

[0027] A category classification module, which uses an automatic recognition model to classify each respiratory sound cycle into pathological features and obtain a confidence probability of the corresponding category;

[0028] The category labeling module presets a threshold. When the confidence probability of the classification is greater than the preset threshold, the respiratory sound cycle is directly marked as the corresponding category. Otherwise, multiple manual judgments are made independently. If all manual judgments are consistent, the respiratory sound cycle is directly marked as the corresponding category. Otherwise, the respiratory sound cycle is discarded, completing the labeling of the respiratory sound audio data.

[0029] According to a third aspect of the present invention, there is provided a semi-automatic respiratory sound annotation device, comprising a user terminal; wherein:

[0030] The user terminal includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor can be used to execute the method according to any one of claims 1 to 6, or to execute the system according to claim 7 when executing the program.

[0031] Preferably, the device further comprises a sound collecting device, and the sound collecting device is connected to the data acquisition module of the system.

[0032] Preferably, the device further comprises a display connected to the processor.

[0033] Due to the adoption of the above technical solution, the present invention has at least one of the following beneficial effects compared with the prior art:

[0034] The semi-automatic labeling method, system, and device for respiratory sounds provided by the present invention solve the problems of long cycle, high cost, and poor result reliability existing in the existing technology that relies on fully manual labeling. A semi-automatic labeling method that combines manual labeling and automatic labeling is adopted. While automatically labeling the pathological feature classification information of respiratory sounds, it ensures the accuracy of the labeled information, saves the cost and time of relying entirely on manual labeling, and has the advantages of short cycle, high accuracy, and good economic benefits.

[0035] The semi-automatic respiratory sound labeling method, system, and device provided by the present invention can accurately identify respiratory sounds and can be applied to auxiliary diagnosis of respiratory diseases.

[0036] The semi-automatic respiratory sound labeling method, system, and device provided by the present invention automatically divide the remaining cycles after manually dividing the respiratory sounds into several cycles, thereby reducing the workload of manual operation; by combining manual labeling and automatic labeling cycles, the work cycle is shortened, the work cost is reduced, and the work accuracy is improved.

[0037] The semi-automatic respiratory sound labeling method, system, and device provided by the present invention further improve the work accuracy by setting thresholds and adopting different labeling methods according to different situations. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0039] Figure 1 FIG. 4 is a flowchart of a semi-automatic respiratory sound annotation method according to an embodiment of the present invention.

[0040] Figure 2 4 is a flowchart of a semi-automatic respiratory sound annotation method in a preferred embodiment of the present invention.

[0041] Figure 3 Schematic diagram of the structure of a semi-automatic respiratory sound annotation system according to one embodiment of the present invention.

[0042] Figure 4 Schematic diagram of the structure of a semi-automatic respiratory sound annotation device in one embodiment of the present invention.

[0043] In the figure: 1 is the user terminal, 2 is the sound collection device, and 3 is the display. DETAILED DESCRIPTION

[0044] The following is a detailed description of an embodiment of the present invention. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process. It should be noted that those skilled in the art may make various modifications and improvements without departing from the scope of the present invention, and these modifications and improvements fall within the scope of protection of the present invention.

[0045] Figure 1 FIG. 4 is a flowchart of a semi-automatic respiratory sound annotation method according to an embodiment of the present invention.

[0046] like Figure 1 As shown, the semi-automatic breath sound annotation method provided in this embodiment may include the following steps:

[0047] S100, obtaining breathing sound audio data;

[0048] S200, dividing a portion of the respiratory sound audio data into a plurality of respiratory sound cycles by manual labeling;

[0049] S300, calculating the division positions of the remaining respiratory sound cycles of the respiratory sound audio data based on the manually marked respiratory sound cycles to obtain all respiratory sound cycles of the respiratory sound audio data;

[0050] At step S400, each respiratory sound cycle is classified according to its pathological characteristics using the automatic recognition model to obtain a confidence probability for the corresponding category. If the confidence probability of the classification is greater than a set threshold, the respiratory sound cycle is directly marked as the corresponding category. Otherwise, multiple human operators perform independent judgments.

[0051] S500: If all manual judgment results are consistent, the respiratory sound cycle is directly marked as the corresponding category; otherwise, the respiratory sound cycle is discarded, and the labeling of the respiratory sound audio data is completed.

[0052] In S100 of this embodiment, as a preferred embodiment, obtaining respiratory sound audio data may include the following steps:

[0053] The respiratory sounds are collected using sound collection equipment to obtain respiratory sound audio data.

[0054] In S200 of this embodiment, as a preferred embodiment, the number of respiratory sound cycles divided by manual annotation is at least two, and there is no overlapping area between two adjacent respiratory sound cycles.

[0055] In S300 of this embodiment, as a preferred embodiment, calculating the division positions of the remaining respiratory sound cycles of the respiratory sound audio data based on the manually marked respiratory sound cycles may include the following steps:

[0056] S301, obtaining the cycle length and cycle interval of the respiratory sound cycle according to the manually marked respiratory sound cycle;

[0057] S302: Calculate an average value of the cycle length and the cycle interval of the respiratory sound cycle, and divide the remaining respiratory sound cycle positions of the respiratory sound audio data according to the average value.

[0058] In S300 of this embodiment, as a preferred embodiment, the following steps may also be included:

[0059] S303: Manually correct all respiratory sound cycles of the acquired respiratory sound audio data.

[0060] In S400 of this embodiment, as a preferred embodiment, the automatic recognition model may adopt but is not limited to the following models: K-nearest nearest neighbor model, support vector machine model, Gaussian mixture model, deep neural network model, etc.

[0061] The semi-automatic respiratory sound labeling method provided by the above-mentioned embodiment of the present invention uses manual division to divide several respiratory cycles; extracts the time length and interval of the manually divided respiratory cycles, and automatically divides the remaining respiratory cycles based on this. This combination of manual and automatic division reduces the number of manual labeling operations required; at the same time, the manual division provides a reference for the automatically labeled cycles. This combination of manual and automatic labeling enables automatic labeling of respiratory sound classification categories, saving the cost and time of relying solely on manual labeling, achieving a short cycle time, high accuracy, and good economic benefits.

[0062] Figure 2 This is a workflow diagram of a method for semi-automatic annotation of respiratory sounds provided in a preferred embodiment of the present invention.

[0063] like Figure 2 As shown, the semi-automatic breath sound annotation method provided by the preferred embodiment may include the following steps:

[0064] Step 1: Acquire breathing sound audio data collected by a sound collection device; then proceed to step 2.

[0065] In step 2, manually divide a portion of the respiratory sound audio data into several respiratory cycles. When dividing, manually determine the start and end positions of the cycles. The number of divided respiratory cycles should be no less than 2, and there should be no overlap between respiratory cycles to ensure that there is a breathing interval. Then proceed to step 3.

[0066] Step 3: Calculate the average of the cycle lengths and cycle intervals of the divided respiratory cycles, and automatically segment and display the remaining respiratory cycles of the respiratory sound audio data accordingly. For example, the average of the divided cycle lengths may be used as the length of the next respiratory cycle. Then, proceed to Step 4.

[0067] Step 4: Manually (eg, by a doctor) checks and corrects the respiratory cycle boundary line; for each respiratory sound cycle, execute step 5.

[0068] In step 5, the automatic recognition model is used to classify and recognize each divided respiratory sound cycle; then step 6 is executed.

[0069] Step 6: Determine the confidence probabilities of all pathological characteristic sounds of respiratory sounds (i.e., the classified categories). If the confidence probability of a certain category exceeds a preset threshold, execute step 7; otherwise, execute step 8.

[0070] In step 7, the respiratory sound cycle is directly marked as a corresponding category, and then step 5 is performed on the next respiratory sound cycle until all the divided respiratory sound cycles are classified.

[0071] Step 8: Submit the respiratory sound cycle to two or more people (such as doctors) for independent judgment.

[0072] Step 9: Determine whether all manual judgment results are consistent. If all manual judgment results are consistent, execute step 10; otherwise, execute step 11.

[0073] In step 10, the respiratory sound cycle is marked as a manual judgment result, and then step 5 is performed on the next respiratory sound cycle until all the divided respiratory sound cycles are classified.

[0074] Step 11: discard the respiratory sound cycle.

[0075] In some embodiments of the present invention:

[0076] Respiratory sound data collection: Respiratory sounds are collected through sound collection equipment; the sound collection equipment can be a digital stethoscope, etc.

[0077] The automatic recognition model is used to recognize the pathological characteristic sounds of the divided respiratory cycles to obtain multiple categories.

[0078] When using an automatic recognition model to identify respiratory sounds, the output of the automatic recognition model is the confidence probability that the respiratory sound cycle belongs to all categories. N categories have N probabilities. When the confidence probability corresponding to any category is greater than a certain threshold, it can be judged as that category.

[0079] Pathological characteristic sounds include but are not limited to: coarse crackles, fine crackles, rhonchi, wheezing, tubular breath sounds, pleural friction rubs, and crepitus.

[0080] For each respiratory cycle, determine its classification confidence probability.

[0081] If the confidence probability of a certain category exceeds a preset threshold, it will be directly marked as the corresponding category.

[0082] If the confidence probabilities of all categories do not exceed the preset threshold, the respiratory cycle data will be submitted to at least two doctors for independent judgment. If the two doctors have the same judgment results, the respiratory cycle will be marked as the corresponding category, otherwise the respiratory cycle will be discarded.

[0083] Figure 3 A schematic diagram of the structure of a semi-automatic respiratory sound annotation system provided in one embodiment of the present invention.

[0084] like Figure 3 As shown, the semi-automatic respiratory sound annotation system provided in this embodiment may include the following modules:

[0085] A data acquisition module, which is used to acquire respiratory sound audio data;

[0086] A manual period division module, which uses manual annotation to divide a portion of the respiratory sound audio data into several respiratory sound periods;

[0087] An automatic period division module calculates the division positions of the remaining respiratory sound periods of the respiratory sound audio data based on the manually marked respiratory sound periods, and obtains all respiratory sound periods of the respiratory sound audio data;

[0088] Category classification module, which uses the automatic recognition model to classify the pathological features of each respiratory sound cycle and obtain the confidence probability of the corresponding category;

[0089] Category labeling module: This module presets a threshold. When the confidence probability of the classification is greater than the preset threshold, the respiratory sound cycle is directly marked as the corresponding category. Otherwise, multiple manual judgments are made independently. If all manual judgments are consistent, the respiratory sound cycle is directly marked as the corresponding category. Otherwise, the respiratory sound cycle is discarded, completing the labeling of the respiratory sound audio data.

[0090] It should be noted that the steps in the method provided by the present invention can be implemented using corresponding modules, devices, units, etc. in the system. Those skilled in the art can refer to the technical solution of the method to implement the composition of the system, that is, the embodiments in the method can be understood as preferred examples of constructing the system, which will not be elaborated here.

[0091] Figure 4This is a schematic diagram of the structure of a semi-automatic respiratory sound annotation device provided by one embodiment of the present invention.

[0092] like Figure 4 As shown, the semi-automatic respiratory sound annotation device provided in this embodiment may include a user terminal; wherein:

[0093] The user terminal includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor can be used to execute the method of any one of claims 1 to 6, or to execute the system of claim 7 when executing the program.

[0094] In this embodiment, as a preferred embodiment, the device further includes a sound collecting device, and the sound collecting device is connected to the data acquisition module of the system.

[0095] In this embodiment, as a preferred embodiment, the device further includes a display, and the display is connected to the processor.

[0096] In some embodiments of the present invention:

[0097] The sound collection device is used to collect respiratory sound audio data, providing a data basis for subsequent semi-automatic respiratory sound annotation. The sound collection device transmits the collected sound data via, but not limited to, Bluetooth, USB, etc.

[0098] The memory may also be used for other information data related to the respiratory sound audio data, which may include any one or more of the following:

[0099] - Relevant information such as the name, gender, age and / or medical history of the owner of the respiratory sound audio data;

[0100] - Basic information such as the duration, collection time and / or collection location of the respiratory sound audio data;

[0101] - Basic information of each breath sound cycle is recorded, including but not limited to: start time, end time, duration and pathological feature type (category).

[0102] The display is used to display the collected respiratory sound audio data sequence, division marks and / or the above information data.

[0103] The semi-automatic labeling method, system and equipment for respiratory sounds provided by the present invention solve the problems of long cycle, high cost and poor result reliability existing in the existing technology that relies on fully manual labeling. The semi-automatic labeling method that combines manual labeling and automatic labeling is adopted. While automatically labeling the pathological feature classification information of respiratory sounds, it ensures the accuracy of the labeled information, saves the cost and time of relying entirely on manual labeling, has the advantages of short cycle, high accuracy and good economic benefits, and can accurately identify respiratory sounds, and can be used in the auxiliary diagnosis of respiratory diseases.

[0104] Matters not mentioned in the above embodiments of the present invention are well known in the art.

[0105] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art may make various variations or modifications within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A semi-automatic respiratory sound annotation system, characterized in that: include: A data acquisition module, which is used to acquire respiratory sound audio data; a period manual division module, which divides a portion of the respiratory sound audio data into a plurality of respiratory sound periods by manual annotation; wherein the number of respiratory sound periods divided by manual annotation is at least two, and there is no overlapping area between two adjacent respiratory sound periods; an automatic period division module, which calculates the division positions of the remaining respiratory sound periods of the respiratory sound audio data based on the manually marked respiratory sound periods to obtain all respiratory sound periods of the respiratory sound audio data; wherein, based on the manually marked respiratory sound periods, the period lengths and period intervals of the respiratory sound periods are obtained; an average of the period lengths and period intervals of the respiratory sound periods is taken, and the remaining respiratory sound period positions of the respiratory sound audio data are divided based on the average; and all obtained respiratory sound periods of the respiratory sound audio data are manually corrected; A category classification module, which uses an automatic recognition model to classify each respiratory sound cycle into pathological features and obtain a confidence probability of the corresponding category; The category labeling module presets a threshold. When the confidence probability of the classification is greater than the preset threshold, the respiratory sound cycle is directly marked as the corresponding category. Otherwise, multiple manual judgments are made independently. If all manual judgments are consistent, the respiratory sound cycle is directly marked as the corresponding category. Otherwise, the respiratory sound cycle is discarded, completing the labeling of the respiratory sound audio data.

2. The semi-automatic respiratory sound annotation system according to claim 1, characterized in that: The obtaining of the respiratory sound audio data includes: using a sound collection device to collect the respiratory sound to obtain the respiratory sound audio data.

3. The semi-automatic respiratory sound annotation system according to claim 1, characterized in that: The automatic recognition model includes: a K-nearest neighbor model, a support vector machine model, a Gaussian mixture model and / or a deep neural network model.

Images