Diagnosis device providing diagnosis information and diagnosis basis information, and method of operating the same

Abstract

A method of operating a diagnostic device for diagnosing a user's disease according to one embodiment of the present disclosure may include: a step of obtaining breath sound data by measuring the user's breath; a step of generating feature variable information including a plurality of variables corresponding to at least one abnormal breath sound based on the analysis of the breath sound data; a step of generating diagnostic information indicating a target disease having the highest model output value among a plurality of diseases by applying the feature variable information to a pre-trained differential diagnostic model; a step of generating diagnostic basis information quantifying the importance of a plurality of variables used to determine the target disease; and a step of outputting the diagnostic information and the diagnostic basis information through a user interface device of the diagnostic device.

Description

Technology Field

[0001] The present disclosure relates to a diagnostic device, and more specifically, to a diagnostic device that provides diagnostic information and diagnostic basis information, and a method of operating the same. Background Technology

[0002] Abnormal breath sounds in patients are a crucial factor in diagnosing diseases. Disease diagnostic models using deep neural networks can detect abnormal breath sounds and classify diseases based on them, but they do not provide specific grounds for such classification. Consequently, there is a problem in that professional institutions, such as medical facilities, find it difficult to trust the disease classification results, despite the accuracy of the diagnostic models.

[0003] Therefore, in order to classify a patient's disease based on their breath sounds and explain the basis for the disease classification, a diagnostic device may be required that can provide the importance of the characteristics of abnormal breath sounds used to diagnose the disease in a form that a physician can understand as the basis for the diagnosis. The problem to be solved

[0004] According to one embodiment of the present disclosure, a diagnostic device that provides diagnostic information and diagnostic basis information and a method of operating the same are provided. means of solving the problem

[0005] According to one embodiment of the present disclosure, a method of operating a diagnostic device for diagnosing a disease of a user comprises: a step of obtaining breath sound data by measuring the user’s breath; a step of generating feature variable information including a plurality of variables corresponding to at least one abnormal breath sound based on an analysis of the breath sound data; a step of generating diagnostic information indicating a target disease having the highest model output value among a plurality of diseases by applying the feature variable information to a pre-trained differential diagnostic model; a step of generating diagnostic basis information quantifying the importance of the plurality of variables used to determine the target disease; and a step of outputting the diagnostic information and the diagnostic basis information through a user interface device of the diagnostic device.

[0006] According to one embodiment, the step of generating diagnostic information pointing to the target disease having the highest model output value among the plurality of diseases by applying the feature variable information to the pre-learned differential diagnostic model comprises: observing a change in the output of the pre-learned differential diagnostic model according to a change in at least one variable among the plurality of variables of the feature variable information; calculating the importance of the plurality of variables according to the change in the output of the pre-learned differential diagnostic model; and generating diagnostic information pointing to the target disease based on the importance of the plurality of variables.

[0007] According to one embodiment, in the method, the importance of the plurality of variables represents the priority of each of the plurality of variables with respect to the target disease.

[0008] According to one embodiment, in the method, the plurality of variables of the feature variable information include at least one of the number of occurrences, intensity of occurrence, start time, end time, duration, lowest frequency, highest frequency, and average frequency of the at least one abnormal breath sound.

[0009] According to one embodiment, the step of generating the feature variable information including the plurality of variables corresponding to the abnormal breath sound based on the analysis of the breath sound data comprises: generating spectrogram data that visually represents the time component and frequency component of the at least one abnormal breath sound based on the breath sound data; generating heatmap data including pixels having temperature values ​​corresponding to the probability of the at least one abnormal breath sound based on the convolutional neural network operation of the spectrogram data; and generating the feature variable information based on the spectrogram data and the heatmap data.

[0010] According to one embodiment, the step of generating the feature variable information based on the spectrogram data and the heatmap data comprises: detecting a blob region by filtering the heatmap data based on a threshold temperature value; extracting the coordinates of the blob region from the heatmap data; and generating feature variable information corresponding to the time component and the frequency component of the at least one abnormal breath sound based on the spectrogram data and the extracted coordinates.

[0011] According to one embodiment, the step of generating feature variable information corresponding to the time component and the frequency component of the at least one abnormal breath sound based on the spectrogram data and the extracted coordinates includes the step of calculating the plurality of variables corresponding to the time component based on the horizontal axis coordinate values ​​of the extracted coordinates and a reference time length, wherein the reference time length is a mathematical formula It is determined based on, where t p indicates the above reference time length, and S rate indicates the sampling rate of the above breath sound data, and L hop represents the hop length of the spectrogram data, and the plurality of variables corresponding to the time component are mathematical formulas It is determined based on, where t p indicates the above reference time length, and b x indicates one of the above horizontal axis coordinate values, and t x indicates one of the plurality of variables corresponding to the above time component.

[0012] According to one embodiment, the step of generating feature variable information corresponding to the time component and the frequency component of the at least one abnormal breath sound based on the spectrogram data and the extracted coordinates includes the step of calculating a plurality of variables corresponding to the frequency component based on the vertical axis coordinate values ​​of the extracted coordinates and a reference frequency magnitude, wherein the reference frequency magnitude is a mathematical formula It is determined based on, where f p indicates the magnitude of the above reference frequency, and S rate indicates the sampling rate of the above breath sound data, and N f indicates the number of reference samples, and multiple variables corresponding to the frequency components are in a mathematical formula It is determined based on, where f p indicates the magnitude of the above reference frequency, and b y indicates one of the above vertical axis coordinate values, and f y indicates one of a plurality of variables corresponding to the above frequency component.

[0013] According to one embodiment, the step of generating the heatmap data including the pixels each having the temperature value corresponding to the probability of at least one abnormal breath sound based on the convolutional neural network operation of the spectrogram data includes the step of generating the heatmap data from the spectrogram data based on a top-down method.

[0014] According to an embodiment of the present disclosure, a diagnostic device comprises: a voice sensor configured to measure the breath of a user and generate breath sound data; a breath sound analysis device configured to generate spectrogram data that visually represents the time component and frequency component of at least one abnormal breath sound based on the analysis of the breath sound data, and to generate heatmap data including pixels having temperature values ​​corresponding to the probability of corresponding to the at least one abnormal breath sound based on the convolutional neural network operation of the spectrogram data; a feature variable extraction device configured to generate feature variable information including a plurality of variables corresponding to the at least one abnormal breath sound based on the spectrogram data and the heatmap data; a differential diagnosis device configured to generate diagnostic information indicating a target disease having the highest model output value among a plurality of diseases by applying the feature variable information to a pre-trained differential diagnosis model, and to generate diagnostic basis information quantifying the importance of the plurality of variables used to determine the target disease; and a user interface device configured to output the diagnostic information and the diagnostic basis information.

[0015] According to one embodiment, the breath sound analysis device includes a spectrogram generator that receives breath sound data from the voice sensor and generates spectrogram data from the breath sound data, and an abnormal breath sound detection device that receives spectrogram data from the spectrogram generator and generates heatmap data from the spectrogram data.

[0016] According to one embodiment, the feature variable extraction device comprises: a blob area detector that receives heatmap data from the abnormal breath sound detection device, detects a blob area by filtering the heatmap data based on a threshold temperature value, extracts coordinates of the blob area from the heatmap data, and generates blob area information pointing to the extracted coordinates; and a feature variable generator that receives spectrogram data from the spectrogram generator, receives blob area information from the blob area detector, and generates feature variable information corresponding to the time component and the frequency component of the at least one abnormal breath sound based on the spectrogram data and the blob area information.

[0017] According to one embodiment, the differential diagnosis device comprises: a diagnostic information generator that receives feature variable information from the feature variable extraction device, observes a change in the output of the pre-learned differential diagnosis model according to a change in at least one of the plurality of variables of the feature variable information, calculates the importance of the plurality of variables according to the change in the output of the pre-learned differential diagnosis model, and generates diagnostic information indicating the target disease based on the importance of the plurality of variables; and a diagnostic basis information generator that generates diagnostic basis information quantifying the importance of the plurality of variables used to determine the target disease. Effects of the invention

[0018] According to one embodiment of the present disclosure, a diagnostic device that provides diagnostic information and diagnostic basis information and a method of operating the same are provided.

[0019] In addition, a diagnostic device and a method of operation thereof are provided, which enhance user convenience and increase usability by generating multiple variables indicating the characteristics of abnormal breath sounds as well as diagnostic information indicating classified diseases, and providing the user with diagnostic basis information that quantifies the importance of each of the multiple variables used to classify diseases. Brief explanation of the drawing

[0020] FIG. 1 is a block diagram illustrating a diagnostic device according to an embodiment of the present disclosure. FIG. 2 is a drawing illustrating the breath sound analysis device of FIG. 1 in accordance with some embodiments of the present disclosure. FIG. 3 is a drawing illustrating the feature variable extraction device of FIG. 1 in accordance with some embodiments of the present disclosure. FIG. 4 is a drawing illustrating the differential diagnosis device of FIG. 1 in accordance with some embodiments of the present disclosure. FIG. 5 is a drawing illustrating blob area information according to some embodiments of the present disclosure. FIG. 6 is a flowchart illustrating a method of operation of a diagnostic device according to some embodiments of the present disclosure. FIG. 7 is a flowchart illustrating a method of operation of a differential diagnosis device according to some embodiments of the present disclosure. Specific details for implementing the invention

[0021] In the following, embodiments of the present invention will be described clearly and in detail so that a person skilled in the art can easily practice the present invention.

[0022] The functional blocks illustrated in the drawings used below may be implemented in the form of software configurations, hardware configurations, or combinations thereof. In order to clearly explain the technical concept of the present invention, detailed descriptions of redundant components are omitted below.

[0023] FIG. 1 is a block diagram illustrating a diagnostic device according to an embodiment of the present disclosure. Referring to FIG. 1, a diagnostic device (100) is illustrated. The diagnostic device (100) may be a device that generates disease diagnosis information and diagnosis basis information for a user and provides them to the user through a user interface device. The user may be a person who has a respiratory disease or is suspected of having a respiratory disease as the subject of respiration. For example, the diagnostic device (100) may be implemented as one of various electronic devices that analyze breath sounds, such as a smartphone, a tablet PC (tablet personal computer), a desktop, a PC, and a laptop.

[0024] The diagnostic device (100) may include a voice sensor (110), a breath sound analysis device (120), a feature variable extraction device (130), a differential diagnosis device (140), and a user interface device (150). For example, the voice sensor (110) and the user interface device (150) are configured to be included in electronic devices such as smartphones and tablet PCs, and the breath sound analysis device (120), the feature variable extraction device (130), and the differential diagnosis device (140) may be servers that communicate with smartphones and tablet PCs, etc.

[0025] The voice sensor (110) can obtain breath sound data by measuring the user's breathing. The voice sensor (110) can measure the user's breathing for a certain period of time. The breath sound data may be voice data corresponding to the user's breath sound. The voice sensor (110) can provide the breath sound data to the breath sound analysis device (120).

[0026] The breath sound analysis device (120) can receive breath sound data from the voice sensor (110). The breath sound analysis device (120) can analyze the breath sound data and generate image data corresponding to at least one abnormal breath sound. The abnormal breath sound may refer to a breath sound that has a correlation with the user's disease. The abnormal breath sound may have time characteristics above a threshold value or frequency characteristics above a threshold value.

[0027] The breath sound analysis device (120) can generate spectrogram data based on breath sound data. The spectrogram data may be image data that visually represents the time component and frequency component of at least one abnormal breath sound. A more detailed description of the spectrogram data will be given later with reference to FIG. 2.

[0028] The breath sound analysis device (120) can generate heat map data based on spectrogram data. The heat map data may be image data containing pixels each having a temperature value corresponding to the probability of at least one abnormal breath sound based on a Convolutional Neural Network (CNN) operation of the spectrogram data. A more detailed description of the heat map data will be given later with reference to FIG. 2.

[0029] The breath sound analysis device (120) can provide spectrogram data and heatmap data to the feature variable extraction device (130).

[0030] The feature variable extraction device (130) can receive spectrogram data and heatmap data from the breath sound analysis device. Based on the spectrogram data and heatmap data, the feature variable extraction device (130) can generate feature variable information including a plurality of variables corresponding to at least one abnormal breath sound. For example, the plurality of variables may include at least one of the number of occurrences, intensity of occurrence, start time, end time, duration, lowest frequency, highest frequency, and average frequency of at least one abnormal breath sound. The feature variable extraction device (130) can provide the feature variable information to the differential diagnosis device (140).

[0031] The differential diagnosis device (140) can receive feature variable information from the feature variable extraction device (130). The differential diagnosis device (140) can generate diagnostic information and diagnostic basis information by applying the feature variable information to a pre-trained diagnostic model. The pre-trained diagnostic model may be a machine learning model that includes neural network operations. The neural network operations may include learning and inference operations. The pre-trained diagnostic model may be obtained by the differential diagnosis device (140) learning based on training data during the learning stage, which is a stage prior to disease inference. For example, the training data may be large-scale data representing disease-breath sounds.

[0032] Diagnostic information may represent the target disease having the highest model output value among multiple diseases in a pre-trained diagnostic model. The model output value may represent the likelihood (e.g., probability) of corresponding to the disease.

[0033] Diagnostic basis information may be information that quantifies the importance of each of the multiple variables used to determine the target disease.

[0034] In some embodiments, the importance of multiple variables may indicate the priority of each of the multiple variables with respect to the target disease.

[0035] The differential diagnosis device (140) can provide diagnosis information and diagnosis basis information to the user interface device (150).

[0036] The user interface device (150) can receive diagnostic information and diagnostic basis information from the differential diagnosis device (140). The user interface device (150) can provide the diagnostic information and diagnostic basis information to the user.

[0037] FIG. 2 is a drawing that embodies the breath sound analysis device of FIG. 1 according to some embodiments of the present disclosure. Referring to FIG. 2, a breath sound analysis device (120) is shown.

[0038] The breath sound analysis device (120) may include a spectrogram generator (121) and an abnormal breath sound detection device (122).

[0039] The spectrogram generator (121) can receive breath sound data (BSD) from the voice sensor (110). The spectrogram generator can generate spectrogram data (SP) from the breath sound data (BSD).

[0040] Spectrogram data (SP) is image data in the form of a graph, where the horizontal axis represents time and the vertical axis represents frequency. The value of each pixel in the spectrogram data (SP) can represent the intensity of the breath sound corresponding to the time and frequency of each pixel. For example, if the intensity of the breath sound corresponding to the pixel's time and frequency is high, the pixel may have a bright color, and if the intensity of the breath sound corresponding to the pixel's time and frequency is low, the pixel may have a dark color.

[0041] The spectrogram generator (121) can provide spectrogram data (SP) to the abnormal breath sound detection device (122) and the feature variable extraction device (130).

[0042] The abnormal breath sound detection device (122) can receive spectrogram data (SP) from the spectrogram generator (121). The abnormal breath sound detection device (122) can detect abnormal breath sounds by applying a convolutional neural network to the spectrogram data.

[0043] The abnormal breathing sound detection device (122) may include a CAM operator. The CAM operator may generate heatmap data (HM) by presenting the elements that served as the basis for the convolutional neural network to detect abnormal breathing sounds. Each pixel of the heatmap data (HM) may have a high temperature value when there is a high probability that it corresponds to at least one abnormal breathing sound, and a low temperature value when there is a low probability that it corresponds to at least one abnormal breathing sound. For example, a pixel with a high temperature value may be displayed in red and referred to as a hot-region. A pixel with a low temperature value may be displayed in blue and referred to as a cold-region.

[0044] In some embodiments, heatmap data (HM) may refer to a top-down heatmap. The CAM calculator may generate a top-down heatmap based on a top-down method.

[0045] Generally, neural networks include multiple layers, and neural network operations can be performed by sequentially applying these layers to image data from lower to higher layers. Lower layers are closer to the input of the neural network and may have larger filter data sizes. Higher layers are closer to the output of the neural network and may have smaller filter data sizes.

[0046] For lower layers, heatmaps generated based on the layer may have higher resolution but lower object detection accuracy. Conversely, for higher layers, heatmaps generated based on the layer may have lower resolution but higher object detection accuracy.

[0047] The top-down approach may be a method that generates a high-resolution heatmap with high object identification accuracy by applying multiple layers to image data in reverse order. In other words, the top-down approach may be a method that applies layers sequentially to image data from the highest layer to the lowest layer.

[0048] The abnormal breathing sound detection device (122) can provide heatmap data (HM) to the feature variable extraction device (130).

[0049] FIG. 3 is a drawing that embodies the feature variable extraction device of FIG. 1 according to some embodiments of the present disclosure. Referring to FIG. 3, a feature variable extraction device (130) may be illustrated.

[0050] The feature variable extraction device (130) may include a blob area detector (131) and a feature variable generator (132).

[0051] The blob area detector (131) can receive heatmap data (HM) from the abnormal breathing sound detection device (122). The blob area detector (131) can detect a blob area by filtering the heatmap data (HM) based on a threshold temperature value. The threshold temperature value can be set by a user. A blob area is an area corresponding to at least one abnormal breathing sound and may be an area containing pixels having a temperature value higher than the threshold temperature value. In other words, the blob area may include a hot-region of the heatmap data (HM).

[0052] Referring to FIG. 3, the heatmap data (HM) may include blob area information. The blob area information is indicated as a rectangular area, but the scope of the present disclosure is not limited thereto.

[0053] The blob area detector (131) can generate blob area information (BI) by extracting the coordinates of the blob area from the heatmap data (HM). The blob area information (BI) may indicate the coordinates of the blob area as the location of the blob area in the heatmap data (HM). The coordinates of the blob area may include the coordinates of each vertex of the blob area and coordinates within the blob area. A more detailed description of the blob area information (BI) will be given later with reference to FIG. 5.

[0054] The blob area detector (131) can provide blob area information (BI) to the feature variable generator (132).

[0055] The feature variable generator (132) can receive spectrogram data (SP) from the spectrogram generator (121) and receive blob area information (BI) from the blob area detector (131). Based on the spectrogram data (SP) and the blob area information (BI), the feature variable generator (132) can generate feature variable information (FP) corresponding to the time component and frequency component of at least one abnormal breath sound. The feature variable information (FP) may include a plurality of variables indicating the time component and frequency component of at least one abnormal breath sound. Each of the plurality of variables may also be referred to as a feature variable.

[0056] The feature variable generator (132) can generate feature variable information (FP) corresponding to the time component of at least one abnormal breath sound by applying the horizontal axis coordinate values ​​of the blob area information (BI) to the spectrogram data (SP). For example, the feature variable information (FP) corresponding to the time component may include at least one of the number of occurrences of the abnormal breath sound, the start time, the end time, and the duration.

[0057] The feature variable generator (132) can generate feature variable information (FP) corresponding to at least one frequency component of an abnormal breath sound by applying vertical axis coordinate values ​​of blob area information (BI) to spectrogram data (SP). For example, the feature variable information (FP) corresponding to the frequency component may include at least one of the occurrence intensity, lowest frequency, highest frequency, and average frequency of the abnormal breath sound.

[0058] A more detailed explanation regarding the process of generating feature variable information (FP) will be provided later with reference to Fig. 5.

[0059] FIG. 4 is a diagram illustrating the differential diagnosis device of FIG. 1 in accordance with some embodiments of the present disclosure. Referring to FIG. 4, the differential diagnosis device (140) and diagnostic basis information (DBI) are illustrated.

[0060] The differential diagnosis device (140) may include a diagnosis information generator (141) and a diagnosis basis information generator (142).

[0061] The diagnostic information generator (141) can receive feature variable information (FP) from the feature variable generator (132). The diagnostic information generator (141) may include a pre-trained differential diagnosis model (DDM). The pre-trained differential diagnosis model (DDM) may be a machine learning model that receives multiple variables of the feature variable information (FP) as input and outputs one of multiple diseases. By applying the feature variable information (FP) to the pre-trained differential diagnosis model (DDM), the diagnostic information generator (141) can generate diagnostic information (DI) that indicates the target disease having the highest model output value among the multiple diseases.

[0062] In some embodiments, the diagnostic information generator (141) can observe a change in the output of a pre-learned differential diagnostic model (DDM) according to a change in at least one of the multiple variables of the feature variable information (FP). For example, the output of the pre-learned differential diagnostic model (DDM) when one of the multiple variables is present and the output of the pre-learned differential diagnostic model (DDM) when one of the multiple variables is absent can be compared.

[0063] The diagnostic information generator (141) can calculate the importance of multiple variables based on changes in the output of a pre-trained differential diagnosis model (DDM). The diagnostic information generator (141) can generate diagnostic information (DI) pointing to a target disease based on the importance of multiple variables.

[0064] The diagnostic information generator (141) can provide diagnostic information (DI) to the user interface device (150). The diagnostic information generator (141) can transmit a pre-trained differential diagnosis model (DDM) to the diagnostic basis information generator (142).

[0065] The diagnostic basis information generator (142) receives feature variable information (FP) from the feature variable generator (132) and can receive a pre-trained differential diagnostic model (DDM) from the diagnostic information generator (141).

[0066] The diagnostic basis information generator (142) can generate diagnostic basis information (DBI) by quantifying the importance of multiple variables used to determine the target disease based on feature variable information (FP) and a pre-learned differential diagnosis model (DDM).

[0067] In some embodiments, the diagnostic basis information (DBI) may quantify and represent the importance of each of the multiple variables for each of the multiple diseases in the diagnosis of the disease. For example, the diagnostic basis information (DBI) may represent the importance of each of the first to nth variables (FP1 to FPn) used to determine one of the first to third diseases as the target disease. The vertical axis may indicate the first to nth variables (FP1 to FPn), and the horizontal axis may indicate the quantified importance.

[0068] In the Diagnostic Basis Information (DBI), a solid line may indicate a first disease. In the Diagnostic Basis Information (DBI), a dotted line may indicate a second disease. In the Diagnostic Basis Information (DBI), a dashed line may indicate a third disease. However, the scope of the present disclosure is not limited thereto.

[0069] For example, regarding the fact that a pre-trained diagnostic model (DDM) has determined the first disease as a target disease, the diagnostic basis information (DBI) may represent the first variable (FP1), the second variable (FP2), and the third variable, etc., as the basis for diagnosing the first disease in order of decreasing quantified importance.

[0070] For example, regarding the fact that a pre-trained diagnostic model (DDM) has determined the second disease as a target disease, the diagnostic basis information (DBI) may represent the third variable (FP3) and the second variable (FP2), etc., as the basis for diagnosing the second disease in order of decreasing quantified importance.

[0071] For example, regarding the pre-trained diagnostic model (DDM) determining the third disease as the target disease, the diagnostic basis information (DBI) may represent the nth variable (FPn) and the (n-1)th variable (FPn-1), etc., as the basis for diagnosing the second disease in order of decreasing quantified importance.

[0072] The diagnostic basis information generator (142) can provide diagnostic basis information (DBI) to the user interface device (150).

[0073] FIG. 5 is a diagram illustrating blob area information according to some embodiments of the present disclosure. Referring to FIG. 5, blob area information (BI) is described. The heatmap data (HM) and blob area information (BI) of FIG. 5 may correspond to the heatmap data (HM) and blob area information (BI) of FIG. 3.

[0074] The blob area detector can set coordinate axes in the heatmap data (HM) and calculate the coordinate values ​​contained within the blob area. The horizontal axis set in the heatmap data (HM) can correspond to the temporal characteristics of the blob area, and the vertical axis set can correspond to the frequency characteristics of the blob area.

[0075] For example, the blob area information may include the coordinates of each of the first to fourth points (P1~P4), which are the vertices of the blob area information.

[0076] The horizontal axis coordinate of any point in the blob area information may indicate the number of horizontal pixels between the origin (O) and any point. The vertical axis coordinate of any point in the blob area information may indicate the number of vertical pixels between the origin (O) and any point. The size of a single pixel in the heatmap data (HM) may be the same as the size of a single pixel in the spectrogram data of the same size as the heatmap data (HM). The size of a single pixel in the spectrogram data may be determined based on a reference time length and a reference frequency length, and a more detailed explanation thereof will be provided later.

[0077] For example, the horizontal axis coordinate of the first point (P1) may be 13, which is the number of horizontal pixels between the origin (O) and the first point (P1). The vertical axis coordinate of the first point (P1) may be 0, which is the number of vertical pixels between the origin (O) and the first point (P1).

[0078] For example, the horizontal axis coordinate of the second point (P2) may be 16, which is the number of horizontal pixels between the origin (O) and the second point (P2). The vertical axis coordinate of the second point (P2) may be 0, which is the number of vertical pixels between the origin (O) and the second point (P2).

[0079] For example, the horizontal axis coordinate of the third point (P3) may be 13, which is the number of horizontal pixels between the origin (O) and the third point (P3). The vertical axis coordinate of the third point (P3) may be 3, which is the number of vertical pixels between the origin (O) and the third point (P3).

[0080] For example, the horizontal axis coordinate of the fourth point (P4) may be 16, which is the number of horizontal pixels between the origin (O) and the fourth point (P4). The vertical axis coordinate of the fourth point (P4) may be 3, which is the number of vertical pixels between the origin (O) and the fourth point (P4).

[0081] Referring to FIG. 5, the blob area information is illustrated as including first to fourth points (P1 to P4), but the scope of the present disclosure is not limited thereto.

[0082] The feature variable generator can generate feature variable information corresponding to the time component of abnormal breath sounds by applying the horizontal axis coordinate values ​​of blob area information to spectrogram data.

[0083] In some embodiments, the size of the spectrogram data and the size of the heatmap data may be the same.

[0084] In some embodiments, the feature variable generator can calculate multiple variables indicating time components based on the horizontal axis coordinate values ​​of the blob area information and a reference time length. The reference time length may indicate the time length of one pixel of the spectrogram data. The reference time length may be determined by Equation 1.

[0085]

[0086] Mathematical Equation 1 can be used to determine the reference time length. Here, t p indicates the reference time length, and S rate indicates the sampling rate of the breath sound data, and L hop can indicate the hop length of spectrogram data.

[0087] The sampling rate of breath sound data may represent the number of samplings per unit time when the voice sensor generates breath sound data from the user's breath. The hop length of the spectrogram data may determine the size of the data contained in one pixel of the spectrogram data based on the size of the breath sound data.

[0088] Multiple variables indicating the time component can be determined by mathematical formula 2.

[0089]

[0090] Mathematical Equation 2 can be used to determine multiple variables indicating the time component. Here, t p indicates the reference time length, and b x indicates one of the horizontal axis coordinate values, and t x can refer to one of multiple variables that refer to the time component.

[0091] For example, the start time of the blob area information can be calculated by multiplying the horizontal axis coordinate value of 13 at the first point (P1) or the third point (P3) by the reference time length. The end time of the first blob area can be calculated by multiplying the horizontal axis coordinate value of 16 at the second point (P2) or the fourth point (P4) by the reference time length.

[0092] The feature variable generator can generate feature variable information corresponding to the frequency components of abnormal breath sounds by applying the vertical axis coordinate values ​​of the blob area information to the spectrogram data.

[0093] In some embodiments, the feature variable generator can calculate a plurality of variables indicating frequency components based on the vertical axis coordinate values ​​of the blob area information and the reference frequency magnitude. The reference frequency magnitude may indicate the frequency magnitude of one pixel of the spectrogram data. The reference frequency magnitude may be determined by Equation 3.

[0094]

[0095] Equation 3 can be used to determine the magnitude of the reference frequency. Here, f p indicates the reference frequency magnitude, and S rate indicates the sampling rate of the breath sound data, and N f can indicate the number of reference samples.

[0096] The number of reference samples can represent the number of breath sound samples to reference per transformation operation when generating spectrogram data from breath sound data.

[0097] Multiple variables indicating frequency components can be determined by mathematical equation 4.

[0098]

[0099] Equation 4 can be used to determine multiple variables indicating frequency components. Here, f p indicates the magnitude of the reference frequency, and b y indicates one of the vertical axis coordinate values, and f y can refer to one of multiple variables that indicate frequency components.

[0100] For example, the lowest frequency of the blob area information can be calculated by multiplying the vertical axis coordinate value of the first point (P1) or the second point (P2), which is 0, by the reference frequency magnitude. For example, the highest frequency of the blob area information can be calculated by multiplying the vertical axis coordinate value of the third point (P3) or the fourth point (P4), which is 3, by the reference frequency magnitude.

[0101] FIG. 6 is a flowchart illustrating a method of operation of a diagnostic device according to some embodiments of the present disclosure. With reference to FIG. 6, a method of operation of the diagnostic device is described. The diagnostic device of FIG. 6 may correspond to the diagnostic device (100) of FIG. 1.

[0102] In step S110, the diagnostic device can obtain breath sound data by measuring the user's breathing.

[0103] In step S120, the diagnostic device can generate feature variable information including multiple variables corresponding to at least one abnormal breath sound based on the analysis of breath sound data.

[0104] In some embodiments, step S120 may include generating spectrogram data that visually represents the time component and frequency component of at least one abnormal breath sound based on breath sound data, generating heatmap data including pixels having temperature values ​​corresponding to the probability of at least one abnormal breath sound based on convolutional neural network operations of the spectrogram data, and generating feature variable information based on the spectrogram data and heatmap data.

[0105] In some embodiments, the size of the spectrogram data and the size of the heatmap data may be the same.

[0106] In some embodiments, step S120 may further include detecting a blob area by filtering heatmap data based on a threshold temperature value, extracting coordinates of the blob area from the heatmap data, and generating feature variable information corresponding to the time component and frequency component of at least one abnormal breath sound based on the spectrogram and the extracted coordinates.

[0107] In some embodiments, step S120 may further include generating feature variable information corresponding to the time component and frequency component of at least one abnormal breath sound of the blob area data by overlapping the heatmap data and the spectrogram data, that is, by applying the coordinates extracted from the heatmap data to the spectrogram data.

[0108] In step S130, the diagnostic device can generate diagnostic information pointing to the target disease having the highest model output value among a plurality of diseases by applying feature variable information to a pre-trained differential diagnosis model.

[0109] In step S140, the diagnostic device can generate diagnostic basis information that quantifies the importance of multiple variables used to determine the target disease.

[0110] In step S150, the diagnostic device can output diagnostic information and diagnostic basis information through the user interface device of the diagnostic device.

[0111] FIG. 7 is a flowchart illustrating a method for generating diagnostic information of a differential diagnostic device according to some embodiments of the present disclosure. Referring to FIG. 7, a method for generating diagnostic information of a differential diagnostic device is described. The differential diagnostic device of FIG. 7 may correspond to the differential diagnostic device (140) of FIG. 1.

[0112] In step S231, the differential diagnosis device can observe a change in the output of a pre-learned differential diagnosis model according to a change in at least one variable among a plurality of variables of feature variable information.

[0113] In some embodiments, step S231 may further include comparing the output of a pre-trained differential diagnosis model (DDM) when one of the plurality of variables is present and the output of a pre-trained differential diagnosis model (DDM) when one of the plurality of variables is absent.

[0114] In step S232, the differential diagnosis device can calculate the importance of multiple variables based on changes in the output of a pre-trained differential diagnosis model.

[0115] In step S233, the differential diagnosis device (140) can generate diagnostic information indicating a target disease based on the importance of multiple variables.

[0116] The above description describes specific embodiments for implementing the present invention. The present invention will include not only the embodiments described above, but also embodiments that can be simply modified or easily modified. Furthermore, the present invention will include technologies that can be easily modified and implemented using the embodiments. Accordingly, the scope of the present invention should not be limited to the embodiments described above, but should be defined by the claims set forth below as well as equivalents to the claims of this invention. Explanation of the symbols

[0117] 100: Diagnostic device 110: Voice sensor 120: Breath sound analyzer 121: Spectrogram Generator 122: Abnormal breath sound detection device 130: Feature Extraction Device 131: Blob Area Detector 132: Feature Variable Generator 140: Differential diagnostic device 141: Diagnostic Information Generator 142: Diagnostic Basis Information Generator 150: User interface device

Claims

Claim 1 A method of operating a diagnostic device for diagnosing a disease of a user comprises: a step of obtaining respiratory sound data by measuring the user's respiration; a step of generating feature variable information including a plurality of variables corresponding to at least one abnormal respiratory sound based on the analysis of the respiratory sound data; a step of generating diagnostic information indicating a target disease having the highest model output value among a plurality of diseases by applying the feature variable information to a pre-trained differential diagnostic model; a step of generating diagnostic basis information quantifying the importance of the plurality of variables used to determine the target disease; and a step of outputting the diagnostic information and the diagnostic basis information through a user interface device of the diagnostic device, wherein the step of generating the diagnostic information indicating the target disease having the highest model output value among the plurality of diseases by applying the feature variable information to the pre-trained differential diagnostic model comprises: a step of observing a change in the output of the pre-trained differential diagnostic model according to a change in at least one variable among the plurality of variables of the feature variable information; and a step of calculating the importance of the plurality of variables according to the change in the output of the pre-trained differential diagnostic model. A method comprising the step of generating diagnostic information indicating the target disease based on the importance of the plurality of variables. Claim 2 delete Claim 3 In claim 1, the importance of the plurality of variables indicates the priority of each of the plurality of variables with respect to the target disease. Claim 4 A method according to claim 1, wherein the plurality of variables of the characteristic variable information include at least one of the number of occurrences, intensity of occurrence, start time, end time, duration, lowest frequency, highest frequency, and average frequency of at least one abnormal breath sound. Claim 5 In claim 1, the step of generating the feature variable information including the plurality of variables corresponding to the abnormal breath sound based on the analysis of the breath sound data comprises: generating spectrogram data that visually represents the time component and frequency component of the at least one abnormal breath sound based on the breath sound data; generating heatmap data including pixels each having a temperature value corresponding to the probability of the at least one abnormal breath sound based on the convolutional neural network operation of the spectrogram data; and generating the feature variable information based on the spectrogram data and the heatmap data. Claim 6 In claim 5, the step of generating the feature variable information based on the spectrogram data and the heatmap data comprises: detecting a blob region by filtering the heatmap data based on a threshold temperature value; extracting the coordinates of the blob region from the heatmap data; and generating feature variable information corresponding to the time component and the frequency component of the at least one abnormal breath sound based on the spectrogram data and the extracted coordinates. Claim 7 In claim 6, the step of generating the feature variable information corresponding to the time component and the frequency component of the at least one abnormal breath sound based on the spectrogram data and the extracted coordinates comprises: the step of calculating the plurality of variables corresponding to the time component based on the horizontal axis coordinate values ​​of the extracted coordinates and a reference time length, wherein the reference time length is a mathematical formula It is determined based on, where t p indicates the above reference time length, and S rate indicates the sampling rate of the above breath sound data, and L hop indicates the hop length of the spectrogram data, and the plurality of variables corresponding to the time component are mathematical formulas It is determined based on, where t p indicates the above reference time length, and b x indicates one of the above horizontal axis coordinate values, and t x A method indicating one of the plurality of variables corresponding to the above time component. Claim 8 In claim 6, the step of generating the feature variable information corresponding to the time component and the frequency component of the at least one abnormal breath sound based on the spectrogram data and the extracted coordinates comprises: the step of calculating a plurality of variables corresponding to the frequency component based on the vertical axis coordinate values ​​of the extracted coordinates and a reference frequency magnitude, wherein the reference frequency magnitude is a mathematical formula It is determined based on, where f p indicates the magnitude of the above reference frequency, and S rate indicates the sampling rate of the above breath sound data, and N f indicates the number of reference samples, and the plurality of variables corresponding to the above frequency components are mathematical formulas It is determined based on, where f p indicates the magnitude of the above reference frequency, and b y indicates one of the above vertical axis coordinate values, and f y A method indicating one of a plurality of variables corresponding to the above frequency component. Claim 9 In claim 5, the step of generating the heatmap data including the pixels each having the temperature value corresponding to the probability of corresponding to at least one abnormal breath sound, based on the convolutional neural network operation of the spectrogram data, comprises: a method including the step of generating the heatmap data from the spectrogram data based on a top-down method. Claim 10 A voice sensor configured to measure a user's breathing and generate breath sound data; a breath sound analysis device configured to generate spectrogram data that visually represents the time and frequency components of at least one abnormal breath sound based on the analysis of the breath sound data, and to generate heatmap data including pixels having temperature values ​​corresponding to the probability of corresponding to the at least one abnormal breath sound based on the convolutional neural network operation of the spectrogram data; a feature variable extraction device configured to generate feature variable information including a plurality of variables corresponding to the at least one abnormal breath sound based on the spectrogram data and the heatmap data; and a differential diagnosis device configured to generate diagnostic information pointing to a target disease having the highest model output value among a plurality of diseases by applying the feature variable information to a pre-trained differential diagnosis model, and to generate diagnostic basis information that quantifies the importance of the plurality of variables used to determine the target disease. The diagnostic device comprises a user interface device configured to output the diagnostic information and the diagnostic basis information, wherein the differential diagnostic device comprises: a diagnostic information generator that receives the feature variable information from the feature variable extraction device, observes a change in the output of the pre-learned differential diagnostic model according to a change in at least one of the plurality of variables of the feature variable information, calculates the importance of the plurality of variables according to the change in the output of the pre-learned differential diagnostic model, and generates the diagnostic information indicating the target disease based on the importance of the plurality of variables; and a diagnostic basis information generator that generates the diagnostic basis information quantifying the importance of the plurality of variables used to determine the target disease. Claim 11 In claim 10, the respiratory sound analysis device comprises: a spectrogram generator that receives respiratory sound data from the voice sensor and generates spectrogram data from the respiratory sound data; and an abnormal respiratory sound detection device that receives spectrogram data from the spectrogram generator and generates heatmap data from the spectrogram data. Claim 12 In claim 11, the feature variable extraction device comprises: a blob area detector that receives the heatmap data from the abnormal breath sound detection device, detects a blob area by filtering the heatmap data based on a threshold temperature value, extracts the coordinates of the blob area from the heatmap data, and generates blob area information pointing to the extracted coordinates; and a feature variable generator that receives the spectrogram data from the spectrogram generator, receives the blob area information from the blob area detector, and generates the feature variable information corresponding to the time component and the frequency component of the at least one abnormal breath sound based on the spectrogram data and the blob area information. Claim 13 In claim 10, the differential diagnosis device comprises: a diagnostic information generator that receives the feature variable information from the feature variable extraction device, observes the change in the output of the pre-learned differential diagnosis model according to the change in at least one of the plurality of variables of the feature variable information, calculates the importance of the plurality of variables according to the change in the output of the pre-learned differential diagnosis model, and generates the diagnostic information indicating the target disease based on the importance of the plurality of variables; and a diagnostic basis information generator that generates the diagnostic basis information quantifying the importance of the plurality of variables used to determine the target disease.

Images