Method and apparatus for diagnosing voice disease on basis of time domain data

Abstract

Disclosed are a method and apparatus for diagnosing a voice disease on the basis of time domain data. According to an aspect of the present disclosure, provided is a diagnostic apparatus comprising: a memory for storing instructions; and at least one processor, wherein the at least one processor is configured to execute the instructions to perform: a process of generating one-dimensional time series data indicating the intensity of a voice at a series of time points from a voice signal of an utterer; and a process of diagnosing a voice disease of the utterer by inputting the one-dimensional time series data to a trained neural network model.

Description

Method and device for diagnosing voice disorders based on time domain data

[0001] The present disclosure relates to a method and device for diagnosing a voice disorder based on time domain data.

[0002] The content described below merely provides background information related to the present embodiment and does not constitute prior art.

[0003] Recently, with the spread of deep learning technology, research is being conducted to distinguish between voice disorders and voice disorders from a speaker's voice using deep learning models.

[0004] One example of deep learning-based voice disease technology involves extracting voice pathology indices (e.g., fundamental frequency, temporal intensity variability, frequency intensity variability, noise-to-overtone ratio, etc.) from speech signals and then using them as input to a deep learning model. In this approach, much of the information contained in the voice is lost during the index extraction process, retaining only information (or statistics) based on prior experience. For example, information inherent in the voice but whose association with voice disease has not yet been identified may be lost.

[0005] Another example of deep learning-based voice disorder technology involves extracting frequency components, such as spectrograms, from a voice signal, visualizing them, and then using them as input for a deep learning model. However, this process of converting a voice signal expressed in the time domain to the frequency domain can result in information loss. Furthermore, while spectrograms are effective in areas such as speech recognition and speaker identification, they are inefficient for direct application in the field of voice disorder diagnosis. Spectrogram-based information extraction, developed to adapt to human auditory perception, focuses on extracting representative components within a voice signal that humans recognize. In contrast, voice disorder diagnosis focuses on components not intended by the speaker, in addition to the representative components. Typical examples include unintended tremor, unintended windy sounds, murmurs of enunciation, and variations and distortions in pronunciation. For these reasons, the process of converting a voice signal to a spectrogram can result in the loss or distortion of information necessary for voice disorder diagnosis.

[0006] Moreover, in the aforementioned methods, the series of processes for extracting features (voice pathology indices or spectrograms) from the speech signal operates independently from the deep learning model's training process and is unaffected by the learning results. In other words, features are extracted based on predefined rules, independent of the training data, which runs counter to the direction pursued by deep learning technology.

[0007] The present disclosure aims to provide a method and device capable of diagnosing a voice disorder by directly inputting a time domain voice signal into a deep learning model without extracting voice pathological parameters from the voice signal or converting it into a frequency domain.

[0008] The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

[0009] According to one aspect of the present disclosure, there is provided a diagnostic device comprising: a memory for storing commands; and at least one processor, wherein the at least one processor executes the commands to generate one-dimensional time series data representing the intensity of a voice at a series of points in time from a voice signal of a speaker; and a process for diagnosing a voice disorder of the speaker by inputting the one-dimensional time series data into a trained neural network model, wherein the neural network model comprises: a feature extraction block for extracting a first feature vector from the one-dimensional time series data using a plurality of first kernels; an abstraction block for extracting a second feature vector from the first feature vector using a plurality of second kernels expanded at different intervals; and a classification block for calculating a probability value for at least one voice disorder from the second feature vector.

[0010] According to another aspect of the present disclosure, a computer-implemented method is provided, comprising: a step of generating one-dimensional time series data representing the intensity of a voice at a series of points in time from a voice signal; a step of inputting the one-dimensional time series data into a trained neural network model to obtain information on at least one voice disorder, wherein the neural network model comprises: a feature extraction block that extracts a first feature vector from the one-dimensional time series data using a plurality of first kernels; an abstraction block that extracts a second feature vector from the first feature vector using a plurality of second kernels expanded at different intervals; and a classification block that calculates a probability value for the at least one voice disorder from the second feature vector.

[0011] According to another aspect of the present disclosure, a computer-implemented method is provided, comprising: a step of generating one-dimensional time series data representing the intensity of a voice at a series of points in time from a voice signal of a speaker; a step of updating parameters of a neural network model based on a loss between an output of a neural network model for the one-dimensional time series data and a voice disorder label corresponding to the voice signal; wherein the neural network model comprises: a feature extraction block for extracting a first feature vector from the one-dimensional time series data using a plurality of first kernels; an abstraction block for extracting a second feature vector from the first feature vector using a plurality of second kernels expanded at different intervals; and a classification block for calculating a probability value for at least one voice disorder from the second feature vector.

[0012] According to another aspect of the present disclosure, a computer-readable recording medium having stored thereon a command, the command causing the computer to execute each process included in the computer implementation method described above when executed by the computer is provided.

[0013] According to embodiments of the present disclosure, voice disorders can be diagnosed by directly inputting a time-domain voice signal into a deep learning model, without extracting voice pathology indices from the voice signal or converting it to the frequency domain. This has the effect of preventing information loss that inevitably occurs during the process of extracting voice pathology indices or spectrograms.

[0014] According to embodiments of the present disclosure, feature extraction, feature abstraction, and classification functions for voice disorder diagnosis can be integrated into a single end-to-end deep learning model, simplifying the system configuration. Furthermore, the organic linking of feature extraction, feature abstraction, and classification tasks can enhance voice disorder diagnosis performance.

[0015] The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

[0016] FIG. 1 is a block diagram schematically showing a diagnostic device according to one embodiment of the present disclosure.

[0017] FIG. 2 is a block diagram schematically illustrating the structure of a neural network model according to one embodiment of the present disclosure.

[0018] FIG. 3 is a schematic diagram of a feature extraction block according to one embodiment of the present disclosure.

[0019] FIGS. 4A to 4C are drawings for reference to explain a short-time Fourier transform process according to a comparative example and the operation of a feature extraction block according to various embodiments of the present disclosure.

[0020] FIG. 5 is a schematic diagram illustrating an abstract block according to one embodiment of the present disclosure.

[0021] FIG. 6 is an exemplary diagram showing various examples of expanded kernels according to one embodiment of the present disclosure.

[0022] FIG. 7 is a drawing referenced to explain the operation of an abstract block according to one embodiment of the present disclosure.

[0023] Figure 8 is a schematic diagram illustrating a classification block according to one embodiment of the present disclosure.

[0024] Figure 9 is an exemplary diagram schematically illustrating the learning process of a neural network model according to one embodiment of the present disclosure.

[0025] Fig. 10 is an exemplary diagram schematically illustrating a method for diagnosing a voice disorder according to one embodiment of the present disclosure.

[0026] FIG. 11 is a block diagram schematically illustrating an exemplary computing device that can be used to implement the devices and methods described in the present disclosure.

[0027] Hereinafter, some embodiments of the present disclosure will be described in detail using exemplary drawings. When designating components in each drawing, it should be noted that, where possible, identical components are given the same reference numerals, even if they appear in different drawings. Furthermore, when describing the present disclosure, detailed descriptions of related known structures or functions will be omitted if they are deemed to obscure the gist of the present disclosure.

[0028] In describing components of embodiments according to the present disclosure, symbols such as first, second, i), ii), a), b) may be used. These symbols are only for distinguishing the components from other components, and the nature, order, or sequence of the components are not limited by the symbols. When a part in the specification is said to "include" or "have" a component, this does not mean that other components are excluded, but rather that other components may be included, unless explicitly stated otherwise.

[0029] The detailed description set forth below, together with the accompanying drawings, is intended to explain exemplary embodiments of the present disclosure and is not intended to represent the only embodiments in which the present disclosure may be practiced.

[0030] FIG. 1 is a block diagram schematically showing a diagnostic device according to one embodiment of the present disclosure.

[0031] The diagnostic device (10) is a device that non-invasively diagnoses voice disorders based on recorded voice. The diagnostic device (10) can diagnose the presence or absence of a voice disorder and / or the type of voice disorder of a speaker from any voice signal using a trained neural network model (140). The voice disorder is a voice disorder or a laryngeal disease that causes the voice disorder, and may include, but is not limited to, vocal fold nodules, vocal fold polyps, laryngeal cancer, and / or spasmodic dysphonia.

[0032] To this end, the diagnostic device (10) may include all or part of the preprocessing unit (100), the diagnostic unit (120), and the neural network model (140). Not all blocks illustrated in FIG. 1 are essential components, and some blocks included in other embodiments may be added, changed, or deleted. Meanwhile, the components illustrated in FIG. 1 represent functionally distinct elements, and at least one of the components may be implemented in an integrated form in an actual physical environment.

[0033] The preprocessing unit (100) can receive a voice signal and perform basic preprocessing. The preprocessing unit (100) can process the voice signal into a form suitable for processing in the neural network model (140). The preprocessing unit (100) can standardize specifications that may vary for each voice signal. For example, the preprocessing unit (100) can resample the voice signal at a predetermined sampling frequency. As another example, the preprocessing unit (100) can apply a band pass filter configured to pass only a predetermined frequency band to the voice signal. As yet another example, the preprocessing unit (100) can perform padding and / or truncation to standardize the voice signal to a predetermined length. For example, the preprocessing unit (100) can resample a voice signal to 22.05 kHz and cut it into 4-second lengths to generate one-dimensional time series data with a length of 88200. The operations performed by the preprocessing unit (100) are not limited to the examples described above, and the preprocessing unit (100) can perform various transformations to standardize audio.

[0034] The preprocessing unit (100) can maintain the expression domain of the input voice signal and the preprocessed voice signal in the time domain. Therefore, like the input voice signal, the preprocessed voice signal can be expressed as one-dimensional time-series data representing the intensity of the voice at a series of points in time.

[0035] The diagnosis unit (120) can diagnose a voice disorder for a speaker of the preprocessed voice signal based on the preprocessed voice signal. The diagnosis unit (120) can input the preprocessed voice signal into a neural network model (140) to obtain information for diagnosing the voice disorder of the speaker. For example, the diagnosis unit (120) can obtain a probability value for at least one type of voice disorder from the neural network model (140). The diagnosis unit (120) can output the probability value for each type of voice disorder as a diagnosis result. As another example, the diagnosis unit (120) can diagnose the type of voice disorder with the highest probability value as the type of voice disorder of the speaker, and output an identifier representing the type and / or its probability value.

[0036] The neural network model (140) is a deep learning model with a neural network structure trained to perform binary classification or multi-classification for at least one voice disorder by receiving a preprocessed voice signal as input. The neural network model (140) may be implemented as a convolutional neural network (CNN) including one or more convolutional layers.

[0037] FIG. 2 is a block diagram schematically illustrating the structure of a neural network model according to one embodiment of the present disclosure.

[0038] The neural network model (140) may include all or part of a feature extraction block (200), at least one abstraction block (220), and a classification block (240).

[0039] The feature extraction block (200) can extract a first feature vector from a preprocessed voice signal, i.e., one-dimensional time series data. The feature extraction block (200) can extract features that express relationships between consecutive elements within a certain time interval from the one-dimensional time series data. The feature extraction block (200) can extract a first feature vector optimized for voice disorder diagnosis.

[0040] The abstraction block (220) can abstract the first feature vector. The abstraction block (220) can extract a second feature vector of a higher level than the first feature vector through abstraction. The abstraction block (220) can extract features that express relationships between elements within a wider time interval than the feature extraction block (200). Depending on the complexity of the data, the abstraction block (220) may be configured in multiple units.

[0041] The classification block (240) can perform binary classification or multi-classification for at least one voice disorder from the second feature vector. For example, the classification block (240) can produce a probability vector composed of probability values ​​corresponding to each of at least one voice disorder.

[0042] FIG. 3 is a schematic diagram of a feature extraction block according to one embodiment of the present disclosure.

[0043] The feature extraction block (200) can extract a first feature vector (32) from one-dimensional time series data (30) using multiple kernels.

[0044] The feature extraction block (200) may include a plurality of convolution layers (300 and 302) corresponding to a plurality of kernels. In order to normalize the scale of the output of the convolution layers (300 and 302) and add non-linearity, the feature extraction block (200) may further include a normalization layer (304) and an activation layer (306) subsequent to the convolution layers (300 and 302). In order to increase the computational efficiency of the neural network, the feature extraction block (200) may further include a pooling layer (320). The normalization layer (304), the activation layer (306), and the pooling layer (302) may be, but are not limited to, a batch normalization layer, a ReLu (Rectified Linear Unit) layer, and a max pooling layer, respectively. Additionally, to prevent overfitting during the training phase, the feature extraction block (200) may further include a dropout layer (340). The dropout layer (304) may, for example, follow the pooling layer (320). The dropout rate may be set to, for example, 0.5, but is not limited thereto.

[0045] In each convolution layer (300 and 302), a one-dimensional convolution operation can be performed between a vector input to the corresponding layer and one or more kernels. The convolution layers (300 and 302) can use kernels with smaller sizes as the layer is located closer to the output layer of the feature extraction block (200).

[0046] Each convolution layer (300 and 302) can produce a single feature value by adding up the element-wise product between the weights of individual kernels and the elements of the input vector corresponding to the position of the kernel. The vector output by each convolution layer (300 and 302) can be defined as a set of feature values ​​produced by moving one or more kernels in the length direction of the input vector by a predetermined stride unit. In some embodiments, the stride can be set in advance based on the size of the kernel. For example, the stride can be set to 1 / 4 of the size of the kernel.

[0047] Each kernel may have trained weights. The trained weights may refer to weights whose values ​​are optimized by learning data and a loss function during the training process. The feature extraction block (200) may be understood as a filter that extracts a first feature vector (32) optimized for voice disorder diagnosis from one-dimensional time series data (30) using weights determined based on learning data.

[0048] Meanwhile, the number of convolution layers (300 and 302) included in the feature extraction block (200), the number of kernels used in each layer, the size of the kernels, the stride, etc. are hyper-parameters set by the user, and can be designed in consideration of the amount of computation and accuracy, etc. Hereinafter, with reference to FIGS. 4a to 4c, an example of the configuration of the feature extraction block (200) for extracting a first feature vector (32) that can replace a spectrogram that is commonly used in the field of speech processing will be described.

[0049] FIGS. 4A to 4C are drawings for reference to explain a short-time Fourier transform process according to a comparative example and the operation of a feature extraction block according to various embodiments of the present disclosure.

[0050] In the field of image processing, convolution operations are mainly performed using kernels with sizes of 3×3 or 5×5. However, unlike images in the form of a two-dimensional array, speech signals are represented as time-series data with a long length in one dimension. For example, a 4-second speech signal with a sampling rate of 22.05 kHz can be represented as one-dimensional data with a length of 88,200. When a speech signal that is spread out in one dimension is analyzed in units of 3 or 5 elements, values ​​close to random walk are observed. Therefore, in order to extract meaningful features from a speech signal, the kernel must be sufficiently large.

[0051] From a similar perspective, the Short Time Fourier Transform (STFT), a technique primarily used in the analysis of time-series data, analyzes data in units of frames (or windows). As illustrated in Figure 4a, the STFT divides the input data (Input) into frames of a fixed size (fr1 and fr2), and applies a window function and a Fast Fourier Transform (FFT) to each frame to extract frequency coefficients for each frame. For example, if the frame size is 1024 and the number of frequency bins from which coefficients are to be extracted from each frame is 512, the STFT can be viewed as a type of filter that extracts 512 features from 1024 pieces of data. Meanwhile, to prevent data discontinuity between adjacent frames, some areas of adjacent frames may overlap. The movement interval between adjacent frames may be referred to as the hop size (n_hops). The hop size can be set to, for example, 256, which is one-fourth of the frame size.

[0052] Figure 4b shows an example of extending the above-described STFT concept to a convolutional layer. The convolutional layer can act as a filter that extracts features corresponding to the number of kernels from a receptive field corresponding to the kernel size. That is, the kernel size and the number of kernels can correspond to the frame size and the number of frequency bins in the STFT, respectively. For example, the convolutional layer can perform a convolution operation between 512 kernels of size 1024 and one-dimensional time series data (30), thereby extracting 512 features for every 1024 data. Meanwhile, the stride, which represents the movement width of the kernel, can be conceptualized as the same as the hop size of the STFT. Therefore, the stride can be set to 256, which is 1 / 4 of the kernel size.

[0053] The convolution operation illustrated in Figure 4b analyzes the same data region as the STFT illustrated in Figure 4a, extracting the same number of features. Furthermore, unlike the STFT, which uses fixed filters regardless of the application, the convolution layer utilizes kernels with weights determined through learning, allowing it to extract features that better represent the information necessary for voice disorder diagnosis than the STFT. However, in deep learning algorithms, if the kernel size used in a single layer becomes too large, the complexity rapidly increases, and learning becomes difficult due to the curse of dimensionality.

[0054] To solve these problems, the present disclosure divides a kernel having a large size into multiple kernels having smaller sizes, and utilizes the divided kernels in multiple layers.

[0055] Fig. 4c shows an example of dividing one convolution layer illustrated in Fig. 4b into two convolution layers (300 and 302) and a pooling layer (320). Comparing Figs. 4b and 4c, instead of using 512 kernels of size 1024 in one convolution layer, kernels of sizes 100, 5, and 2 can be used in two convolution layers (300 and 302) and a pooling layer (320), respectively. For example, in the first convolution layer (300), a convolution operation may be performed between each of 512 kernels with a size of 100 and one-dimensional time series data (30) to extract a first output vector with 512 channels, and in the second convolution layer (302), a convolution operation may be performed between each of 512 kernels with a size and depth of 5×512 and the first output vector to extract a second output vector with 512 channels. In the pooling layer (320), a max pooling operation with a kernel size of 2 may be performed on the second output vector to condense two adjacent data within the second output vector into one. In the example of Fig. 4c, the feature extraction block (200) can theoretically extract 512 features from 100 × 5 × 2 = 1000 pieces of data, and thus can be assumed to be a filter that operates similarly to the STFT of Fig. 4a and one convolutional layer of Fig. 4b.

[0056] Depending on the implementation example, the feature extraction block (200) may include a greater number of convolutional layers. However, as the number of layers increases, complexity increases, which may hinder learning. Therefore, the number of layers should be appropriately adjusted. Preferably, the feature extraction block (200) may include two or three convolutional layers.

[0057] FIG. 5 is a schematic diagram illustrating an abstract block according to one embodiment of the present disclosure.

[0058] The abstraction block (220) can extract a second feature vector (50) from a first feature vector (32) by using a plurality of kernels expanded at different intervals.

[0059] The abstraction block (220) may include a plurality of layer groups (500, 510, 520, and 530) corresponding to a plurality of kernels. The plurality of layer groups (500, 510, 520, and 530) may be sequentially connected. Each layer group (500, 510, 520, and 530) may include a dilated convolution layer (502, 512, 522, and 532) that performs a one-dimensional convolution operation using one or more kernels that are dilated at regular intervals. The kernels used in a specific layer group may be dilated at the same interval. Each layer group (500, 510, 520, and 530) may further include a normalization layer (504, 514, 524, and 534) and an activation layer (506, 516, 526, and 536) following the dilated convolution layers (502, 512, 522, and 532). The normalization layers (504, 514, 524, and 534) and the activation layers (506, 516, 526, and 536) may be, but are not limited to, a batch normalization layer and a ReLu layer, respectively. The abstraction block (220) may further include a pooling layer (540) and a dropout layer (550). The pooling layer (540) and the dropout layer (550) may be successive to the last layer group (530) among the multiple layer groups (500, 510, 520, and 530). The kernel size of the pooling layer (540) and the dropout ratio of the dropout layer (550) may be set to, for example, 2 and 0.5, but are not limited thereto.

[0060] As mentioned above, speech, a one-dimensional time-series data, is represented by a large amount of data stretched out. Given this characteristic, understanding speech characteristics requires an abstraction method that can simultaneously examine a wide range of categories. Typically, increasing the kernel size or performing downsampling can be considered to expand the receptive field in convolutional neural networks. However, increasing the kernel size increases the complexity of the neural network, as mentioned above, making training more difficult. Furthermore, it also increases the amount of data required for training, making it unsuitable for applications in the medical field, where training data is limited. While downsampling effectively reduces the data length, it also carries the risk of losing fine-grained information. This makes downsampling difficult to apply to applications impacted by speech quality. This is especially true for applications such as voice disorder diagnosis, which require detailed analysis of speech.

[0061] To expand the receptive area without using the aforementioned methods, the present disclosure utilizes dilated convolution layers (502, 512, 522 and 532) that perform one-dimensional convolution operations using dilated kernels.

[0062] FIG. 6 is an exemplary diagram showing various examples of expanded kernels according to one embodiment of the present disclosure.

[0063] A dilated kernel refers to a kernel in which the weights within the kernel are spaced at regular intervals. The spacing between the weights within the kernel can be referred to as the dilation rate. In Fig. 6, elements of the kernel whose values ​​are not fixed to zero, i.e. elements with trained (or trainable) weights, are indicated with hatching.

[0064] Figure 6 (a) shows a kernel with an expansion ratio of 1. A kernel with an expansion ratio of 1 can operate in the same way as a kernel applied to a general convolution. Figures 6 (b) to (d) show kernels with expansion ratios of 2, 4, and 8, respectively. In a kernel with an expansion ratio of 2, one element with a value of 0 can be inserted between the weights. In a kernel with an expansion ratio of 4, three elements with a value of 0 can be inserted between the weights. Similarly, in a kernel with an expansion ratio of 8, seven elements with a value of 0 can be inserted between the weights. In this way, adding zero padding to a kernel can expand the receptive region of the kernel while keeping the amount of computation constant.

[0065] Meanwhile, in Fig. 6, the size of each kernel (i.e., the number of weights) is illustrated as 2 and the number of channels is 1; however, this is for convenience of explanation and the present disclosure is not limited thereto. That is, it should be noted that kernels may have a greater number of weights and / or a greater number of channels.

[0066] FIG. 7 is a drawing referenced to explain the operation of an abstract block according to one embodiment of the present disclosure.

[0067] Multiple layer groups (500, 510, 520, and 530) can perform convolution operations using kernels with larger dilation ratios in lower layer groups closer to the output layer. This may correspond to the concept of examining data in detail in upper layer groups closer to the input layer, while examining a wider range in lower layer groups closer to the output layer. As one moves from upper layer groups to lower layer groups, the dilation ratio of the kernel applied in each layer group may increase exponentially. For example, a first layer group (500) may include a first dilated convolution layer (502) that performs a convolution operation using a kernel with an expansion ratio of 1, a second layer group (510) may include a second dilated convolution layer (512) that performs a convolution operation using a kernel with an expansion ratio of 2, a third layer group (520) may include a third dilated convolution layer (522) that performs a convolution operation using a kernel with an expansion ratio of 4, and a fourth layer group (530) may include a fourth dilated convolution layer (532) that performs a convolution operation using a kernel with an expansion ratio of 8.

[0068] A convolution operation using an expanded kernel may include performing a weighted sum on elements spaced at intervals equal to the expansion ratio of the kernel within the input feature vector. For example, the first element of the first output vector (70) extracted from the first layer group (500) may be the result of applying a normalization and / or activation function to the weighted sum between the first element and the second element of the first feature vector (32) input to the layer group (500). Similarly, the first element of the second output vector (72) extracted from the second layer group (510) may be generated based on the weighted sum between the first element and the third element of the first output vector (70). The first element of the third output vector (74) extracted from the third layer group (520) may be generated based on the weighted sum between the first element and the fifth element of the second output vector (72). The first element of the fourth output vector (76) extracted from the fourth layer group (530) can be generated based on the weighted sum between the first element and the ninth element of the third output vector (74). Meanwhile, in the above-described example, the elements skipped during the weighted sum operation are used to extract feature values ​​at other locations. For example, the second element of the first output vector (70) can be weighted-summed with the fourth element and used to determine the second element of the second output vector (72).

[0069] Figure 7 shows the first element of the fourth output vector (76) and the elements of other feature vectors (32 and 70-74) considered in generating it, marked with hatching. As illustrated in Figure 7, the abstraction block (220) can have a receptive area that covers a wide range while viewing the first feature vector (32) in detail.

[0070] Meanwhile, the number of kernels applied in each dilated convolution layer (502, 51, 522, and 524) may be fixed or adjusted within a limited range. For example, the first to fourth dilated convolution layers (502, 512, 522, and 524) may each include 256 kernels. That is, the number of channels of the feature vectors (70 to 76) output by the first to fourth dilated convolution layers (502, 512, 522, and 524) may be maintained at a constant 256. This is in contrast to performing abstraction by increasing the number of channels in a general convolutional neural network. As another example, the number of channels of the feature vectors (70 to 76) may be adjusted within a range of about 0.5 to 2 times.

[0071] As described above, the abstraction block (220) utilizes multiple kernels with various expansion rates to abstract features while simultaneously considering a broad range of data while viewing a portion of the data in detail. Furthermore, the abstraction block (220) can extract information that is as easily interpretable as possible within the given representational framework while fixing the number of information to be expressed (i.e., the number of channels). The processes performed in the abstraction block (220) can also be viewed as a simplification or understanding process that analyzes the features of the speech signal in detail while also examining a broad range to understand the big picture.

[0072] Figure 8 is a schematic diagram illustrating a classification block according to one embodiment of the present disclosure.

[0073] The classification block (240) may include all or part of a global average pooling (GAP) layer (800), a flatten layer (810), one or more linear layers (820 and 840), and a softmax layer (850).

[0074] In the global average pooling layer (800), an operation is performed to calculate an average for feature values ​​within the same channel. For example, when the number of channels of the second feature vector (50) is C2, the global average layer (800) can receive C2 one-dimensional vectors, each with a size of N, and output C2 one-dimensional vectors, each with a size of 1. In the flatten layer (810), the input data can be converted into one one-dimensional data. For example, the flatten layer (810) can convert C2 one-dimensional vectors, each with a size of 1, into one one-dimensional vector, each with a size of C2.

[0075] In the linear layers (820 and 840), a weighted-sum operation is performed on all elements of the input vector. The linear layers (820 and 840) may also be referred to as fully connected (FC) layers or dense layers. In order to add non-linearity to the outputs of the linear layers (820 and 840), the classification block (240) may further include activation layers (822 and 842) subsequent to the linear layers (820 and 840). In the activation layers (82 and 842), a Gaussian error linear unit (GeLu) function may be applied as an activation function, but is not limited thereto.

[0076] In the softmax layer (850), an operation is performed to convert the input vector into a probability vector (80) in which each element has a value between 0 and 1 and the total sum of the elements is 1. The elements of the probability vector (80) can represent a probability value for each of at least one voice disorder to be classified.

[0077] To prevent overfitting during the training phase, the classification block (240) may further include a dropout layer (830). The dropout layer (830) may be positioned, for example, between the linear layers (820 and 840). The dropout ratio may be set to, for example, 0.5, but is not limited thereto.

[0078] FIG. 9 is an exemplary diagram schematically illustrating the learning process of a neural network model according to one embodiment of the present disclosure.

[0079] The learning process of a neural network model according to one embodiment of the present disclosure is executed by a learning device, which may be executed on a computing device. The learning device may perform each function by one or more processors available to the computing device, and may include computer-readable storage connected to such processors and having instructions stored therein, but is not limited thereto.

[0080] A learning device can collect voice signals from various speakers. The collected voice signals can be assigned labels indicating the presence or absence of a voice disorder and / or the type of voice disorder of the speaker. The learning device can use the labels to separate the entire dataset into a training dataset, a validation dataset, and a test dataset. For example, the learning device can separate the training dataset, the validation dataset, and the test dataset so that the distribution of the presence or absence of a voice disorder and / or the type of voice disorder in each dataset is similar. Depending on the implementation example, the learning device may additionally collect basic information, such as the gender and / or age of the speakers. The learning device can further separate the entire dataset using the basic information of the speakers. For example, the learning device can separate the dataset so that the separated datasets have similar class distributions, gender distributions, and age distributions.

[0081] The learning device can fit the neural network model (140) to a training dataset. The learning device can train the neural network model (140) to classify the speaker's voice disorder label from the preprocessed voice signal. For example, the learning device can preprocess the voice signal in the training dataset, input the preprocessed voice signal into the neural network model (140), and calculate a loss function based on the probability vector output by the neural network model (140) and the voice disorder label assigned to the corresponding voice signal. Cross entropy may be used as the loss function, but is not limited thereto. The learning device can update the parameters of the neural network model (140) in a direction in which the loss function is minimized. For example, the learning device can update the weights of the layers included in individual blocks in the order of the classification block (240), the abstraction block (220), and the feature extraction block (200) based on a backpropagation algorithm. Meanwhile, since the preprocessing performed by the learning device may be identical to or corresponding to the operation of the preprocessing unit (100) described above, details thereof will be omitted. During the training process, the learning device can verify the neural network model (140) using a verification dataset. For example, the learning device can perform preprocessing on a voice signal in the verification dataset, input the preprocessed signal into the neural network model (140), and verify the neural network model (140) based on the probability vector output by the neural network model (140) and the voice disorder label assigned to the corresponding voice signal. The learning device can adjust (or select) the hyperparameters of the neural network model (140) based on the verification result.

[0082] The learning device can evaluate the performance of the trained neural network model (140) using a test dataset. If the performance of the trained neural network model (140) satisfies the required conditions, the learning device can store the trained neural network model (140) and distribute it to another device (e.g., a diagnostic device (10)).

[0083] Fig. 10 is an exemplary diagram schematically illustrating a method for diagnosing a voice disorder according to one embodiment of the present disclosure.

[0084] The diagnostic device (10) can generate one-dimensional time series data representing the intensity of a voice at a series of points in time from a speaker's voice signal (S1000). For example, the diagnostic device (10) can perform one or more of resampling, bandpass filtering, padding, and truncation on the voice signal.

[0085] The diagnostic device (10) can input one-dimensional time series data into a trained neural network model (140) to obtain information on at least one voice disorder (S1020). The neural network model (140) may be an end-to-end model including a feature extraction block (200), an abstraction block (220), and a classification block (240). That is, the feature extraction block (200), the abstraction block (220), and the classification block (240) are included in a single neural network model (140) and can be trained end-to-end based on a plurality of training samples consisting of pairs of a time domain voice signal and a voice disorder label.

[0086] In process S1020, the diagnostic device (10) may extract a first feature vector from one-dimensional time series data using a plurality of first kernels. The diagnostic device (10) may hierarchically perform a convolution operation with a plurality of first kernels on the one-dimensional time series data. The plurality of first kernels may have different sizes. For example, the diagnostic device (10) may perform a convolution operation between one-dimensional time series data and one or more 1-1 kernels having a first size, and then perform a convolution operation between a feature vector produced as a result of the convolution operation and one or more 1-2 kernels having a second size. The first kernel (e.g., the 1-1 kernel) applied to a preceding convolution operation may have a larger size. The plurality of first kernels may correspond to a plurality of convolution layers of the feature extraction block (200). The plurality of convolution layers may be sequentially connected.

[0087] In process S1020, the diagnostic device (10) can extract a second feature vector from the first feature vector by using a plurality of second kernels expanded with different intervals. The diagnostic device (10) can hierarchically perform a convolution operation with the plurality of second kernels on the first feature vector. For example, the diagnostic device (10) can perform a convolution operation between the first feature vector and one or more 2-1 kernels expanded with a first interval, and then perform a convolution operation between the feature vector (or the normalization / activation result thereof) produced as a result of the convolution operation and one or more 2-2 kernels expanded with a second interval. The second kernel (e.g., the 2-2 kernel) applied to the subsequent convolution operation may be expanded with a wider interval. The plurality of second kernels may correspond to a plurality of expanded convolution layers of the abstraction block (220). The abstraction block (220) may include a plurality of layer groups, each of which includes a dilated convolution layer, a normalization layer, and an activation layer, each corresponding to one of a plurality of second kernels. The plurality of layer groups may be sequentially connected.

[0088] In process S1020, the diagnostic device (10) can calculate a probability value for at least one voice disease from the second feature vector.

[0089] FIG. 11 is a block diagram schematically illustrating an exemplary computing device that can be used to implement the devices and methods described in the present disclosure.

[0090] The computing device (11) may include some or all of a memory (1100), a processor (1120), storage (1140), an input / output interface (1160), and a communication interface (1180). The computing device (11) may structurally and / or functionally include at least a part of the diagnostic device (10) and / or the learning device. The computing device (11) may be a stationary computing device such as a desktop computer, a server, etc., as well as a mobile computing device such as a laptop computer, a smart phone, etc. The computing device (11) may be implemented with any specialized hardware accelerator capable of efficiently processing operations for an artificial intelligence model. For example, the computing device (11) may include a graphic processing unit (GPU), a tensor processing unit (TPU), or a neural processing unit (NPU).

[0091] The memory (1100) may store a program that causes the processor (1120) to perform a method or operation according to various embodiments of the present disclosure. For example, the program may include a plurality of instructions executable by the processor (1120), and the method illustrated in FIG. 10 may be performed by executing the plurality of instructions by the processor (1120). The memory (1100) may be a single memory or a plurality of memories. In this case, information required to perform the method or operation according to various embodiments of the present disclosure may be stored in a single memory or may be divided and stored in the plurality of memories. When the memory (1100) is composed of a plurality of memories, the plurality of memories may be physically separated. The memory (1100) may include at least one of a volatile memory and a non-volatile memory. The volatile memory includes a static random access memory (SRAM) or a dynamic random access memory (DRAM), and the non-volatile memory includes a flash memory.

[0092] The processor (1120) may include at least one core capable of executing at least one instruction. The processor (1120) may execute instructions stored in the memory (1100). The processor (1120) may be a single processor or multiple processors.

[0093] Storage (1140) maintains stored data even when power supplied to the computing device (11) is cut off. For example, storage (1140) may include non-volatile memory, or may include storage media such as magnetic tape, optical disk, or magnetic disk. A program stored in storage (1140) may be loaded into memory (1100) before being executed by processor (1120). Storage (1140) may store a file written in a programming language, and a program generated from the file by a compiler or the like may be loaded into memory (1100). Storage (1140) may store data to be processed by processor (1120) and / or data processed by processor (1120). For example, storage (1140) may store trained weights of a neural network model (140).

[0094] The input / output interface (1160) may include input devices such as a microphone, a keyboard, a mouse, etc., and may include output devices such as a display device, a printer, etc. The user may trigger the execution of a program by the processor (1120) and / or check the processing result of the processor (1120) through the input / output interface (1160). For example, the computing device (11) may receive a voice signal for diagnosing a voice disorder through a microphone, and may provide the diagnosis result to the user through a display device.

[0095] The communication interface (1180) may provide access to an external network. The computing device (11) may communicate with other devices via the communication interface (1180). For example, the computing device (11) may distribute a trained neural network model or receive a voice signal from another device via the communication interface (1180).

[0096] Each component of the device or method according to the present invention may be implemented in hardware, software, or a combination of hardware and software. Furthermore, the functions of each component may be implemented in software, with a microprocessor executing the software functions corresponding to each component.

[0097] Various implementations of the systems and techniques described herein may be implemented as digital electronic circuits, integrated circuits, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations of one or more computer programs executable on a programmable system. The programmable system includes at least one programmable processor (which may be a special purpose processor or a general purpose processor) coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device. Computer programs (also known as programs, software, software applications, or code) include instructions for the programmable processor and are stored on a "computer-readable recording medium."

[0098] A computer-readable recording medium includes any type of recording device that stores data that can be read by a computer system. Such a computer-readable recording medium may be a non-volatile or non-transitory medium such as a ROM, CD-ROM, magnetic tape, floppy disk, memory card, hard disk, magneto-optical disk, storage device, and may further include a transitory medium such as a data transmission medium. Furthermore, the computer-readable recording medium may be distributed across network-connected computer systems, so that computer-readable code can be stored and executed in a distributed manner.

[0099] Although the flowchart / timing diagram of this specification describes each process as being executed sequentially, this is merely an illustrative description of the technical idea of ​​one embodiment of the present disclosure. In other words, a person of ordinary skill in the art to which one embodiment of the present disclosure belongs may modify and apply various modifications and variations by changing the order described in the flowchart / timing diagram without departing from the essential characteristics of one embodiment of the present disclosure, or by executing one or more of the processes in parallel. Therefore, the flowchart / timing diagram is not limited to a chronological order.

[0100] The above description is merely an example of the technical idea of ​​the present embodiment, and those skilled in the art will appreciate that various modifications and variations can be made without departing from the essential characteristics of the present embodiment. Therefore, the present embodiments are not intended to limit the technical idea of ​​the present embodiment, but rather to explain it, and the scope of the technical idea of ​​the present embodiment is not limited by these embodiments. The scope of protection of the present embodiment should be interpreted by the claims below, and all technical ideas within a scope equivalent thereto should be interpreted as being included in the scope of rights of the present embodiment.

[0101]

[0102] CROSS-REFERENCE TO RELATED APPLICATION

[0103] This patent application claims priority to Korean Patent Application No. 10-2023-0093367, filed July 18, 2023, which is incorporated herein by reference in its entirety.

Claims

1. A memory for storing instructions; and at least one processor, At least one processor executes the instructions, A process of generating one-dimensional time series data representing the intensity of a voice at a series of points in time from a speaker's voice signal; and The above one-dimensional time series data is input into a trained neural network model, and a process of diagnosing the speaker's voice disorder is performed. The above neural network model is, A feature extraction block that extracts a first feature vector from the one-dimensional time series data using a plurality of first kernels; An abstraction block that extracts a second feature vector from the first feature vector using a plurality of second kernels expanded at different intervals; and A diagnostic device characterized by including a classification block that calculates a probability value for at least one voice disorder from the second feature vector.

2. In paragraph 1, The above abstraction block includes a plurality of dilated convolution layers corresponding to the plurality of second kernels, A diagnostic device in which the plurality of second kernels are dilated at wider intervals the closer the corresponding dilated convolution layer is located to the output layer of the abstraction block.

3. In paragraph 1, The above abstract block includes a plurality of layer groups that are sequentially connected, A diagnostic device, wherein each of the plurality of layer groups includes a dilated convolution layer, a normalization layer, and an activation layer corresponding to any one of the plurality of second kernels.

4. In paragraph 1, The above feature extraction block includes a plurality of convolution layers corresponding to the plurality of first kernels, A diagnostic device wherein the plurality of first kernels have a smaller size the closer the corresponding convolutional layer is located to the output layer of the feature extraction block.

5. In paragraph 1, The above neural network model is, A diagnostic device that is trained end-to-end based on a plurality of training samples consisting of pairs of time-domain speech signals and speech disorder labels.

6. In paragraph 1, The above generating process is, A diagnostic device comprising a process of performing at least one of re-sampling, band pass filtering, padding, and truncation on the above voice signal.

7. As a computer implementation method, A process of generating one-dimensional time series data representing the intensity of a voice at a series of points in time from a voice signal; A process of inputting the above one-dimensional time series data into a trained neural network model to obtain information on at least one vocal disorder, The above neural network model is, A feature extraction block that extracts a first feature vector from the one-dimensional time series data using a plurality of first kernels; An abstraction block that extracts a second feature vector from the first feature vector using a plurality of second kernels expanded at different intervals; and A computer-implemented method, characterized in that it includes a classification block that calculates a probability value for at least one voice disorder from the second feature vector.

8. In paragraph 7, The above abstraction block includes a plurality of dilated convolution layers corresponding to the plurality of second kernels, A computer-implemented method, wherein the plurality of second kernels are dilated with a wider interval as the corresponding dilated convolution layer is located closer to the output layer of the abstraction block.

9. In paragraph 7, The above abstract block includes a plurality of layer groups that are sequentially connected, A computer-implemented method, wherein each of the plurality of layer groups includes a dilated convolution layer, a normalization layer, and an activation layer corresponding to any one of the plurality of second kernels.

10. In paragraph 7, The above feature extraction block includes a plurality of convolution layers corresponding to the plurality of first kernels, A computer-implemented method, wherein the plurality of first kernels have a smaller size the closer the corresponding convolutional layer is located to the output layer of the feature extraction block.

11. In paragraph 7, The above neural network model is, A computer-implemented method, wherein the method is trained end-to-end based on a plurality of training samples, each of which consists of a pair of a time-domain speech signal and a speech disorder label.

12. In paragraph 7, The above generating process is, A computer-implemented method comprising a process of performing at least one of re-sampling, band pass filtering, padding, and truncation on the above-mentioned voice signal.

13. A computer-readable recording medium having stored thereon a command, wherein the command, when executed by the computer, causes the computer to execute each process included in the computer implementation method according to claim 7.

14. As a computer implementation method, A process of generating one-dimensional time series data representing the intensity of a speaker's voice at a series of points in time from the speaker's voice signal; Including a process of updating the parameters of the neural network model based on the loss between the output of the neural network model for the one-dimensional time series data and the voice disease label corresponding to the voice signal, The above neural network model comprises a feature extraction block that extracts a first feature vector from the one-dimensional time series data using a plurality of first kernels; An abstraction block that extracts a second feature vector from the first feature vector using a plurality of second kernels expanded at different intervals; and A computer-implemented method, characterized in that it includes a classification block that calculates a probability value for at least one voice disorder from the second feature vector.

15. A computer-readable recording medium having stored thereon a command, wherein the command, when executed by the computer, causes the computer to execute each process included in the computer implementation method according to claim 14.

Images