A method and system for detecting dysphagia patients based on tensor decomposition

Abstract

The application provides a swallowing disorder patient detection method and system based on tensor decomposition, wherein the method comprises the following steps: obtaining speech data of swallowing disorder patients and normal persons, pre-processing the speech data, cutting off irrelevant information in the speech data and performing amplitude normalization operation; obtaining a spectrogram of the speech data, and constructing mixed tensors of different forms of vowels; sending the mixed tensors of different forms of vowels into a Tucker decomposition model with added orthogonal constraints, setting a proper rank to obtain a corresponding core tensor; unfolding the core tensor into a vector as a feature input, constructing a training feature set after feature selection, and training a swallowing disorder detection model; detecting to-be-detected speech data based on the detection model to obtain classification results of the swallowing disorder patients and the normal persons. The application is implemented based on speech, is simple to operate, has low cost, has no technical threshold during use, can safely and effectively detect swallowing disorder patients, and is convenient for follow-up.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of dysphagia classification, in particular to a dysphagia patient detection method and system based on tensor decomposition, and also relates to a corresponding computer terminal and computer readable storage medium. BACKGROUND

[0002] Dysphagia refers to the symptoms of swallowing dysfunction occurring between the oral cavity and the stomach, which can be caused by diseases that cause damage to organs or physiological conditions, such as stroke and Alzheimer's disease. Dysphagia affects the quality of life of patients and can lead to aspiration pneumonia, malnutrition, and even endanger the lives of patients. In clinical practice, swallowing radiography (VFSS) and fiber-optic endoscopic swallowing function test (FEES) are generally used as the gold standard for evaluating dysphagia. These evaluation methods require invasion of the body, causing discomfort and radiation exposure to the patient, and the operation requires a professional doctor. Speech analysis is non-invasive and easy to collect cost-effectively. In recent years, many studies have shown that speech data can produce swallowing features that help identify clinical pathological patterns. Implementing dysphagia patient detection based on speech analysis is a safe, convenient and effective way.

[0003] In recent years, machine learning has shown excellent performance in tasks related to classification, which has promoted the development of automatic dysphagia detection technology. Many recent studies collect speech data and propose automatic classification of dysphagia patients based on machine learning. Common speech protocols include swallowing bolus sounds, but this method has safety risks and increases the risk of choking. In contrast, continuous vowel speech signals are easier to collect and more secure, and have an advantage in detecting dysphagia. However, current research either only uses a single vowel speech signal or does not fully utilize the potential correlation between different vowels. Therefore, how to efficiently utilize effective features in vowel signals for dysphagia patient detection is a hot topic of research.

[0004] The prior art mostly uses classic speech features for dysphagia classification. The spectrogram obtained from the speech signal also contains many speech features, such as the flicker and jitter features of the subject's voice in the STFT spectrogram, and the fundamental frequency information in the MFCC spectrogram, which are all helpful in distinguishing between dysphagia patients and normal people. Although these features are used in current work to assist in screening, few studies have jointly explored key features representing dysphagia patients. SUMMARY

[0005] The present application provides a dysphagia patient detection method and system based on tensor decomposition, and also provides a corresponding computer terminal and computer readable storage medium.

[0006] According to an aspect of the present application, there is provided a swallowing disorder patient detection method based on tensor decomposition, comprising:

[0007] Respectively acquiring voice data of swallowing disorder patients and normal persons, the voice data containing any one or any multiple of vowel data;

[0008] Pretreating the voice data, cutting off irrelevant information in the voice data and performing amplitude normalization operation;

[0009] Acquiring a spectrogram of the voice data, and constructing different forms of vowel mixed tensors;

[0010] Sending the different forms of vowel mixed tensors into a Tucker decomposition model with added orthogonal constraints, and setting a proper rank to obtain a corresponding core tensor;

[0011] Unfolding the core tensor into a vector as a feature input, and constructing a training feature set after feature selection to train a swallowing disorder detection model;

[0012] Detecting voice data to be detected based on the swallowing disorder detection model to obtain classification results of swallowing disorder patients and normal persons.

[0013] Preferably, the acquiring of the spectrogram of the voice data comprises:

[0014] Acquiring an STFT spectrogram, comprising:

[0015] Performing short-time Fourier transform on the voice data to convert a one-dimensional voice signal into a two-dimensional STFT spectrogram;

[0016] Acquiring an MFCC spectrogram, comprising:

[0017] Mapping the STFT spectrogram onto a mel frequency scale to form a mel spectrum;

[0018] Taking a logarithm of the mel spectrum, performing discrete cosine transform to obtain MFCC features, converting a one-dimensional voice signal into a two-dimensional matrix containing time-varying MFCC coefficients to obtain an MFCC spectrogram;

[0019] The constructing of different forms of vowel mixed tensors comprises:

[0020] Constructing a vowel mixed tensor based on an MFCC spectrogram, comprising:

[0021] For each subject, MFCC spectrograms are respectively calculated from vowel data, and then the obtained MFCC spectrograms are constructed into a three-dimensional tensor Wherein, I m represents the number of vowel data used.m represents the number of MFCC feature vectors; K m represents the time dimension, i.e. the total number of frames;

[0022] constructing vowel mixing tensors based on the whole STFT spectrogram, including:

[0023] obtaining STFT spectrograms for each subject respectively, and then constructing the obtained STFT spectrograms into a three-dimensional tensor wherein, I s represents the number of vowel data used; J s represents the total number of frequency components in the frequency range; K s represents the time dimension, i.e. the total number of frames;

[0024] constructing vowel mixing tensors based on different frequency bands of the STFT spectrogram, including:

[0025] after constructing the vowel mixing tensor S for each subject based on the whole STFT spectrogram, dividing it into n sub-tensors B in the frequency dimension j , j = 1,..., n, representing n frequency band sub-tensors.

[0026] Preferably, the different forms of vowel mixing tensors are fed into a Tucker decomposition model with orthogonal constraints, and a suitable rank is set to obtain the corresponding core tensor, including:

[0027] constructing a Tucker decomposition model with orthogonal constraints as:

[0028]

[0029] wherein, X ∈ R I×J×K is a three-dimensional tensor, representing a unified designation of three forms of vowel mixing tensors, R represents a matrix, I, J, K represent the sizes of the three dimensions of the three-dimensional tensor X respectively; G ∈ R P×Q×L represents the core tensor obtained after Tucker decomposition of the vowel mixing tensor, containing the mutual relationship between different modes, wherein P, Q, L represent the sizes of the three dimensions of the core tensor G respectively; A ∈ R I×P , B ∈ R J×Q , C ∈ R K×L respectively represent the factor matrices in the three modes obtained after Tucker decomposition, wherein ×1, ×2, ×3 represent the product between the tensor and the matrix; g pql , a p , b q , c l respectively represent the specific elements of G, A, B, C; ○ represents the vector outer product;

[0030] The optimal rank setting in the interval is obtained by searching in a certain interval, that is, the classification accuracy is optimal in the set rank interval, and then the final kernel tensor is obtained.

[0031] Preferably, the flattening of the kernel tensor into a vector as a feature input comprises:

[0032] The corresponding kernel tensors obtained by mixing different forms of vowel tensors are respectively flattened into vectors to obtain corresponding feature sets as feature inputs.

[0033] Preferably, the feature selection comprises:

[0034] Firstly, coarse feature selection is performed by t-test method, and the obtained coarse feature training feature set is used to distinguish dysphagia patients and healthy people according to the overall distribution;

[0035] Secondly, fine features are obtained by precise matching analysis, specifically, by calculating the features of each range according to the distribution of the original data, adjusting the threshold setting to screen the features, and the obtained fine feature training feature set is used to reduce the influence of unbalanced data.

[0036] Preferably, the construction of the training feature set comprises:

[0037] The feature sets obtained based on different forms of vowel mixing tensors are respectively taken as the first training feature set;

[0038] The feature sets obtained based on the vowel mixing tensors constructed based on the STFT spectrogram and the vowel mixing tensors constructed based on the MFCC spectrogram are spliced and fused to obtain the second training feature set;

[0039] The multiple feature sets obtained based on the vowel mixing tensors constructed based on the STFT spectrogram in different frequency bands are respectively spliced and fused with the feature set obtained based on the vowel mixing tensors of the MFCC spectrogram to obtain the third training feature set;

[0040] The multiple feature sets obtained based on the vowel mixing tensors constructed based on the STFT spectrogram in different frequency bands are spliced and fused to obtain the fourth training feature set;

[0041] The fourth training feature set and the feature set obtained based on the vowel mixing tensors constructed based on the MFCC spectrogram are spliced and fused to obtain the fifth training feature set.

[0042] Preferably, the training of the swallowing disorder detection model comprises:

[0043] A support vector machine is provided as an initial model;

[0044] The initial model is trained using the training feature set and data labeling to obtain a swallowing disorder detection model.

[0045] According to another aspect of the present application, there is provided a swallowing disorder patient detection system based on tensor decomposition, comprising:

[0046] a data acquisition module configured to acquire voice data of swallowing disorder patients and normal persons, respectively, the voice data comprising any one or any plurality of vowel data;

[0047] a data processing module configured to pre-process the voice data, crop irrelevant information in the voice data and perform amplitude normalization operation;

[0048] a training data construction module configured to acquire a spectrogram of the voice data, construct mixed tensors of different forms of vowels, send the mixed tensors of different forms of vowels into a Tucker decomposition model with added orthogonal constraint, set a proper rank to obtain a corresponding core tensor, flatten the core tensor into a vector as a feature input, construct a training feature set after feature selection;

[0049] a swallowing disorder detection module configured to train a swallowing disorder detection model using the training feature set, detect voice data to be detected based on the swallowing disorder detection model, and obtain classification results of swallowing disorder patients and normal persons.

[0050] According to a third aspect of the present application, there is provided a computer terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor is configured to execute the computer program to implement the method according to any one of the preceding aspects of the present application, or to run the system according to the preceding aspects of the present application.

[0051] According to a fourth aspect of the present application, there is provided a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executable by a processor to implement the method according to any one of the preceding aspects of the present application, or to run the system according to the preceding aspects of the present application.

[0052] Thanks to the above technical solutions, the present application has at least one of the following beneficial effects compared with the prior art:

[0053] The present application can automatically detect swallowing disorder patients from input vowel voice data without human intervention, output detection results to assist medical personnel in diagnosis and treatment and subsequent follow-up, and save labor costs to a certain extent.

[0054] The application constructs three different forms of vowel mixed tensors based on two kinds of spectrograms, can take advantage of the advantage that high-order tensors can retain the interaction information between different vowels, capture multiple features that help detect dysphagia patients, greatly improve the accuracy of detection, and better represent the differences between dysphagia patients and normal people in speech expression.

[0055] The Tucker decomposition adopted by the application belongs to an effective mechanism for decomposing high-order tensors, and compared with unconstrained Tucker decomposition, orthogonal Tucker decomposition can better reveal the underlying structure and internal relationship in data. Meanwhile, the core tensor obtained by Tucker decomposition is more sparse and more representative, which helps to capture and retain key features, reduce data compression, and reduce computing overhead and time.

[0056] The multiple feature set fusion proposed by the application based on three different forms of vowel mixed tensors can integrate the features captured under different forms, and in the way of jointly training the detection model with multiple features, the detection model learns more comprehensive key features to improve the model performance. BRIEF DESCRIPTION OF DRAWINGS

[0057] Other features, objects and advantages of the application will become more apparent from the following detailed description of non-limiting embodiments with reference to the attached drawings:

[0058] Figure 1 The overall flowchart of the dysphagia patient detection method based on tensor decomposition in a preferred embodiment of the application.

[0059] Figure 2 The training feature set construction principle diagram based on spectrograms in a preferred embodiment of the application.

[0060] Figure 3 The two vowel mixed tensor construction schematic diagram in a specific application example of the application.

[0061] Figure 4 The one vowel mixed tensor construction schematic diagram in a specific application example of the application.

[0062] Figure 5 The overall composition schematic diagram of the dysphagia patient detection system based on tensor decomposition in a preferred embodiment of the application. DETAILED DESCRIPTION

[0063] The application will be described in detail below with reference to specific embodiments. The following embodiments will help those skilled in the art to further understand the application, but do not limit the application in any form. It should be noted that for those skilled in the art, without departing from the concept of the application, a number of modifications and improvements can be made. These all belong to the protection scope of the application.

[0064] To further explore the key features of characterizing dysphagia, an embodiment of the present application provides a tensor decomposition-based dysphagia patient detection method, which is based on a vowel mixed tensor and Tucker decomposition, and assists in detecting dysphagia patients, thereby improving the detection progress.

[0065] Specifically, as shown in the figure, Figure 1 the tensor decomposition-based dysphagia patient detection method provided by the embodiment can include the following operations:

[0066] S1, a collection device (such as a microphone) acquires speech data of a subject, wherein the subject includes a dysphagia patient and a normal person, and the speech data of each subject contains any one or any multiple vowel data;

[0067] S2, pre-processing the speech data, cropping irrelevant information in the speech data and performing amplitude normalization operation; this step makes the dysphagia detection model pay more attention to the features related to dysphagia itself, so as to improve the accuracy of subsequent detection;

[0068] S3, constructing different forms of vowel mixed tensors based on the speech spectrum of each subject's speech data;

[0069] S4, extracting features from each vowel mixed tensor based on orthogonal constraint Tucker decomposition to form a training feature set;

[0070] S5, training a dysphagia detection model according to the training feature set;

[0071] S6, detecting the speech data to be detected based on the dysphagia detection model to obtain the classification results of dysphagia patients and normal persons.

[0072] In some preferred embodiments, the irrelevant information in the speech data in the above S2 mainly includes the following information: some noise at the beginning and end of the recording.

[0073] In order to extract key features from speech signals for better classification and detection, in some preferred embodiments, the above S3 calculates the following two kinds of speech spectrum according to each speech signal: STFT speech spectrum and MFCC speech spectrum, and on the basis of the two kinds of speech spectrum, three forms of vowel mixed tensors are constructed, as shown in the figure, Figure 3 , Figure 4 which are: vowel mixed tensor based on MFCC speech spectrum, vowel mixed tensor based on entire STFT speech spectrum, and frequency band vowel mixed tensor based on different frequency bands of STFT speech spectrum.

[0074] Further, both STFT and MFCC spectrograms are calculated, and three forms of vowel mixed tensors are constructed, and the following operations can be further included:

[0075] S301, the pre-processed speech data is subjected to short-time Fourier transform to obtain an STFT spectrogram, and MFCCs are extracted from the speech data; specifically:

[0076] The speech data is subjected to short-time Fourier transform to convert the one-dimensional speech signal into a two-dimensional STFT spectrogram; the STFT spectrogram is mapped to a mel frequency scale to form a mel spectrum; the mel spectrum is logarithmically transformed and subjected to discrete cosine transform to obtain MFCC features, thereby converting the one-dimensional speech signal into a two-dimensional matrix containing time-varying MFCC coefficients, and an MFCC spectrogram is obtained; wherein:

[0077] ①STFT spectrogram: the speech data is subjected to short-time Fourier transform to convert the one-dimensional speech signal into a two-dimensional STFT spectrogram; short-time Fourier transform is to divide the original speech data into multiple short-time signals, which are considered to be stationary within the short-time signals; then, FFT transform is performed on each short-time signal, which is a faster form of discrete Fourier transform;

[0078] ②MFCC spectrogram: the STFT spectrogram is mapped to a mel frequency scale to form a mel spectrum, thereby simulating the nonlinear perception of the human auditory system to sound frequencies; then, the mel spectrum is logarithmically transformed and subjected to discrete cosine transform to obtain MFCC features, thereby converting the one-dimensional speech signal into a two-dimensional matrix containing time-varying MFCC coefficients;

[0079] S302, the vowel speech data of each subject is calculated to obtain an STFT spectrogram, which is spliced in the vowel dimension to construct a three-dimensional vowel mixed tensor based on the STFT spectrogram; the vowel speech data of each subject is calculated to obtain an MFCC spectrogram, which is also spliced in the vowel dimension to construct a three-dimensional vowel mixed tensor based on the MFCC spectrogram; wherein:

[0080] In a specific application example, the speech data of patients with dysphagia and normal people contains three types of vowel data: / a / , / o / , and / e / , based on which:

[0081] ①Constructing a vowel mixed tensor based on the MFCC spectrogram includes: calculating three MFCC spectrograms for each subject's / a / , / o / , and / e / speech data, and constructing a three-dimensional tensor from the three MFCC spectrograms in the vowel dimension wherein, I m represents the number of vowel signals used, which is 3 in the example of the present application, and Jm The number of MFCC feature vectors, which is 20 in the example of the present application; K m The time dimension, i.e. the total number of frames; Specifically, the MFCC spectrograms of the three vowels are used as the horizontal slices M(v,∶,∶), v=1,2,3 of the vowel mixing tensor, where v=1,2,3 represent the MFCC spectrograms of the vowels ” / a / ”, ” / o / ”, ” / e / ” respectively;

[0082] ② Constructing the vowel mixing tensor based on the entire STFT spectrogram, including: for each subject's ” / a / ”, ” / o / ”, ” / e / ” speech data, the STFT spectrogram is calculated by using the Hamming window respectively, and the vowel mixing tensor based on the entire STFT spectrogram The process of constructing the vowel mixing tensor based on the MFCC spectrogram is similar. The three dimensions of the tensor S are vowels, frequencies and time, where I s represents the number of vowel signals used, which is 3 in the example of the present application, J s represents the total number of frequency components in the frequency range, which is 20 in the example of the present application; K s represents the time dimension, i.e. the total number of frames, and the horizontal slice of S represents the correlation of frequency and time within a single vowel; as Figure 3 shown;

[0083] S303, for the three-dimensional vowel mixing tensor constructed based on the STFT spectrogram, the tensor is divided in the frequency dimension to obtain n three-dimensional frequency band sub-tensors; wherein:

[0084] ③ Constructing the vowel mixing tensor based on different frequency bands of the STFT spectrogram, including: for each subject, the vowel mixing tensor S based on the entire STFT spectrogram is obtained, which is divided into n frequency band sub-tensors B j , j=1,..., n, n can be preferably 4; wherein is the number of frequency components in the frequency band frequency range of the i-th sub-tensor; Specifically, the two-dimensional frequency band matrix in the same frequency range of the three STFT spectrograms is spliced in the vowel dimension to construct a new three-dimensional tensor; as Figure 4 shown.

[0085] It should be noted that for the case of containing other vowel data, the method of constructing the three-dimensional vowel mixing tensor is the same, which will not be described here.

[0086] In order to capture the potential relationship between the vowels and more complex patterns from each constructed vowel mixing tensor, in some preferred embodiments, the above S4, the Tucker decomposition based on orthogonal constraints extracts features from each vowel mixing tensor to constitute a training feature set, which can further include the following operations:

[0087] The feature extraction method based on orthogonal constraint Tucker decomposition is adopted, different forms of vowel mixed tensors are input into the Tucker decomposition model with orthogonal constraint, and the corresponding core tensor is obtained by setting appropriate rank. The feature extraction method processes the correlation and interaction in high-order data to form a feature set. Among them:

[0088] The Tucker decomposition model with orthogonal constraint is constructed as follows:

[0089]

[0090] Wherein, X∈R I×J×K is a three-dimensional tensor, which represents the unified representation of three forms of vowel mixed tensors, R represents a matrix, I, J, K represent the size of three dimensions of the three-dimensional tensor X respectively; G∈R P×Q×L represents the core tensor obtained after the Tucker decomposition of the vowel mixed tensor, which contains the mutual relationship between different modes, wherein P, Q, L represent the size of three dimensions of the core tensor G respectively; A∈R I×R , B∈R J×Q , C∈R K×L respectively represent the factor matrices in three modes obtained after Tucker decomposition, wherein ×1, ×2, ×3 represent the product between tensor and matrix; g pql , a p , b q , c l respectively represent the specific elements of G, A, B and C; ○ represents the vector outer product.

[0091] For the setting of the core tensor rank, the searching method in a certain interval is adopted, and the relatively optimal rank setting in the interval is finally obtained, and then the final core tensor is obtained.

[0092] The corresponding core tensors obtained from different forms of vowel mixed tensors are respectively flattened into vectors to obtain the corresponding feature sets.

[0093] Based on the corresponding feature sets, the training feature sets are further constructed.

[0094] Specifically, the above S4 further includes the following operations:

[0095] S401, input the constructed different forms of vowel mixed tensors into the Tucker decomposition model with orthogonal constraint, and obtain the core tensor after rank optimization;

[0096] S402, according to the core tensor result output after each vowel mixed tensor passes through the Tucker decomposition model, flatten it into a vector and obtain the respective feature set after feature selection; wherein the feature selection mainly includes the following principles and methods:

[0097] Firstly, the t-test method is used for coarse feature selection to distinguish dysphagia patients from healthy people by overall distribution;

[0098] Secondly, the fine features are obtained by the exact matching analysis method. Specifically, the features of each range are calculated according to the distribution of the original data to reduce the impact of unbalanced data, and the threshold setting is adjusted to screen the features.

[0099] S403, based on the respective feature sets, a training feature set is constructed, as shown in Figure 2 , including:

[0100] ①The feature sets obtained after the Tucker decomposition model of the three different forms of vowel mixed tensors are selected as the three training feature sets respectively;

[0101] ②The final training feature set is obtained by splicing and fusing the feature set obtained based on the entire STFT spectrogram vowel mixed tensor and the feature set obtained based on the MFCC spectrogram vowel mixed tensor; the training feature set contains effective key distinguishing features in the STFT spectrogram and the MFCC spectrogram;

[0102] ③The final training feature set is obtained by splicing and fusing the feature set obtained based on the different STFT spectrogram frequency bands and the feature set obtained based on the MFCC spectrogram vowel mixed tensor; the training feature set contains the entire frequency band information; the training feature set contains effective key distinguishing features in the STFT spectrogram and the MFCC spectrogram;

[0103] ④The final training feature set is obtained by splicing and fusing the multiple feature sets obtained based on the n frequency band sub-tensors divided by the STFT spectrogram different frequency band constructed vowel mixed tensor;

[0104] ⑤The final training feature set is obtained by splicing and fusing the feature set obtained based on the MFCC spectrogram vowel mixed tensor and the training feature set obtained in ④.

[0105] Specifically, the feature sets obtained based on the STFT spectrogram different frequency band constructed vowel mixed tensor are fused together to contain the entire frequency band information; the feature sets obtained based on the STFT spectrogram different frequency band constructed vowel mixed tensor and the feature set obtained based on the STFT spectrogram vowel mixed tensor are fused with the feature set obtained based on the MFCC spectrogram vowel mixed tensor to contain effective key distinguishing features in the STFT spectrogram and the MFCC spectrogram.

[0106] In some preferred embodiments, the swallowing disorder detection model S5 can employ a common machine learning model such as SVM, etc. To improve the detection performance under small sample, the present application preferably employs a SVM model with polynomial kernel, which can increase the complexity of the model; the machine learning model is trained using the training feature set and data labeling to obtain the swallowing disorder detection model.

[0107] In some preferred embodiments, the S5 can further include the following operations:

[0108] Based on the detection model, the test voice data set is detected to obtain the classification results of the swallowing disorder patients and normal persons, and the detection model is verified.

[0109] The swallowing disorder detection method based on tensor decomposition provided by the above embodiments of the present application can safely and effectively detect whether the current subject has a swallowing disorder, which is convenient for doctors to diagnose and follow up; the three-dimensional vowel mixed tensor can capture the key acoustic features in each vowel data and deeply capture the potential correlation between the three vowels; the orthogonal constrained Tucker decomposition is used to reduce the dimension and extract the features of the three-dimensional vowel mixed tensor, and the obtained features are more representative and more critical, which helps to improve the performance of the swallowing disorder detection model.

[0110] The technical solutions provided by the above embodiments of the present application will be further described in detail below with reference to a specific application example.

[0111] In this specific application example, the voice data of swallowing disorder patients and normal persons are collected by a common microphone, such as a common microphone with 16-bit sampling and a sampling rate of 44.1 kHz, which records the voice data of each participant. All data are stored as waveform data, and all participants are required to read out each required content loudly and individually according to the doctor's instructions during the recording process.

[0112] The voice content collected for each participant is three vowel data " / a / ", " / o / ", and " / e / ". In recent years, sustained vowel signals have been used for swallowing disorder screening and diagnosis, and good application results have been achieved. For the three different vowel signals (i.e. " / a / ", " / o / ", and " / e / "), they reflect different sound positions and sound conditions, so different voice feature information can be captured. Among them, the subject is required to read out the corresponding content as long as possible.

[0113] The preprocessing of the collected voice data mainly includes cutting off irrelevant information in the voice data and amplitude normalization operation, so that the dysphagia detection model pays more attention to the characteristics of dysphagia itself, so as to improve the accuracy of subsequent detection.

[0114] The classical features obtained for each voice data are STFT spectrogram and MFCC spectrogram. STFT spectrogram and MFCC spectrogram are widely used to extract key features from voice signals. Among them, the MFCC spectrogram contains the fundamental frequency information that can distinguish healthy participants from dysphagia patients. In addition, dysphagia patients usually show significant fluctuations in their voice and amplitude and frequency, which can be described in the STFT spectrogram, which shows the changes of frequency and amplitude over time.

[0115] Based on the above process, in this specific application instance, further by the professional doctor for each subject to carry out dysphagia assessment, wherein the assessment method adopts the clinically classic Katakura drinking test, finally obtains the labeled data, specifically, the label of normal person is set as 0, and the label of dysphagia patient is set as 1. Three kinds of vowel data of each subject are collected, " / a / ", " / o / ", and " / e / ".

[0116] In this specific application instance, for each subject, the three kinds of vowel data collected by the subject, " / a / ", " / o / ", and " / e / " are cut and amplitude normalized to obtain spectrograms, respectively, wherein the cutting length is 2s.

[0117] In this specific application instance, the construction of the feature set based on tensor is an important innovative step, which includes:

[0118] ① Feature set based on MFCC spectrogram: First, the mel frequency cepstral coefficient MFCC is extracted from the three kinds of vowel data, respectively. In this application instance, the number of MFCC features extracted is 20, and the number of frames in the time dimension is 87. For the three MFCC spectrograms obtained from the voice data of the same subject, they are spliced into a three-dimensional vowel mixed tensor in the voice dimension Where I m represents the number of vowel signals used, which is 3 in this work; J m represents the number of MFCC feature vectors, which is 20 in this work; K mrepresent the time dimension, i.e. the total number of frames 87. Then the constructed vowel mixture tensor based on MFCC spectrogram is input into the Tucker decomposition model. Since the rank of the core tensor needs to be set for optimization, in this application example, a simple artificial setting method is adopted. Specifically, the rank of the vowel mixture tensor based on MFCC spectrogram in the Tucker decomposition rank optimization is in the range of 3x6x6 to 3x20x20, and the ranks of the last two dimensions are the same and each takes a value interval of 2. For the core tensor obtained after the Tucker decomposition model, it is flattened into a vector and input into the feature selection module to obtain the feature set for subsequent training and verification of the dysphagia patient detection model. Specifically, in this application example, the optimal rank in the range obtained after rank optimization is 3x10x10.

[0119] ②Feature set based on STFT spectrogram: First, the STFT spectrogram of each speech signal is calculated by using the Hamming window, and the duration of the Hamming window is about 46 milliseconds and the hop length is 23 milliseconds. The number of discrete Fourier points and frames is 2048 and 87 respectively. The construction process of the entire vowel mixture tensor based on STFT spectrogram is similar to that of the vowel mixture tensor based on MFCC spectrogram. The three modes of tensor S are vowels, frequencies and times, where I s and K s represent the vowel and time dimensions respectively, which have the same meaning as in tensor M, and J s is the total number of frequency components in the frequency range, i.e. 1025. The horizontal slice of S represents the correlation between frequency and time within a single vowel. The rank of the vowel mixture tensor based on STFT spectrogram in the Tucker decomposition rank optimization is in the range of 3x16x16 to 3x30x30, and the rank of the last two dimensions is the same and each takes a value interval of 2. Similarly, the core tensor obtained after the Tucker decomposition model of the vowel mixture tensor based on STFT spectrogram is input into the feature selection module after being flattened into a vector to obtain the feature set for inputting into the detection model. Specifically, in this application example, the optimal rank in the range obtained after rank optimization is 3x24x24.

[0120] ③Feature set based on different frequency band STFT spectrogram: constructed from the vowel mixture tensor based on STFT spectrogram, in order to explore the influence of dividing the entire STFT spectrogram into multiple frequency bands on feature mining, the entire three-dimensional STFT spectrogram vowel mixture tensor is divided into four frequency band sub-tensors B j , j = 1,..., 4, where The number of frequency components in the frequency band frequency range of the ith sub-tensor, where when j = 1, the frequency range of the sub-tensor is 1 to 256, when j = 2, the frequency range of the sub-tensor is 257 to 512, when j = 3, the frequency range of the sub-tensor is 513 to 768, and when j = 4, the frequency range of the sub-tensor is 769 to 1025. For each frequency band sub-tensor, the same key feature mining is performed through the Tucker decomposition module. Wherein, the rank of each frequency band sub-tensor in the Tucker decomposition is optimized in the range of 3x8x8 to 3x40x40, and the obtained core tensor is flattened into a vector and input into the feature selection module to obtain the final feature set input into the detection model. Specifically, in the application example, the optimal rank in the range obtained after the rank optimization of different frequency band sub-tensors is not the same, specifically, when four sub-tensors are divided according to the frequency from low to high, the optimal rank of the first sub-tensor is 3x24x24, the optimal rank of the second frequency band sub-tensor is 3x12x12, the optimal rank of the third frequency band sub-tensor is 3x36x36, and the optimal rank of the fourth frequency band sub-tensor is 3x40x40.

[0121] ④Fusion feature set based on four different frequency band sub-tensor features: the core tensor under the optimal rank in the range of the four frequency band sub-tensors obtained in the above ③ is constructed into four feature sets, and the features learned based on each frequency band are different, so the four feature sets contain different key features for swallowing disorder detection. Therefore, the optimal feature set in the range obtained from the four frequency band sub-tensors is spliced and fused into a new feature set to fully utilize the detailed features captured in each frequency band and explore whether the frequency division of multiple frequency bands can better focus on feature extraction in each frequency band than the entire spectrogram.

[0122] ⑤Fusion feature set based on entire STFT spectrogram and MFCC spectrogram features: the core tensor under the optimal rank based on the MFCC spectrogram and the feature under the optimal rank based on the STFT spectrogram obtained in the above ① and ② are spliced into a long vector, which is input into the feature selection module as the feature, and the feature set after feature selection is the final feature set input into the detection model.

[0123] ⑥Fusion feature set based on different frequency band STFT spectrogram and MFCC spectrogram features: the feature set under the optimal rank based on the MFCC spectrogram and the feature set under the optimal rank based on the four different frequency band STFT spectrogram sub-tensors obtained in the above ① and ③ are spliced and fused, and the obtained long vector is input into the feature selection module to obtain the final feature set input into the detection model. Specifically, the features obtained from the four frequency band sub-tensors are fused with the features obtained from the MFCC spectrogram to obtain four new feature sets.

[0124] ⑦ Four fusion feature sets based on different frequency band sub-tensors and MFCC spectrogram features: the fusion feature set of the optimal rank of the four frequency band sub-tensors obtained above, and the feature set of the optimal rank of the MFCC spectrogram obtained above, are spliced into a new feature set.

[0125] In this specific application example, the dysphagia patient detection model is constructed based on machine learning, and the structure can be a support vector machine (SVM), which is specifically implemented using the Matlab language. In addition, the support vector machine with a polynomial kernel is used for classification tasks, and the support vector machine with a polynomial sum increases the model complexity, and the fitting effect of the data in this application is better. In order to find a more optimal SVM model, a search is performed within a certain range. Specifically, the degree of the polynomial kernel is in the range of 1 to 4, and the size of the polynomial kernel is in the range of 1 to 10. Among them, the detection model is used for training and verification on the constructed feature set to evaluate the effectiveness and keyness of the feature set constructed in different ways.

[0126] Through a certain amount of experiments, the following experimental results are obtained in this specific application example:

[0127] The speech data of three vowels of 234 subjects are used as the data set, and the data of 176 subjects are used as the training set, including 60 normal people and 116 dysphagia patients. The speech data of 58 subjects are used as the test machine, including 20 normal people and 38 dysphagia patients. The indicators used are as follows:

[0128]

[0129]

[0130] Among them, TP, TN, FP, and FN are true positive, true negative, false positive, and false negative samples, respectively;

[0131] In addition, the area under the ROC curve, i.e. AUC, is calculated as another indicator.

[0132] After experiments on multiple proposed feature sets, when the third frequency band sub-tensor based on the STFT spectrogram feature and the feature based on the MFCC spectrogram feature are spliced and fused, the optimal result is obtained, and the optimal result is shown in Table 1:

[0133] Table 1

[0134]

[0135] It can be known from the above specific application examples and quantitative test results that the swallowing disorder patient detection method based on tensor decomposition provided by the above embodiments has the advantages of being based on voice, simple operation, low cost, no technical threshold in use, being capable of effectively obtaining the detection result of the swallowing disorder patient, and being capable of safely and effectively detecting the swallowing disorder patient, thereby facilitating subsequent follow-up of doctors.

[0136] Based on the same inventive concept, an embodiment of the present application further provides a swallowing disorder patient detection system based on tensor decomposition.

[0137] Specifically, as shown in the drawings, the swallowing disorder patient detection system based on tensor decomposition provided by the embodiment can include the following modules. Figure 5

[0138] The data acquisition module is configured to acquire voice data of swallowing disorder patients and normal persons respectively, and the voice data contains any one or any multiple of vowel data.

[0139] The data processing module is configured to pre-process the voice data, crop irrelevant information in the voice data, and perform amplitude normalization operation.

[0140] The training data construction module is configured to acquire a spectrogram of the voice data, construct mixed tensors of different forms of vowels, send the mixed tensors of different forms of vowels into a Tucker decomposition model with added orthogonal constraints, set a proper rank to obtain a corresponding core tensor, flatten the core tensor into a vector as a feature input, and construct a training feature set after feature selection.

[0141] The swallowing disorder detection module is configured to train a swallowing disorder detection model by using the training feature set, detect the voice data to be detected based on the swallowing disorder detection model, and obtain classification results of the swallowing disorder patients and the normal persons.

[0142] It should be noted that the steps in the method provided by the present application can be implemented by corresponding modules in the system, and those skilled in the art can implement the technical solution of the system by referring to the step flow of the method, that is, the embodiments in the method can be understood as preferred examples of implementing the system, and details are not described herein.

[0143] An embodiment of the present application further provides a computer terminal including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor is configured to execute the computer program to implement the method of any one of the above embodiments of the present application, or run the system of any one of the above embodiments of the present application.

[0144] ​Optionally, the memory is configured to store a computer program. The memory can include volatile memory (for example, random access memory, abbreviated as RAM, such as static random access memory, abbreviated as SRAM, double data rate synchronous dynamic random access memory, abbreviated as DDR SDRAM, etc.), and can also include non-volatile memory (for example, flash memory). The memory is configured to store computer programs (for example, application programs, functional modules, etc. for implementing the above method), computer instructions, etc. The above computer programs, computer instructions, etc. can be stored in one or more memories in a partitioned manner. Moreover, the above computer programs, computer instructions, data, etc. can be invoked by the processor.

[0145] The above computer programs, computer instructions, etc. can be stored in one or more memories in a partitioned manner. Moreover, the above computer programs, computer instructions, data, etc. can be invoked by the processor.

[0146] The processor is configured to execute the computer program stored in the memory, so as to implement each step in the method or each module of the system according to the above embodiments. For details, refer to the related description in the above method and system embodiments.

[0147] The processor and the memory can be an independent structure, or can be an integrated structure. When the processor and the memory are an independent structure, the memory and the processor can be coupled and connected through a bus.

[0148] An embodiment of the present application further provides a computer readable storage medium, which stores a computer program. The computer program is configured to be executed by a processor, so as to implement the method in any one of the above embodiments of the present application, or run the system in any one of the above embodiments of the present application.

[0149] Among them, the computer readable medium includes computer storage medium and communication medium, wherein the communication medium includes any medium facilitating computer program transmission from one place to another. The storage medium can be any available medium accessible by a general or special purpose computer. An exemplary storage medium is coupled to the processor, so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in the user equipment. Of course, the processor and the storage medium can also exist as discrete components in the communication device.

[0150] Those skilled in the art know that, in addition to implementing the system provided by the present application and each device thereof in a pure computer readable program code manner, the system provided by the present application and each device thereof can also be implemented in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers and embedded microcontrollers by logically programming the method steps to achieve the same functions. Therefore, the system provided by the present application and each device thereof can be considered as a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component; the devices for implementing various functions can also be considered as both software modules implementing methods and structures within hardware components.

[0151] The tensor decomposition-based dysphagia patient detection method and system provided by the above-mentioned embodiments of the present application adopts to construct three forms of vowel mixed tensors based on the spectrogram of the speech data (which can form a speech database) of each subject. The high-order tensor constructed by multiple vowels can retain the interaction information therebetween. Among them, the vowel mixed tensor constructed based on the MFCC spectrogram can capture the fundamental frequency information, the vowel mixed tensor constructed based on the entire STFT spectrogram can capture the stable features of the speech, and the vowel mixed tensor constructed based on the sub-band of the STFT spectrogram can capture more detailed features within the current frequency band.

[0152] The tensor decomposition-based dysphagia patient detection method and system provided by the above-mentioned embodiments of the present application performs feature extraction on the corresponding vowel mixed tensor based on the Tucker decomposition model, wherein the core tensor rank is optimized within a certain range, uses the obtained feature set to construct a training feature set and performs data labeling, trains the dysphagia patient detection model based on the support vector machine based on the support vector machine, compares the results of different forms of feature sets, and obtains the trained model. The model is used to directly predict whether to suffer from dysphagia through the speech signal.

[0153] The specific embodiments of the present application are described above. It needs to be understood that the present application is not limited to the specific embodiments described above, and various modifications or changes can be made by those skilled in the art within the scope of the claims, which does not affect the essential content of the present application. The above preferred features can be combined in any manner without conflict, and used.

Claims

1. A method for detecting dysphagia patients based on tensor decomposition, characterized in that: include: Acquiring speech data of a dysphagia patient and a normal person respectively, wherein the speech data includes any one or more vowel data; Preprocessing the voice data, cutting out irrelevant information in the voice data and performing amplitude normalization operations; Obtaining a spectrogram of the speech data and constructing vowel mixture tensors of different forms; The vowel mixture tensors of different forms are fed into a Tucker decomposition model with orthogonal constraints, and a suitable rank is set to obtain a corresponding core tensor; Flattening the kernel tensor into a vector as a feature input, constructing a training feature set after feature selection to train a swallowing disorder detection model; Detecting the speech data to be detected based on the dysphagia detection model to obtain classification results of dysphagia patients and normal people; The obtaining of the spectrogram of the speech data includes: Get STFT spectrogram, including: Performing a short-time Fourier transform on the speech data to convert the one-dimensional speech signal into a two-dimensional STFT spectrogram; Get MFCC spectrogram, including: Mapping the STFT spectrogram to a Mel frequency scale to form a Mel spectrum; After taking the logarithm of the Mel spectrum, a discrete cosine transform is performed to obtain MFCC features, and the one-dimensional speech signal is converted into a two-dimensional matrix containing MFCC coefficients that vary with time to obtain an MFCC spectrogram; The construction of different forms of vowel mixture tensors includes: Construct a vowel mixture tensor based on the MFCC spectrogram, including: The MFCC spectrogram is calculated for each subject’s vowel data, and then the obtained MFCC spectrogram is constructed into a three-dimensional tensor Among them, I m Represents the number of vowel data used; J m Represents the number of MFCC feature vectors; K m Represents the time dimension, that is, the total number of frames; Construct a vowel mixture tensor based on the entire STFT spectrogram, including: The STFT spectrogram is calculated for each subject’s vowel data, and then the obtained STFT spectrogram is constructed into a three-dimensional tensor Among them, I s Represents the number of vowel data used; J s Represents the total number of frequency components within the frequency range; K s Represents the time dimension, that is, the total number of frames; Construct vowel mixture tensors based on different frequency bands of the STFT spectrogram, including: After constructing the vowel mixture tensor S for each subject based on the entire STFT spectrogram, it is divided into n frequency band sub-tensors B in the frequency dimension. j , j=1,...,n.

2. The method for detecting dysphagia patients based on tensor decomposition according to claim 1, characterized in that: The different forms of vowel mixture tensors are fed into a Tucker decomposition model with orthogonal constraints, and a suitable rank is set to obtain a corresponding core tensor, including: The Tucker decomposition model with orthogonal constraints is constructed as follows: Where X∈R I×J×K It is a three-dimensional tensor, representing the unified representation of three forms of vowel mixture tensors, R represents the matrix, I, J, K represent the size of the three dimensions of the three-dimensional tensor X respectively; G∈R P×Q×L Represents the core tensor obtained after Tucker decomposition of the vowel mixture tensor, which contains the relationship between different modes, where P, Q, and L represent the sizes of the three dimensions of the core tensor G respectively; A∈R I×P , B∈R J×Q , C∈R K×L They represent the three factor matrices obtained after Tucker decomposition, where ×1, ×2, and ×3 represent the product between the tensor and the matrix; g pql 、a p 、b q 、c l Represent the specific elements of G, A, B, and C respectively; represents the vector outer product; By searching within a certain interval, the optimal rank setting within the interval is obtained, and then the final kernel tensor is obtained.

3. The method for detecting dysphagia patients based on tensor decomposition according to claim 1, characterized in that: Flattening the kernel tensor into a vector as a feature input includes: The corresponding kernel tensors obtained from different forms of vowel mixture tensors are flattened into vectors respectively, and the corresponding feature sets are obtained as feature inputs.

4. The method for detecting dysphagia patients based on tensor decomposition according to claim 1, characterized in that: The feature selection includes: The coarse feature selection was performed by t-test method, and the obtained coarse feature training feature set was used to distinguish dysphagia patients from healthy people based on the overall distribution; The exact matching analysis method is used to calculate the features of each range according to the distribution of the original data, and the threshold setting is adjusted to filter the features. The obtained fine feature training feature set is used to reduce the impact of unbalanced data.

5. The method for detecting dysphagia patients based on tensor decomposition according to claim 1, characterized in that: The constructing of the training feature set includes: The feature sets obtained based on different forms of vowel mixture tensors are respectively used as the first training feature sets; The feature set obtained by the vowel mixture tensor constructed based on the STFT spectrogram and the feature set obtained by the vowel mixture tensor constructed based on the MFCC spectrogram are concatenated and fused to obtain the second training feature set; Multiple feature sets obtained from vowel mixture tensors constructed based on different frequency bands of the STFT spectrogram are concatenated and fused with the feature set obtained from the vowel mixture tensor based on the MFCC spectrogram to obtain a third training feature set; Multiple feature sets obtained from vowel mixture tensors constructed based on different frequency bands of the STFT spectrogram are concatenated and fused to obtain a fourth training feature set; The fourth training feature set and the feature set obtained by the vowel mixture tensor constructed based on the MFCC spectrogram are concatenated and fused to obtain the fifth training feature set.

6. The method for detecting dysphagia patients based on tensor decomposition according to claim 1, characterized in that: The training of a swallowing disorder detection model includes: Provide a support vector machine as an initial model; The initial model is trained using the training feature set and performing data annotation to obtain a dysphagia detection model.

7. A swallowing disorder patient detection system based on tensor decomposition, characterized in that: include: A data acquisition module, which is used to respectively acquire speech data of dysphagia patients and normal people, wherein the speech data includes any one or more vowel data; A data processing module, which is used to pre-process the voice data, cut out irrelevant information in the voice data and perform amplitude normalization operations; a training data construction module, the module being configured to obtain a spectrogram of the speech data and construct vowel mixture tensors of different forms; feed the vowel mixture tensors of different forms into a Tucker decomposition model with an orthogonal constraint, set an appropriate rank to obtain a corresponding core tensor; flatten the core tensor into a vector as a feature input, and construct a training feature set after feature selection; A swallowing disorder detection module uses the training feature set to train a swallowing disorder detection model, detects the speech data to be detected based on the swallowing disorder detection model, and obtains classification results of swallowing disorder patients and normal people.

8. A computer terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When executing the computer program, the processor can be used to perform the method according to any one of claims 1 to 6, or run the system according to claim 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it can be used to perform the method according to any one of claims 1 to 6, or to run the system according to claim 7.

Images