Tensor decomposition-based dysphagia patient detection method and system

Abstract

The invention provides a swallowing disorder patient detection method and system based on tensor decomposition, and the method comprises the steps: obtaining the voice data of a swallowing disorder patient and a normal person, carrying out the preprocessing of the voice data, cutting irrelevant information in the voice data, and carrying out the amplitude normalization operation; obtaining a speech spectrogram of the speech data, and constructing vowel mixed tensors in different forms; different forms of vowel mixed tensors are sent to a Tucker decomposition model added with orthogonal constraints, and a proper rank is set to obtain a corresponding nuclear tensor; flattening the kernel tensor into a vector, taking the vector as feature input, and constructing a training feature set to train a dysphagia detection model after feature selection; and detecting the to-be-detected voice data based on the detection model to obtain classification results of the dysphagia patients and the normal people. The method is realized based on voice, is simple to operate and relatively low in cost, has no technical threshold in use, can safely and effectively detect the dysphagia patient, and is convenient for follow-up.

Description

Technical Field

[0001] The present invention relates to the technical field of dysphagia classification, and in particular to a method and system for detecting dysphagia patients based on tensor decomposition, as well as a corresponding computer terminal and a computer-readable storage medium. Background Art

[0002] Dysphagia refers to the symptoms of swallowing dysfunction that occurs between the mouth and the stomach, which may be caused by diseases that cause damage to organs or physiological conditions, such as stroke and Alzheimer's disease. Dysphagia affects the quality of life of patients and can lead to aspiration pneumonia, malnutrition, and even endanger the patient's life. In clinical practice, swallow contrast study (VFSS) and fiber optic endoscopic swallowing function test (FEES) are generally used as the gold standard for evaluating dysphagia. These evaluation methods require invasion of the body, which causes discomfort and radiation exposure to patients, and the operation requires professional doctors. Speech analysis is non-invasive, easy to collect and cost-effective. In recent years, many studies have shown that speech data can generate swallowing characteristics, which is helpful for the clinical identification of pathological patterns. Detection of patients with dysphagia based on speech analysis is a safe, convenient and effective way.

[0003] In recent years, machine learning has demonstrated excellent performance in tasks related to classification, which has promoted the development of automated detection technology for dysphagia. Many recent studies have collected voice data and proposed automatic classification of patients with dysphagia based on machine learning. Commonly used voice protocols include the sound of swallowing water boluses, but such methods have safety risks and can easily increase the risk of choking. In contrast, continuous vowel voice signals are easier to collect and safer, and have advantages in detecting dysphagia. However, current studies either only use single vowel voice signals or do not fully utilize the potential correlation between different vowels. For this reason, how to efficiently use the effective features in vowel signals to detect patients with dysphagia is a major research hotspot.

[0004] In the existing technology, classic speech features are mostly used to classify dysphagia. The spectrogram obtained from the speech signal also contains many speech features. For example, the STFT spectrogram can reflect the flickering and jitter characteristics of the subject's voice, and the MFCC spectrogram contains fundamental frequency information. These features are helpful in distinguishing patients with dysphagia from normal people. Although these features are mostly used to assist screening in current work, very few studies have combined them to explore the key features that characterize patients with dysphagia. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the present invention provides a method and system for detecting dysphagia patients based on tensor decomposition, and also provides a corresponding computer terminal and computer-readable storage medium.

[0006] According to one aspect of the present invention, a method for detecting dysphagia patients based on tensor decomposition is provided, comprising:

[0007] Acquire speech data of dysphagia patients and normal people respectively, wherein the speech data includes any one or more types of vowel data;

[0008] Preprocessing the speech data, cutting out irrelevant information in the speech data and performing amplitude normalization operation;

[0009] Obtaining a spectrogram of the speech data and constructing vowel mixture tensors in different forms;

[0010] The different forms of vowel mixture tensors are sent to a Tucker decomposition model with orthogonal constraints, and a suitable rank is set to obtain a corresponding core tensor;

[0011] Flatten the kernel tensor into a vector as feature input, construct a training feature set after feature selection to train a swallowing disorder detection model;

[0012] The speech data to be detected is detected based on the dysphagia detection model to obtain classification results of dysphagia patients and normal people.

[0013] Preferably, the obtaining of the spectrogram of the speech data comprises:

[0014] Get STFT spectrogram, including:

[0015] Performing short-time Fourier transform on the speech data to convert the one-dimensional speech signal into a two-dimensional STFT spectrogram;

[0016] Get MFCC spectrogram, including:

[0017] Mapping the STFT spectrogram to a Mel frequency scale to form a Mel spectrum;

[0018] After taking the logarithm of the Mel spectrum, a discrete cosine transform is performed to obtain MFCC features, and the one-dimensional speech signal is converted into a two-dimensional matrix containing MFCC coefficients that vary with time to obtain an MFCC spectrogram;

[0019] The construction of different forms of vowel mixture tensors includes:

[0020] Construct vowel mixture tensor based on MFCC spectrogram, including:

[0021] The MFCC spectrogram is calculated for each subject’s vowel data, and then the obtained MFCC spectrogram is constructed into a three-dimensional tensor Among them, I m Represents the number of vowel data used; Jm Represents the number of MFCC feature vectors; K m Represents the time dimension, that is, the total number of frames;

[0022] Construct a vowel mixture tensor based on the entire STFT spectrogram, including:

[0023] The STFT spectrogram is calculated for each subject's vowel data, and then the obtained STFT spectrogram is constructed into a three-dimensional tensor Among them, I s Represents the number of vowel data used; J s Represents the total number of frequency components within the frequency range; K s Represents the time dimension, that is, the total number of frames;

[0024] Construct vowel mixture tensors based on different frequency bands of STFT spectrogram, including:

[0025] After constructing the vowel mixture tensor S for each subject based on the entire STFT spectrogram, it is divided into n sub-tensors B in the frequency dimension. j , j = 1, ..., n, representing n frequency band sub-tensors.

[0026] Preferably, the step of sending the vowel mixture tensors of different forms into a Tucker decomposition model with orthogonal constraints and setting a suitable rank to obtain a corresponding core tensor includes:

[0027] The Tucker decomposition model with orthogonal constraints is constructed as follows:

[0028]

[0029] Where X∈R I×J×K is a three-dimensional tensor, representing the unified representation of three forms of vowel mixture tensors, R represents a matrix, I, J, K represent the sizes of the three dimensions of the three-dimensional tensor X respectively; G∈R P×Q×L represents the core tensor obtained after Tucker decomposition of the vowel mixture tensor, which contains the relationship between different modes, where P, Q, and L represent the sizes of the three dimensions of the core tensor G respectively; A∈R I×P , B∈R J×Q , C∈R K×L They represent the factor matrices in the three modes obtained after Tucker decomposition, where ×1, ×2, and ×3 represent the product between the tensor and the matrix; g pql 、a p , b q 、c l Represent the specific elements of G, A, B, and C respectively; ○ represents the vector outer product;

[0030] By searching within a certain interval, the optimal rank setting within the interval is obtained, that is, the classification accuracy is optimal within the set rank interval, and then the final kernel tensor is obtained.

[0031] Preferably, the step of flattening the core tensor into a vector as a feature input comprises:

[0032] The corresponding kernel tensors obtained from different forms of vowel mixture tensors are flattened into vectors respectively, and the corresponding feature sets are obtained as feature inputs.

[0033] Preferably, the feature selection includes:

[0034] First, coarse feature selection was performed using the t-test method, and the obtained coarse feature training feature set was used to distinguish dysphagia patients from healthy people based on their overall distribution;

[0035] Secondly, fine features are obtained through exact matching analysis. Specifically, by calculating the features of each range according to the distribution of the original data and adjusting the threshold setting to filter the features, the obtained fine feature training feature set is used to alleviate the impact of unbalanced data.

[0036] Preferably, the constructing of the training feature set includes:

[0037] The feature sets obtained based on different forms of vowel mixture tensors are respectively used as the first training feature sets;

[0038] The feature set obtained by the vowel mixture tensor constructed based on the STFT spectrogram and the feature set obtained by the vowel mixture tensor constructed based on the MFCC spectrogram are concatenated and fused to obtain a second training feature set;

[0039] Multiple feature sets obtained from vowel mixture tensors constructed based on different frequency bands of the STFT spectrogram are concatenated and fused with feature sets obtained from vowel mixture tensors based on the MFCC spectrogram to obtain a third training feature set;

[0040] Multiple feature sets obtained from vowel mixture tensors constructed based on different frequency bands of the STFT spectrogram are concatenated and fused to obtain a fourth training feature set;

[0041] The fourth training feature set and the feature set obtained by the vowel mixture tensor constructed based on the MFCC spectrogram are concatenated and fused to obtain the fifth training feature set.

[0042] Preferably, the training of a swallowing disorder detection model comprises:

[0043] Provide a support vector machine as an initial model;

[0044] The initial model is trained using the training feature set and performing data annotation to obtain a dysphagia detection model.

[0045] According to another aspect of the present invention, a swallowing disorder patient detection system based on tensor decomposition is provided, comprising:

[0046] A data acquisition module, the module is used to respectively acquire speech data of dysphagia patients and normal people, wherein the speech data includes any one or more vowel data;

[0047] A data processing module, which is used to pre-process the voice data, cut out irrelevant information in the voice data and perform amplitude normalization operations;

[0048] A training data construction module, which is used to obtain the spectrogram of the speech data and construct vowel mixture tensors of different forms; send the vowel mixture tensors of different forms to a Tucker decomposition model with orthogonal constraints, set a suitable rank to obtain a corresponding core tensor; flatten the core tensor into a vector as a feature input, and construct a training feature set after feature selection;

[0049] A swallowing disorder detection module uses the training feature set to train a swallowing disorder detection model, detects the speech data to be detected based on the swallowing disorder detection model, and obtains classification results of swallowing disorder patients and normal people.

[0050] According to a third aspect of the present invention, there is provided a computer terminal comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, can be used to execute any one of the methods described above in the present invention, or to run the system described above in the present invention.

[0051] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can be used to execute any of the methods described above in the present invention, or to run the system described above in the present invention.

[0052] Due to the adoption of the above technical solution, the present invention has at least one of the following beneficial effects compared with the prior art:

[0053] The present invention can automatically detect swallowing disorder patients without human intervention on the input vowel voice data, and output the test results to assist medical personnel in diagnosis and treatment and subsequent follow-up, thus saving labor costs to a certain extent.

[0054] The present invention constructs three different forms of vowel mixture tensors based on two spectrograms. It can fully utilize the advantage of high-order tensors in retaining the interactive information between different vowels, while capturing a variety of features that are helpful for detecting patients with dysphagia. It can greatly improve the accuracy of detection and better characterize the differences in speech expression between patients with dysphagia and normal people.

[0055] The Tucker decomposition used in the present invention is an effective mechanism for decomposing high-order tensors. Compared with the unconstrained Tucker decomposition, the orthogonal Tucker decomposition can better reveal the underlying structure and intrinsic relationship in the data. At the same time, the core tensor obtained by Tucker decomposition is sparser and more representative, which helps to compress data and reduce computational overhead and time while capturing and retaining key features.

[0056] The fusion of multiple feature sets proposed by the present invention based on three different forms of vowel mixture tensors can integrate the features captured in different forms. By jointly training the detection model with multiple features, the detection model can learn more comprehensive key features to improve the model performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0058] Figure 1 The figure is an overall flow chart of a method for detecting dysphagia patients based on tensor decomposition in a preferred embodiment of the present invention.

[0059] Figure 2 This is a schematic diagram of the construction principle of a training feature set based on a spectrogram in a preferred embodiment of the present invention.

[0060] Figure 3 The figure is a schematic diagram of constructing two vowel mixed tensors in a specific application example of the present invention.

[0061] Figure 4 The figure is a schematic diagram of constructing a vowel mixture tensor in a specific application example of the present invention.

[0062] Figure 5 It is a schematic diagram of the overall composition of a swallowing disorder patient detection system based on tensor decomposition in a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0063] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements may be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0064] In order to further explore the key features that characterize dysphagia, an embodiment of the present invention provides a method for detecting dysphagia patients based on tensor decomposition. The method is based on vowel mixture tensor and Tucker decomposition, assists in detecting dysphagia patients, and improves the detection progress.

[0065] Specifically, Figure 1 As shown, the method for detecting dysphagia patients based on tensor decomposition provided in this embodiment may include the following operations:

[0066] S1, a collection device (such as a microphone or other collection device) obtains voice data of a subject, wherein the subject includes: a swallowing disorder patient and a normal person, wherein the voice data of each subject includes any one or any multiple vowel data;

[0067] S2, preprocessing the speech data, cutting out irrelevant information in the speech data and performing amplitude normalization operation; this step enables the swallowing disorder detection model to pay more attention to the characteristics of the swallowing disorder itself, so as to improve the accuracy of subsequent detection;

[0068] S3, construct different forms of vowel mixture tensors based on the spectrogram for each subject’s speech data;

[0069] S4, extract features from each vowel mixture tensor based on orthogonal constraint Tucker decomposition to form a training feature set;

[0070] S5, training a dysphagia detection model based on the training feature set;

[0071] S6, detecting the speech data to be detected based on the dysphagia detection model, and obtaining classification results of dysphagia patients and normal people.

[0072] In some preferred implementations, the above S2, irrelevant information in the voice data, mainly includes the following information: some noise at the beginning and end of the recording.

[0073] In order to extract key features from speech signals for better classification and detection, in some preferred implementations, the above S3 mainly calculates the following two spectrograms according to each speech signal: STFT spectrogram and MFCC spectrogram. Based on these two spectrograms, three forms of vowel mixture tensors are constructed, such as Figure 3 , Figure 4 As shown, they are: vowel mixture tensor based on MFCC spectrogram, vowel mixture tensor based on the entire STFT spectrogram, and band vowel mixture tensor based on different frequency bands of the STFT spectrogram.

[0074] Furthermore, two types of spectrograms, STFT and MFCC, are calculated, and three forms of vowel mixture tensors are constructed, which may further include the following operations:

[0075] S301, performing short-time Fourier transform on the pre-processed speech data to obtain an STFT spectrogram, and at the same time, extracting Mel frequency cepstral coefficients MFCC from the speech data; specifically:

[0076] Perform short-time Fourier transform on the speech data to convert the one-dimensional speech signal into a two-dimensional STFT spectrogram; map the STFT spectrogram to the Mel frequency scale to form a Mel spectrum; take the logarithm of the Mel spectrum, perform discrete cosine transform to obtain MFCC features, and convert the one-dimensional speech signal into a two-dimensional matrix containing MFCC coefficients that vary with time to obtain an MFCC spectrogram; where:

[0077] ①STFT spectrogram: Perform short-time Fourier transform on the speech data to convert the one-dimensional speech signal into a two-dimensional STFT spectrogram. Short-time Fourier transform is to divide the original speech data into multiple short-time signals. The signal can be considered stable within the short-time signal. Then, FFT transform is performed on each short-time signal. FFT transform is a faster form of discrete Fourier transform.

[0078] ②MFCC spectrogram: The STFT spectrum is mapped to the Mel frequency scale to form a Mel spectrum, thereby simulating the nonlinear perception of the human auditory system to the sound frequency. Then, the Mel spectrum is logarithmically transformed and discrete cosine transformed to obtain the MFCC features, thereby converting the one-dimensional speech signal into a two-dimensional matrix, which reflects the MFCC coefficients that change over time;

[0079] S302, after the vowel speech data of each subject are calculated, an STFT spectrogram is obtained, and a three-dimensional vowel mixture tensor based on the STFT spectrogram is constructed by splicing in the vowel dimension; after the vowel speech data of each subject are calculated, an MFCC spectrogram is obtained, and a three-dimensional vowel mixture tensor based on the MFCC spectrogram is also constructed by splicing in the vowel dimension; wherein:

[0080] In a specific application example, the speech data of patients with dysphagia and normal people include three vowel data of " / a / ", " / o / ", and " / e / ". On this basis:

[0081] ① Construct vowel mixture tensor based on MFCC spectrogram, including: calculating three MFCC spectrograms for each subject's " / a / ", " / o / ", " / e / " speech data respectively, and constructing the three obtained MFCC spectrograms into a three-dimensional tensor in the vowel dimension Among them, I m represents the number of vowel signals used, which is 3 in the present embodiment, Jm represents the number of MFCC feature vectors, which is 20 in the present embodiment; K m Represents the time dimension, that is, the total number of frames; specifically, the MFCC spectrograms of the three vowels are used as horizontal slices of the vowel mixture tensor M(v,∶,∶), v=1,2,3, where v=1,2,3 represent the MFCC spectrograms of the vowels " / a / ", " / o / ", and " / e / ", respectively;

[0082] ② Constructing vowel mixture tensor based on the entire STFT spectrogram, including: for each subject's " / a / ", " / o / ", " / e / " speech data, respectively, the STFT spectrogram is calculated by using the Hamming window, and the vowel mixture tensor based on the entire STFT spectrogram is constructed. The construction of S is similar to the process of constructing vowel mixture tensors based on MFCC spectrograms. The three dimensions of tensor S are vowels, frequency, and time, where I s represents the number of vowel signals used, which is 3 in the present embodiment, J s represents the total number of frequency components within the frequency range, which is 20 in the present embodiment; K s represents the time dimension, i.e., the total number of frames, and the horizontal slices of S represent the correlation between frequency and time within a single vowel; Figure 3 As shown;

[0083] S303, for the three-dimensional vowel mixture tensor constructed based on the STFT spectrogram, divide the tensor in the frequency dimension to obtain n three-dimensional frequency band sub-tensors; wherein:

[0084] ③ Construct vowel mixture tensors based on different frequency bands of the STFT spectrogram, including: obtaining a vowel mixture tensor S based on the entire STFT spectrogram for each subject, and dividing it into n frequency band sub-tensors B in the frequency dimension j , j = 1, ..., n, n can be preferably 4; wherein is the number of frequency components within the frequency band range of the i-th sub-tensor; specifically, the vowel dimension of the two-dimensional frequency band matrices within the same frequency range of the three STFT spectrograms is concatenated to construct a new three-dimensional tensor; Figure 4 shown.

[0085] It should be noted that, for the case of including other vowel data, the method of constructing a three-dimensional vowel mixture tensor is the same and will not be repeated here.

[0086] In order to capture the potential relationship between vowels and more complex patterns from each constructed vowel mixture tensor, in some preferred embodiments, the above S4, extracting features from each vowel mixture tensor based on orthogonal constrained Tucker decomposition to form a training feature set, may further include the following operations:

[0087] The feature extraction method based on orthogonal constraint Tucker decomposition is adopted. The vowel mixture tensors of different forms are sent to the Tucker decomposition model with orthogonal constraint, and the corresponding core tensor is obtained by setting the appropriate rank. This feature extraction method processes the correlation and interaction in high-order data to form a feature set. Among them:

[0088] The Tucker decomposition model with orthogonal constraints is constructed as follows:

[0089]

[0090] Where X∈R I×J×K is a three-dimensional tensor, representing the unified representation of three forms of vowel mixture tensors, R represents a matrix, I, J, K represent the sizes of the three dimensions of the three-dimensional tensor X respectively; G∈R P×Q×L represents the core tensor obtained after Tucker decomposition of the vowel mixture tensor, which contains the relationship between different modes, where P, Q, and L represent the sizes of the three dimensions of the core tensor G respectively; A∈R I×R , B∈R J×Q , C∈R K×L They represent the factor matrices in the three modes obtained after Tucker decomposition, where ×1, ×2, and ×3 represent the product between the tensor and the matrix; g pql 、a p , b q 、c l Represent the specific elements of G, A, B, and C respectively; ○ represents the vector outer product;

[0091] The setting of the rank of the core tensor is performed by searching within a certain interval, and finally a relatively optimal rank setting within the interval is obtained, and then the final core tensor is obtained;

[0092] The corresponding core tensors obtained from different forms of vowel mixture tensors are flattened into vectors to obtain the corresponding feature sets;

[0093] Based on the corresponding feature set, a training feature set is further constructed.

[0094] Specifically, the above S4 further includes the following operations:

[0095] S401, inputting the constructed vowel mixture tensors of different forms into the Tucker decomposition model with orthogonal constraints respectively, and obtaining the core tensor after rank optimization;

[0096] S402, according to the core tensor result output by the Tucker decomposition model of each vowel mixture tensor, flatten it into a vector and perform feature selection to obtain respective feature sets; wherein, feature selection mainly includes the following principles and methods:

[0097] First, coarse feature selection was performed using the t-test method to distinguish dysphagia patients from healthy subjects based on overall distribution;

[0098] Secondly, refined features are obtained through the exact matching analysis method. Specifically, the features of each range are calculated according to the distribution of the original data to alleviate the impact of unbalanced data, and the threshold settings are adjusted to filter the features.

[0099] S403, based on the respective feature sets, construct training feature sets, such as Figure 2 As shown, including:

[0100] ① The feature sets obtained by the three different forms of vowel mixture tensors after Tucker decomposition model are used as three training feature sets after feature selection;

[0101] ② The final training feature set is obtained by concatenating and fusing the feature set obtained from the vowel mixture tensor based on the entire STFT spectrogram and the feature set obtained from the vowel mixture tensor based on the MFCC spectrogram; the training feature set contains effective key distinguishing features in both the STFT spectrogram and the MFCC spectrogram;

[0102] ③ The feature sets obtained from the vowel mixture tensors constructed based on different STFT spectrogram frequency bands are concatenated and fused with the feature sets obtained from the vowel mixture tensors based on the MFCC spectrogram to obtain the final training feature set, which contains the information of the entire frequency band; the training feature set also contains the effective key distinguishing features in the STFT spectrogram and the MFCC spectrogram;

[0103] ④ Multiple feature sets obtained by dividing the vowel mixed tensor constructed based on different frequency bands of the STFT spectrogram into n frequency band sub-tensors are concatenated and fused to obtain the final training feature set;

[0104] ⑤ The feature set obtained by the vowel mixture tensor based on the MFCC spectrogram and the training feature set obtained in ④ are concatenated and fused to obtain the final training feature set.

[0105] Specifically, the feature sets obtained from the vowel mixture tensors constructed based on different frequency bands of the STFT spectrogram are fused together to include the information of the entire frequency band; the feature sets obtained from the vowel mixture tensors constructed based on different frequency bands of the STFT spectrogram and the feature sets obtained from the vowel mixture tensors based on the STFT spectrogram are respectively fused with the feature sets obtained from the vowel mixture tensors based on the MFCC spectrogram to simultaneously include the effective key distinguishing features in the STFT spectrogram and the MFCC spectrogram.

[0106] In some preferred embodiments, the above S5, dysphagia detection model, can adopt common machine learning models, such as SVM, etc. In order to improve the detection performance under small samples, the present invention preferably adopts a polynomial kernel SVM model, which can increase the complexity of the model; the machine learning model is trained using a training feature set and data annotation to obtain a dysphagia detection model.

[0107] In some preferred implementations, the above S5 may further include the following operations:

[0108] The test speech dataset is tested based on the detection model to obtain the classification results of patients with dysphagia and normal people, and the detection model is verified.

[0109] The method for detecting dysphagia patients based on tensor decomposition provided in the above-mentioned embodiment of the present invention obtains multiple vowel speech data of the subject and performs dysphagia patient detection, which can safely and effectively detect whether the current subject has dysphagia, and is convenient for doctors to diagnose and treat and follow up; constructing a three-dimensional vowel mixture tensor based on the spectrogram through multiple vowel data can capture the key acoustic features in each vowel data while deeply capturing the potential correlation between the three vowels; by adopting orthogonal constrained Tucker decomposition to perform dimensionality reduction and feature extraction operations on the three-dimensional vowel mixture tensor, the obtained features are more representative and more critical, which helps to improve the performance of the dysphagia detection model.

[0110] The technical solution provided by the above embodiment of the present invention is further described in detail below in conjunction with a specific application example.

[0111] In this specific application example, speech data of patients with dysphagia and normal people are collected through ordinary microphones, such as directly using an ordinary microphone with digital 16-bit sampling and a sampling rate of 44.1kHz to record the speech data of each participant. All data are stored as waveform data. During the recording process, all participants are asked to read out each required content loudly and individually according to the doctor's instructions.

[0112] The speech content collected from each participant is three kinds of vowel data " / a / ", " / o / ", " / e / ". In recent years, continuous vowel signals have been used for swallowing disorder screening and diagnosis, and have achieved good application results. For three different vowel signals (i.e. " / a / ", " / o / ", " / e / "), they reflect different phonation positions and phonation conditions, so different speech feature information can be captured. Among them, the subjects were asked to read the corresponding content as long as possible.

[0113] The preprocessing of the collected speech data mainly includes cutting out irrelevant information in the speech data and amplitude normalization operations, so that the dysphagia patient detection model can focus more on the characteristics of the dysphagia itself, so as to improve the accuracy of subsequent detection.

[0114] The classic features obtained for each speech data are STFT spectrogram and MFCC spectrogram. STFT spectrogram and MFCC spectrogram are widely used to extract key features from speech signals. Among them, MFCC spectrogram contains fundamental frequency information that can distinguish healthy participants from dysphagia patients. In addition, dysphagia patients usually show significant fluctuations in their voice and amplitude and frequency, which can be described in STFT spectrogram, which shows the changes in frequency and amplitude over time.

[0115] Based on the above process, in this specific application example, a professional doctor further performs a swallowing disorder assessment on each subject, wherein the assessment method adopts the clinical classic Kubota drinking test, and finally obtains the labeled data. Specifically, the label of a normal person is set to 0, and the label of a swallowing disorder patient is set to 1. Three vowel data are collected for each subject, " / a / ", " / o / ", and " / e / ".

[0116] In this specific application example, for each subject, the three vowel data collected, " / a / ", " / o / ", and " / e / ", are cropped and amplitude normalized, and then spectrograms are calculated respectively, where the cropping length is 2s.

[0117] In this specific application example, constructing a feature set based on tensors is an important innovative step, including:

[0118] ① Feature set based on MFCC spectrogram: First, extract the Mel frequency cepstral coefficients MFCC from the three vowel data. In this application example, the number of extracted MFCC features is 20, and the number of frames in the time dimension is 87. For the three MFCC spectrograms obtained from the same subject's speech data, they are concatenated in the speech dimension to construct a three-dimensional vowel mixture tensor Among them I m represents the number of vowel signals used, which is 3 in this work; J m represents the number of MFCC feature vectors, which is 20 in this work; K mRepresents the time dimension, that is, the total number of frames is 87. Then, each constructed vowel mixture tensor based on the MFCC spectrogram is input into the Tucker decomposition model. Since the rank of the core tensor finally obtained needs to be set for optimization, a simple manual setting method is adopted in this application example. Specifically, the rank range of the vowel mixture tensor based on the MFCC spectrogram in the Tucker decomposition is 3×6×6 to 3×20×20, and the ranks of the latter two dimensions are the same and the interval of each value is 2. For the core tensor obtained after the Tucker decomposition model, it is flattened into a vector and input into the feature selection module to finally obtain a feature set for subsequent training and verification of the swallowing disorder patient detection model. Specifically, in this application example, the optimal rank in the range obtained after rank optimization is 3×10×10.

[0119] ② Feature set based on STFT spectrogram: First, the STFT spectrogram of each speech signal is calculated by using a Hamming window with a duration of approximately 46 milliseconds and a hop length of 23 milliseconds. The number of discrete Fourier points and frames is 2048 and 87, respectively. The entire vowel mixture tensor based on the STFT spectrogram The construction process of S is similar to that of the vowel mixture tensor based on MFCC spectrogram. The three modes of tensor S are vowel, frequency and time, where I s and K s Represents vowel and time dimension respectively, which have the same meaning as in the previous tensor M, J s is the total number of frequency components in the frequency range, which is 1025. The horizontal slices of S represent the correlation between frequency and time within a single vowel. The rank range of the vowel mixture tensor based on the STFT spectrogram in the Tucker decomposition is 3×16×16 to 3×30×30, and the ranks of the last two dimensions are 2 each time. Similarly, the core tensor obtained by the Tucker decomposition model of the vowel mixture tensor based on the STFT spectrogram is flattened into a vector and input into the feature selection module to obtain the feature set of the final input detection model. Specifically, in this application example, the optimal rank in the range obtained after rank optimization is 3×24×24.

[0120] ③ Feature set based on STFT spectrograms of different frequency bands: constructed by vowel mixture tensor based on STFT spectrogram. In order to explore the impact of dividing the entire STFT spectrogram into multiple frequency bands on feature mining, the entire three-dimensional STFT spectrogram vowel mixture tensor is divided into four frequency band sub-tensors B in the frequency dimension. j , j = 1, ..., 4, where is the number of frequency components within the frequency band of the ith sub-tensor, where j=1, the frequency range of the sub-tensor is 1 to 256, j=2, the frequency range of the sub-tensor is 257 to 512, j=3, the frequency range of the sub-tensor is 513 to 768, and j=4, the frequency range of the sub-tensor is 769 to 1025. For each frequency band sub-tensor, similarly, the Tucker decomposition module is used to mine key features. The rank of each frequency band sub-tensor in Tucker decomposition ranges from 3×8×8 to 3×40×40, and the rank of the two dimensions is taken at an interval of 4 each time. The obtained core tensor is flattened into a vector and then input into the feature selection module to obtain the feature set of the final input detection model. Specifically, in this application example, the optimal ranks within the ranges obtained after rank optimization for different frequency band sub-tensors are different. Specifically, when the four sub-tensors are divided from low to high frequency, the optimal rank of the first sub-tensor is 3×24×24, the optimal rank of the second frequency band sub-tensor is 3×12×12, the optimal rank of the third frequency band sub-tensor is 3×36×36, and the optimal rank of the fourth frequency band sub-tensor is 3×40×40.

[0121] ④ Fusion feature set based on features of four different frequency band sub-tensors: The core tensors under the optimal rank within the range of the four frequency band sub-tensors obtained in ③ above are constructed into four feature sets, and the features learned based on each frequency band are different, so the four feature sets contain different key features for swallowing disorder detection. Therefore, the optimal feature sets within the range obtained from the four frequency band sub-tensors are spliced ​​and fused into a new feature set to make full use of the detailed features captured in each frequency band, and explore whether the frequency division method of multiple frequency bands can better focus on feature extraction within each frequency band compared to the entire spectrogram, and obtain more detailed features.

[0122] ⑤ A fused feature set based on the entire STFT spectrogram and MFCC spectrogram features: The kernel tensor based on the optimal rank of the MFCC spectrogram and the features based on the optimal rank of the STFT spectrogram obtained in the above ① and ② are respectively concatenated into a long vector as the features of the input feature selection module, and the feature set after feature selection is used as the feature set of the final input detection model.

[0123] ⑥ Fusion feature set based on STFT spectrograms and MFCC spectrogram features of different frequency bands: In the above ① and ③, the feature set under the optimal rank based on MFCC spectrogram and the feature set under the optimal rank based on four sub-tensors of STFT spectrograms of different frequency bands are obtained respectively. Similarly, the long vector obtained after splicing and fusion is input into the feature selection module to obtain the feature set of the final input detection model. Specifically, the features obtained from the four frequency band sub-tensors are fused with the features obtained from the MFCC spectrogram to obtain four new feature sets.

[0124] ⑦ Four fused feature sets based on different frequency band sub-tensors and MFCC spectrogram features: The fused feature sets of the four frequency band sub-tensors obtained above at the optimal rank and the feature set of the MFCC spectrogram obtained at the optimal rank are concatenated into a new feature set.

[0125] In this specific application example, the swallowing disorder patient detection model is constructed based on machine learning, and the structure can be a support vector machine (SVM), which is specifically implemented using the Matlab language. In addition, a support vector machine with a polynomial kernel is used for classification tasks. The increased model complexity of the support vector machine of the polynomial and has a better fitting effect on the data in this application. In order to find an SVM model with better performance, a search is performed within a certain range. Specifically, the degree of the polynomial kernel is in the range of 1 to 4, and the size of the polynomial kernel is in the range of 1 to 10. Among them, the detection model is used on the constructed feature set for training and verification to evaluate the effectiveness and criticality of feature sets constructed in different ways.

[0126] This specific application example has been tested for a certain amount of time, and the following test results are obtained:

[0127] The speech data of three vowels of 234 subjects were used as the data set, of which 176 subjects were used as the training set, including 60 normal subjects and 116 dysphagia patients. The speech data of 58 subjects were used as the test set, including 20 normal subjects and 38 dysphagia patients. The indicators used are as follows:

[0128]

[0129]

[0130] Among them, TP, TN, FP, and FN are true positive, true negative, false positive, and false negative samples, respectively;

[0131] In addition, the area under the ROC curve, i.e., AUC, was calculated as another indicator.

[0132] After experiments on multiple proposed feature sets, the best result was obtained when the features of the third frequency band sub-tensor based on the STFT spectrogram were selected and fused with the features based on the MFCC spectrogram. The best result is shown in Table 1:

[0133] Table 1

[0134]

[0135] Through the above specific application examples and quantitative test results, it can be seen that the method for detecting dysphagia patients based on tensor decomposition provided in the above embodiment of the present invention is based on voice implementation, is simple to operate, has low cost, and has no technical threshold when used. It can effectively obtain the detection results of dysphagia patients, and then can safely and effectively detect dysphagia patients, which is convenient for doctors to follow up.

[0136] Based on the same inventive concept, an embodiment of the present invention further provides a swallowing disorder patient detection system based on tensor decomposition.

[0137] Specifically, Figure 5 As shown, the swallowing disorder patient detection system based on tensor decomposition provided in this embodiment may include the following modules:

[0138] A data acquisition module, which is used to respectively acquire speech data of dysphagia patients and normal people, wherein the speech data includes any one or more types of vowel data;

[0139] A data processing module, which is used to pre-process the speech data, cut out irrelevant information in the speech data and perform amplitude normalization operations;

[0140] The training data construction module is used to obtain the spectrogram of speech data and construct vowel mixture tensors of different forms; send the vowel mixture tensors of different forms to the Tucker decomposition model with orthogonal constraints, set the appropriate rank to obtain the corresponding core tensor; flatten the core tensor into a vector as a feature input, and construct a training feature set after feature selection;

[0141] The swallowing disorder detection module uses the training feature set to train a swallowing disorder detection model, detects the speech data to be detected based on the swallowing disorder detection model, and obtains the classification results of swallowing disorder patients and normal people.

[0142] It should be noted that the steps in the method provided by the present invention can be implemented by using corresponding modules, etc. in the system. Those skilled in the art can refer to the step flow of the method to implement the technical solution of the system, that is, the embodiments in the method can be understood as preferred examples for implementing the system, which will not be elaborated here.

[0143] An embodiment of the present invention further provides a computer terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the processor can be used to execute any method of the above-mentioned embodiments of the present invention, or to run any system of the above-mentioned embodiments of the present invention.

[0144] Optionally, the memory is used to store programs; the memory may include volatile memory (English: volatile memory), such as random-access memory (English: random-access memory, abbreviated: RAM), such as static random-access memory (English: static random-access memory, abbreviated: SRAM), double data rate synchronous dynamic random access memory (English: Double Data Rate Synchronous Dynamic Random Access Memory, abbreviated: DDR SDRAM), etc.; the memory may also include non-volatile memory (English: non-volatile memory), such as flash memory (English: flash memory). The memory is used to store computer programs (such as applications, functional modules, etc. that implement the above method), computer instructions, etc., and the above computer programs, computer instructions, etc. can be partitioned and stored in one or more memories. And the above computer programs, computer instructions, data, etc. can be called by the processor.

[0145] The above-mentioned computer programs, computer instructions, etc. may be stored in one or more memories in partitions, and the above-mentioned computer programs, computer instructions, data, etc. may be called by a processor.

[0146] The processor is used to execute the computer program stored in the memory to implement the various steps of the method or various modules of the system involved in the above embodiments. For details, please refer to the relevant descriptions in the above method and system embodiments.

[0147] The processor and the memory may be independent structures or integrated structures. When the processor and the memory are independent structures, the memory and the processor may be coupled and connected via a bus.

[0148] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it can be used to execute any method of the above embodiments of the present invention, or to run any system of the above embodiments of the present invention.

[0149] Among them, computer-readable media include computer storage media and communication media, wherein the communication media include any media that facilitates the transmission of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general or special-purpose computer. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a user device. Of course, the processor and the storage medium can also be present in a communication device as discrete components.

[0150] Those skilled in the art know that, in addition to implementing the system and its various devices provided by the present invention in a purely computer-readable program code, the system and its various devices provided by the present invention can be made to implement the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices provided by the present invention can be considered as a hardware component, and the devices for implementing various functions included therein can also be considered as structures within the hardware component; the devices for implementing various functions can also be considered as both software modules for implementing the method and structures within the hardware component.

[0151] The swallowing disorder patient detection method and system based on tensor decomposition provided in the above-mentioned embodiment of the present invention use the speech data of each subject (which can form a speech database) to construct three forms of vowel mixture tensors based on the spectrogram. This high-order tensor constructed by multiple vowels can retain the interaction information between them. Among them, the vowel mixture tensor constructed based on the MFCC spectrogram can capture the fundamental frequency information, the vowel mixture tensor constructed based on the entire STFT spectrogram can capture the stable characteristics of the speech, and the vowel mixture tensor based on the STFT spectrogram sub-band can capture more detailed features within the current frequency band.

[0152] The above-mentioned embodiments of the present invention provide a method and system for detecting dysphagia patients based on tensor decomposition, which extracts features of the corresponding vowel mixture tensor based on the Tucker decomposition model, wherein the rank of the core tensor is obtained by optimizing within a certain range, and uses the obtained feature set to construct a training feature set and perform data annotation. The dysphagia patient detection model based on a support vector machine is trained based on a support vector machine, and the results of feature sets in different forms are compared to obtain a trained model, which is used to directly predict whether a person suffers from dysphagia through a speech signal.

[0153] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various modifications or variations within the scope of the claims, which does not affect the essence of the present invention. The above preferred features can be used in any combination without conflicting with each other.

Claims

1. A method for detecting dysphagia patients based on tensor decomposition, characterized in that: include: Acquire speech data of dysphagia patients and normal people respectively, wherein the speech data includes any one or more types of vowel data; Preprocessing the speech data, cutting out irrelevant information in the speech data and performing amplitude normalization operation; Obtaining a spectrogram of the speech data and constructing vowel mixture tensors in different forms; The different forms of vowel mixture tensors are sent to a Tucker decomposition model with orthogonal constraints, and a suitable rank is set to obtain a corresponding core tensor; Flatten the kernel tensor into a vector as feature input, construct a training feature set after feature selection to train a swallowing disorder detection model; The speech data to be detected is detected based on the dysphagia detection model to obtain classification results of dysphagia patients and normal people.

2. The method for detecting dysphagia patients based on tensor decomposition according to claim 1, characterized in that: The obtaining of the spectrogram of the speech data comprises: Get STFT spectrogram, including: Performing short-time Fourier transform on the speech data to convert the one-dimensional speech signal into a two-dimensional STFT spectrogram; Get MFCC spectrogram, including: Mapping the STFT spectrogram to a Mel frequency scale to form a Mel spectrum; After taking the logarithm of the Mel spectrum, a discrete cosine transform is performed to obtain MFCC features, and the one-dimensional speech signal is converted into a two-dimensional matrix containing MFCC coefficients that vary with time to obtain an MFCC spectrogram; The construction of different forms of vowel mixture tensors includes: Construct vowel mixture tensor based on MFCC spectrogram, including: The MFCC spectrogram is calculated for each subject’s vowel data, and then the obtained MFCC spectrogram is constructed into a three-dimensional tensor Among them, I m Represents the number of vowel data used; J m Represents the number of MFCC feature vectors; K m Represents the time dimension, that is, the total number of frames; Construct a vowel mixture tensor based on the entire STFT spectrogram, including: The STFT spectrogram is calculated for each subject's vowel data, and then the obtained STFT spectrogram is constructed into a three-dimensional tensor Among them, I s Represents the number of vowel data used; J s Represents the total number of frequency components within the frequency range; K s Represents the time dimension, that is, the total number of frames; Construct vowel mixture tensors based on different frequency bands of STFT spectrogram, including: After constructing the vowel mixture tensor S for each subject based on the entire STFT spectrogram, it is divided into n frequency band sub-tensors B in the frequency dimension. j , j=1,...,n.

3. The method for detecting dysphagia patients based on tensor decomposition according to claim 1, characterized in that: The different forms of vowel mixture tensors are sent to the Tucker decomposition model with orthogonal constraints, and a suitable rank is set to obtain a corresponding core tensor, including: The Tucker decomposition model with orthogonal constraints is constructed as follows: Where X∈R I×J×K is a three-dimensional tensor, representing the unified representation of three forms of vowel mixture tensors, R represents a matrix, I, J, K represent the sizes of the three dimensions of the three-dimensional tensor X respectively; G∈R P×Q×L represents the core tensor obtained after Tucker decomposition of the vowel mixture tensor, which contains the relationship between different modes, where P, Q, and L represent the sizes of the three dimensions of the core tensor G respectively; A∈R I×P , B∈R J×Q , C∈R K×L They represent the three factor matrices obtained after Tucker decomposition, where ×1, ×2, and ×3 represent the product between the tensor and the matrix; g pql 、a p 、b q 、c l Represent the specific elements of G, A, B, and C respectively; ○ represents the vector outer product; By searching within a certain interval, the optimal rank setting within the interval is obtained, and then the final kernel tensor is obtained.

4. The method for detecting dysphagia patients based on tensor decomposition according to claim 1, characterized in that: Flattening the core tensor into a vector as a feature input includes: The corresponding kernel tensors obtained from different forms of vowel mixture tensors are flattened into vectors respectively, and the corresponding feature sets are obtained as feature inputs.

5. The method for detecting dysphagia patients based on tensor decomposition according to claim 1, characterized in that: The feature selection comprises: The coarse feature selection was performed by t-test method, and the obtained coarse feature training feature set was used to distinguish dysphagia patients from healthy people based on the overall distribution; The exact matching analysis method is used to calculate the features of each range according to the distribution of the original data, and the threshold setting is adjusted to filter the features. The obtained fine feature training feature set is used to reduce the impact of unbalanced data.

6. The method for detecting dysphagia patients based on tensor decomposition according to claim 1, characterized in that: The constructing of the training feature set comprises: The feature sets obtained based on different forms of vowel mixture tensors are respectively used as the first training feature sets; The feature set obtained by the vowel mixture tensor constructed based on the STFT spectrogram and the feature set obtained by the vowel mixture tensor constructed based on the MFCC spectrogram are concatenated and fused to obtain a second training feature set; Multiple feature sets obtained from vowel mixture tensors constructed based on different frequency bands of the STFT spectrogram are concatenated and fused with feature sets obtained from vowel mixture tensors based on the MFCC spectrogram to obtain a third training feature set; Multiple feature sets obtained from vowel mixture tensors constructed based on different frequency bands of the STFT spectrogram are concatenated and fused to obtain a fourth training feature set; The fourth training feature set and the feature set obtained by the vowel mixture tensor constructed based on the MFCC spectrogram are concatenated and fused to obtain the fifth training feature set.

7. The method for detecting dysphagia patients based on tensor decomposition according to claim 1, characterized in that: The training of a swallowing disorder detection model comprises: Provide a support vector machine as an initial model; The initial model is trained using the training feature set and performing data annotation to obtain a dysphagia detection model.

8. A swallowing disorder patient detection system based on tensor decomposition, characterized in that: include: A data acquisition module, the module is used to respectively acquire speech data of dysphagia patients and normal people, wherein the speech data includes any one or more vowel data; A data processing module, which is used to pre-process the voice data, cut out irrelevant information in the voice data and perform amplitude normalization operations; A training data construction module, which is used to obtain the spectrogram of the speech data and construct vowel mixture tensors of different forms; send the vowel mixture tensors of different forms to a Tucker decomposition model with orthogonal constraints, set a suitable rank to obtain a corresponding core tensor; flatten the core tensor into a vector as a feature input, and construct a training feature set after feature selection; A swallowing disorder detection module uses the training feature set to train a swallowing disorder detection model, detects the speech data to be detected based on the swallowing disorder detection model, and obtains classification results of swallowing disorder patients and normal people.

9. A computer terminal comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it can be used to perform the method described in any one of claims 1 to 7, or to run the system described in claim 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it can be used to perform the method described in any one of claims 1 to 7, or to run the system described in claim 8.

Images