Transform-based traditional Chinese medicine sound diagnosis physique identification model and method

Abstract

The invention provides a traditional Chinese medicine sound diagnosis constitution identification model and method based on Transform, and the method comprises the steps: collecting the voice data of a subject, and the voice data comprises vowels and keywords; and preprocessing the voice data to obtain a Mel spectrogram. And inputting the Mel spectrogram into a traditional Chinese medicine sound diagnosis physique identification model based on Transform for physique identification to obtain a physique type. The traditional Chinese medicine sound diagnosis physique identification model based on the Transform is a trained neural network model, and the traditional Chinese medicine sound diagnosis physique identification model based on the Transform is adopted to judge whether the physique type of a subject is mild or damp-heat, so that the problem of subjective deviation caused by adopting a question and answer form to judge the physique type can be solved; the physique type of the subject is judged under a unified standard, so that the identification of the physique type of the subject is relatively high in objectivity. In addition, the voice data is based on vowels and keywords, so that the correlation between the voice data and physique identification is stronger, and the accuracy of physique type identification can be improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent traditional Chinese medicine technology, and in particular to a Transformer-based traditional Chinese medicine acoustic diagnosis constitution identification model and method. Background Art

[0002] At present, most TCM constitutions are determined in the form of questions and answers. This method is affected by the subjective factors of the subjects and leads to deviations, thus increasing the workload of clinicians. Some scholars have proposed using objective signals such as sound signals to identify TCM constitutions. As an objective indicator, sound signals have certain advantages in judging TCM constitutions based on pronunciation. The acoustic characteristics of a peaceful constitution are consistent with the characteristics of harmonious tone and soft and mellow voice, while the speech speed of a damp-heat constitution is faster.

[0003] There is little research on the correlation between the subject's voice signal and TCM constitution in the existing technology. There is a lack of methods from collecting voice data to outputting TCM constitution classification results. In addition, the existing technology cannot improve the accuracy of identifying TCM constitution while eliminating subjective bias. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the purpose of the present invention is to provide a Transformer-based TCM acoustic diagnosis constitution identification model and method, which inputs the collected voice data into a neural network model, and judges the TCM constitution of the subject according to the sound characteristics of the voice data. It has the characteristics of automated calculation, thereby eliminating subjective bias and improving the accuracy of identifying TCM constitution.

[0005] A method for identifying constitution by TCM acoustic diagnosis based on Transformer, comprising:

[0006] collecting speech data of a subject, wherein the speech data includes vowels and keywords;

[0007] Preprocessing the speech data to obtain a mel-spectrogram;

[0008] The mel-spectrogram is input into a Transformer-based TCM acoustic diagnosis constitution identification model for constitution identification to obtain the constitution type.

[0009] In a preferred technical solution of the present invention, the preprocessing of the speech data to obtain a mel-spectrogram includes:

[0010] Performing noise reduction processing on the voice data to obtain noise-reduced voice; wherein the duration of the voice data is t seconds;

[0011] Eliminate abnormal speech from the noise-reduced speech to obtain speech to be extracted, wherein the abnormal speech is speech whose speech noise is greater than a noise threshold;

[0012] Extract the mel-spectrogram of the speech to be extracted.

[0013] In a preferred technical solution of the present invention, the step of extracting the mel-spectrogram of the speech to be extracted includes:

[0014] The speech to be extracted is subjected to pre-emphasis, framing and windowing processing in sequence to obtain windowed speech; wherein the windowing processing is to apply a Hamming window to each frame every 10 ms, and the length of the Hamming window is 25 ms;

[0015] Performing a fast Fourier transform on the windowed speech to obtain a frequency domain signal;

[0016] Inputting the frequency domain signal into a Mel filter bank for bandpass filtering to obtain a bandpass filtered signal;

[0017] The signal after bandpass filtering is subjected to logarithmic operation and DCT transformation in sequence to obtain a Mel spectrum diagram.

[0018] In a preferred technical solution of the present invention, the step of inputting the frequency domain signal into a Mel filter bank for bandpass filtering to obtain a bandpass filtered signal comprises:

[0019] Inputting the frequency domain signal into N Mel filters for bandpass filtering respectively to obtain N spectral values;

[0020] The N spectral values ​​are added together to obtain a bandpass filtered signal.

[0021] In a preferred technical solution of the present invention, after obtaining the Mel-spectrogram, the method further includes:

[0022] Randomly divide the Mel-spectrogram into M groups;

[0023] Determine the input order of the M groups of mel-spectrograms.

[0024] In a preferred technical solution of the present invention, the step of collecting the voice data of the subject includes:

[0025] Reducing environmental noise until the environmental noise is less than a noise threshold;

[0026] adjusting microphone parameters, the microphone parameters including a signal-to-noise ratio of the microphone and a distance between the microphone and the subject's lips;

[0027] The subject's speech was collected using Praat software.

[0028] The present invention also provides a Transformer-based TCM voice diagnosis constitution identification model, which is applied to any of the Transformer-based TCM voice diagnosis constitution identification methods described above, and the model includes an input processing module, a linear projection and embedding module, a Transformer encoder module and an output module; the input processing module is used to preprocess the input Mel spectrum map to change the input form of the Mel spectrum map; the linear projection and embedding module is used to perform linear projection on each block and embed the linear projection results into a high-dimensional vector space; the Transformer encoder module is used to extract the global features and context dependencies of the input data, and the output module is used to map the processed features into specific mapping values.

[0029] In a preferred technical solution of the present invention, the input processing module includes a segmentation layer and a linear projection layer, the input end of the segmentation layer receives the Mel spectrum map, and the output end of the segmentation layer is connected to the input end of the linear projection layer; the segmentation layer is used to segment the Mel spectrum map into multiple image blocks, and the linear projection layer is used to expand each of the image blocks to obtain a one-dimensional embedded feature sequence.

[0030] In a preferred technical solution of the present invention, the Transformer encoder module includes a multi-head attention layer and a splicing layer, and the multi-head attention layer is composed of multiple self-attention layers; the self-attention layer is used to perform different forms of linear transformations on the blocks to generate three matrices, and the splicing layer is used to splice the outputs of all the self-attention layers.

[0031] In a preferred technical solution of the present invention, the output module includes a linear layer and a classification layer, the input end of the linear layer is connected to the output end of the Transformer encoder module, and the output end of the linear layer is connected to the input end of the classification layer; the linear layer is used to map the high-dimensional feature vector into a final feature vector, and the classification layer is used to classify the final feature vector to obtain the physical type.

[0032] The beneficial effects of the present invention are:

[0033] The present invention provides a method for identifying constitution of TCM voice diagnosis based on Transformer, including collecting speech data of a subject, wherein the speech data includes vowels and keywords. The speech data is preprocessed so that the speech data is suitable for TCM constitution identification, and a mel-spectrogram is obtained. The mel-spectrogram is input into a TCM voice diagnosis constitution identification model based on Transformer for constitution identification, and a constitution type is obtained. The TCM voice diagnosis constitution identification model based on Transformer is a neural network model after training, and the TCM voice diagnosis constitution identification model based on Transformer is used to judge whether the constitution type of the subject is a peaceful constitution or a damp-heat constitution, which can overcome the problem of subjective bias caused by determining the constitution type in the form of questions and answers, and judge the constitution type of the subject under a unified standard, so that the present invention identifies the constitution type of the subject with strong objectivity. In addition, the speech data of the present invention is based on vowels and keywords, so that the correlation between the speech data and constitution identification is stronger, and the accuracy of identifying the constitution type can be increased. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a constitution classification model diagram of the Transformer-based TCM acoustic diagnosis constitution identification model of the present invention;

[0035] Figure 2 It is a multi-head attention layer network structure diagram of the present invention;

[0036] Figure 3 It is a result diagram of the speech waveform and spectrum of the pronunciation of the damp-heat subject "a" of the present invention;

[0037] Figure 4 This is a result diagram of the speech waveform and spectrum of the peaceful subject "a" of the present invention;

[0038] Figure 5 It is a flow chart of the method for identifying constitution by TCM acoustic diagnosis based on Transformer of the present invention. DETAILED DESCRIPTION

[0039] The preferred embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to make the present invention more thorough and complete, and to fully convey the scope of the present invention to those skilled in the art.

[0040] Example 1

[0041] like Figure 5 As shown, this embodiment provides a method for identifying constitution by TCM acoustic diagnosis based on Transformer, comprising:

[0042] S1: Collecting speech data of a subject, wherein the speech data includes vowels and keywords.

[0043] S2: Preprocess the speech data to obtain a Mel-spectrogram.

[0044] S3: Input the Mel-spectrogram into the Transformer-based TCM acoustic diagnosis constitution identification model to perform constitution identification and obtain the constitution type.

[0045] Step S1 includes the following steps:

[0046] S11: reducing the environmental noise until the environmental noise is less than a noise threshold.

[0047] S12: Adjust microphone parameters, where the microphone parameters include a signal-to-noise ratio of the microphone and a distance between the microphone and the subject's lips.

[0048] S13: using Praat software to collect the subject's voice.

[0049] Before collecting speech, the ambient noise was reduced to below 35 dB. The subjects were relaxed and sat in a natural and comfortable posture. The microphone model used was AKG HSC271, the signal-to-noise ratio of the microphone was 22 dB, and the distance between the subject's lips and the microphone was 5 cm. The Praat software version 6.1.321 was used to collect the subject's speech data. The sampling frequency of the Praat software was 44100 Hz, and the channel was mono.

[0050] Vowels include "a, o, e, i, u", and keywords include "sea, cloud, heart, ten, mouth, moon, east, mud, square, beauty". Multiple speech data composed of vowels and keywords are composed into a benchmark data set for TCM voice constitution recognition.

[0051] Step S2 includes the following steps:

[0052] S21: performing noise reduction processing on the voice data to obtain noise-reduced voice; wherein the duration of the voice data is t seconds.

[0053] S22: removing abnormal speech from the noise-reduced speech to obtain speech to be extracted, wherein the abnormal speech is speech whose speech noise is greater than a noise threshold.

[0054] S23: Extracting the mel-spectrogram of the speech to be extracted.

[0055] Use LMS adaptive filtering or Wiener filtering to reduce the noise of speech data and obtain the noise-reduced speech. If the noise of the noise-reduced speech is greater than the noise threshold, the noise of the speech is too large. If other noise reduction methods are used to further reduce the noise, the speech may be seriously distorted. Therefore, the speech whose noise after noise reduction is greater than the noise threshold is judged as abnormal speech and eliminated.

[0056] Mel cepstral coefficients are acoustic features related to the human auditory system. Studies on the human auditory mechanism have found that when two tones with similar frequencies are emitted at the same time, people can only hear one tone. Critical bandwidth refers to the bandwidth boundary where people's subjective feelings suddenly change. When the frequency difference between two tones is less than the critical bandwidth, people will hear the two tones as one, which is called the shielding effect. Mel cepstral coefficients can be used to measure critical bandwidth.

[0057] The Transformer-based TCM voice diagnosis and constitution identification model is a trained neural network model. The mel-spectrogram representing the voice characteristics of the subject is input into the Transformer-based TCM voice diagnosis and constitution identification model, and the constitution type is output. The constitution types include balanced constitution and damp-heat constitution.

[0058] After the Transformer-based TCM voice diagnosis constitution identification model (CIT) outputs the constitution type, ResNet, DenseNet and DenseNet-Attention models are used as control models. The three control models and the Transformer-based TCM voice diagnosis constitution identification model are initialized with the weights pre-trained by ImageNet. The performance of the above four recognition models is compared by the five-fold cross-validation method. The accuracy ACC (Accuracy), the area under the ROC curve and the coordinate axis AUC (Area Under Curve), sensitivity Sen (Senstivity) and specificity Spe (Specificity) are used as evaluation indicators. The experimental results are shown in Table 1. The experimental results show that the average accuracy of the five-fold cross-validation is 82.02%. The Transformer-based TCM voice diagnosis constitution identification model improves the classification accuracy by 3.04% and requires the shortest training time.

[0059] Table 1 Comparison results of the TCM acoustic diagnosis constitution identification model based on Transformer and other models

[0060] ACC / % AUC / % Sen / % Spe / % Time / min ResNet 78.98 87.28 77.26 83.96 54 DenseNet 79.92 87.56 77.15 84.66 56 DenseNet-Attention 80.25 88.02 78.05 85.21 52 CIT 82.02 88.74 78.19 85.65 44

[0061] All the research subjects were students from Guangzhou University of Chinese Medicine. The inclusion criteria were that the subjects were over 18 years old, had no acute diseases, no acute pain, and had not taken any medications in the past 3 months. All subjects signed informed consent forms and were evaluated according to the "Classification and Determination of Chinese Medicine Constitution" issued by the Chinese Association of Traditional Chinese Medicine. The internal consistency coefficient of the 9 subscales of the scale was 0.72-0.82, and the performance evaluation of practicality, reproducibility and internal consistency of the scale was good. Each question was scored on a 5-level scale, and the original score and conversion score were calculated, and the constitution type was determined according to the standard. A total of 34 subjects were included in this experiment, and 700 audio recordings were recorded, including 16 with a peaceful constitution and 18 with a damp-heat constitution. The average age of the two groups of subjects was 18-23 years old.

[0062] The present embodiment provides a method for identifying constitution of TCM voice diagnosis based on Transformer, including collecting speech data of a subject, wherein the speech data includes vowels and keywords. The speech data is preprocessed so that the speech data is suitable for TCM constitution identification, and a mel-spectrogram is obtained. The mel-spectrogram is input into the TCM voice diagnosis constitution identification model based on Transformer for constitution identification, and the constitution type is obtained. The TCM voice diagnosis constitution identification model based on Transformer is a neural network model after training, and the TCM voice diagnosis constitution identification model based on Transformer is used to judge whether the constitution type of the subject is a peaceful constitution or a damp-heat constitution, which can overcome the problem of subjective bias caused by determining the constitution type in the form of questions and answers, and judge the constitution type of the subject under a unified standard, so that the present invention identifies the constitution type of the subject with strong objectivity. In addition, the speech data of the present invention is based on vowels and keywords, so that the correlation between the speech data and constitution identification is stronger, and the accuracy of identifying the constitution type can be increased.

[0063] Example 2

[0064] This embodiment provides a method for identifying constitution by TCM acoustic diagnosis based on Transformer. This embodiment only describes the differences from Embodiment 1. The step of extracting the mel-spectrogram of the speech to be extracted includes:

[0065] S231: Pre-emphasize, frame and window the speech to be extracted in sequence to obtain windowed speech; wherein the windowing process is to apply a Hamming window to each frame every 10 ms, and the length of the Hamming window is 25 ms.

[0066] S232: Perform fast Fourier transform on the windowed speech to obtain a frequency domain signal.

[0067] S233: Input the frequency domain signal into a Mel filter bank for bandpass filtering to obtain a bandpass filtered signal.

[0068] S234: performing logarithmic operation and DCT transformation on the bandpass filtered signal in sequence to obtain a Mel-spectrogram.

[0069] Before the extracted speech is windowed, it is pre-emphasized and framed. The purpose of pre-emphasis is to enhance the high-frequency part of the speech signal, make the spectrum of the speech signal flat, and keep the spectrum in the entire frequency band from low frequency to high frequency with the same signal-to-noise ratio. At the same time, it is also to eliminate the effects of the vocal cords and lips during the phonation process, to compensate for the high-frequency part of the speech signal suppressed by the pronunciation system, and thus highlight the high-frequency resonance peak. The speech signal is a non-stationary signal that changes with time. In order to capture the local time-frequency characteristics, the audio signal is divided into several short time frames, and there is a certain overlap between adjacent frames.

[0070] The speech to be extracted with a duration of t seconds is buffered into frames of multiple samples, and a Hamming window with a duration of 25ms is applied to each frame every 10ms. The windowed speech is subjected to a fast Fourier transform, thereby transforming the windowed speech from the time domain to the frequency domain. The Mel filter bank consists of 128 Mel filters. The frequency domain signals are input into 128 Mel filters for bandpass filtering. The 128 Mel filters output 128 spectral values. The 128 spectral values ​​are added together, that is, the output values ​​of the 128 channels of the Mel filter bank are combined to obtain a Mel spectrum map of size 128×100t.

[0071] Logarithmic operation is to calculate the logarithmic energy of each Mel filter output after the spectral signal is processed by the Mel filter bank. Calculating the logarithmic energy simulates the human ear's perception of loudness. The auditory system is more sensitive to changes in energy, and logarithmic processing helps to reduce the impact of signals with larger amplitudes.

[0072] DCT transform is discrete cosine transform. DCT transform is performed on the Mel filter output after logarithmic transformation to obtain Mel cepstral coefficients. Multiple Mel cepstral coefficients form a Mel spectrogram. Mel cepstral coefficients are used to remove the correlation in the speech signal and leave the main information of the spectrum envelope.

[0073] The step of inputting the frequency domain signal into a Mel filter bank for bandpass filtering to obtain a bandpass filtered signal comprises:

[0074] S2331: Input the frequency domain signal into N Mel filters for bandpass filtering to obtain N spectral values.

[0075] S2332: Add the N spectral values ​​to obtain a bandpass filtered signal.

[0076] Compared with the general frequency mechanism, the Mel cepstral coefficient is closer to the auditory mechanism of the human ear. It grows very fast in the low-frequency range, but grows very slowly in the high-frequency range. Each frequency corresponds to a Mel cepstral coefficient.

[0077] The advantages of bandpass filtering are as follows: smoothing the spectrum, eliminating the influence of harmonics, and highlighting the resonance peaks of speech.

[0078] After obtaining the Mel-spectrogram, the method further includes:

[0079] S31': randomly divide the Mel-spectrogram into M groups.

[0080] S32': Determine the input order of the M groups of mel-spectrograms.

[0081] All mel spectrograms were randomly divided into 5 equal parts so that the model could be verified by the 5-fold cross-validation method to determine whether the constitution type of the subject corresponding to the mel spectrogram was a mild constitution or a damp-heat constitution. M groups of mel spectrograms were assigned numbers, and the M groups of mel spectrograms were input into the Transformer-based TCM acoustic diagnosis constitution identification model in sequence according to the numbers to identify the constitution and obtain the constitution type.

[0082] This embodiment extracts the mel-spectrogram of the speech to be extracted, including pre-emphasis, framing and windowing processing of the speech to be extracted in sequence to obtain the windowed speech. Among them, the windowing process is to apply the Hamming window to each frame every 10ms, and the length of the Hamming window is 25ms. The windowed speech is subjected to fast Fourier transform to obtain a frequency domain signal. The frequency domain signal is input into the mel filter group for bandpass filtering to obtain a bandpass filtered signal. The bandpass filtered signal is sequentially subjected to logarithmic operation and DCT transformation to obtain a mel-spectrogram. All mel-spectrograms are divided into 5 equal parts, and the transformer-based traditional Chinese medicine voice diagnosis constitution identification model is verified using the five-fold cross-validation method to determine whether the constitution type of the subject corresponding to the mel-spectrogram is a peaceful constitution or a damp-heat constitution.

[0083] Example 3

[0084] like Figure 1As shown, this embodiment provides a Transformer-based TCM voice diagnosis constitution identification model, which is applied to the Transformer-based TCM voice diagnosis constitution identification method in Example 1 or Example 2, and the model includes an input processing module, a linear projection and embedding module, a Transformer encoder module and an output module; the input processing module is used to preprocess the input Mel spectrum map to change the input form of the Mel spectrum map; the linear projection and embedding module is used to linearly project each block, and embed the linear projection result into a high-dimensional vector space; the Transformer encoder module is used to extract the global features and context dependencies of the input data, and the output module is used to map the processed features into specific mapping values.

[0085] A model to be trained based on the Transformer architecture was constructed, and speech data of multiple subjects were collected. The speech data was divided into a training set, a test set, and a validation set in a ratio of 7:2:1. The training set was input into the model to be trained based on the Transformer architecture, and the model to be trained based on the Transformer architecture was trained. The entire training process included multiple trainings. The loss function of the training adopted the cross entropy loss function, which is often used for classification problems. The cross entropy loss function can measure the difference between two different probability distributions in a random variable. The smaller the value of the cross entropy loss function, the higher the probability that the constitution type predicted by the Transformer-based TCM voice diagnosis constitution identification model is consistent with the actual constitution type of the subject, and the better the prediction effect. During the training process, the value of the cross entropy loss function gradually decreases, and the model parameters such as the learning rate and weight can be continuously optimized, thereby improving the accuracy of constitution identification of the Transformer-based TCM voice diagnosis constitution identification model.

[0086] This example uses the Adam optimizer with a batch size of 48 as an example, and uses the grid search technology provided by ray tune to obtain the optimal hyperparameters. Ray tune is a hyperparameter optimization framework based on Python. Ray tune integrates multiple hyperparameter optimization methods and runs on a distributed computing framework. After experiments, the optimal learning rate of the Transformer-based TCM acoustic diagnosis and constitution identification model is 1e -5 , the optimal weight is 1e -3 .

[0087] Example 4

[0088] This embodiment provides a model for identifying constitution by voice diagnosis in traditional Chinese medicine based on Transformer, which is applied to the method for identifying constitution by voice diagnosis in traditional Chinese medicine based on Transformer in Embodiment 1 or Embodiment 2. This embodiment only describes the differences from Embodiment 3.

[0089] The input processing module includes a segmentation layer and a linear projection layer, the input end of the segmentation layer receives the Mel frequency spectrum map, and the output end of the segmentation layer is connected to the input end of the linear projection layer; the segmentation layer is used to segment the Mel frequency spectrum map into multiple image blocks, and the linear projection layer is used to expand each of the image blocks to obtain a one-dimensional embedded feature sequence.

[0090] The input to the segmentation layer is the Mel spectrogram of the speech signal. The segmentation layer divides the Mel spectrogram of the input t seconds of audio into 12*(100t-16) / 10 small blocks of size 16×16. The overlap between different small blocks in the time and frequency dimensions is 6. Each 16×16 small block is flattened into a one-dimensional embedding feature sequence of size 768 through the linear projection layer to retain more context information. The input processing module converts the spectrogram into multiple small spectrum blocks and provides the multiple small spectrum blocks to the linear projection and embedding module.

[0091] The Transformer encoder module includes a multi-head attention layer and a splicing layer, wherein the multi-head attention layer is composed of multiple self-attention layers; the self-attention layer is used to perform different forms of linear transformations on the blocks to generate three matrices, and the splicing layer is used to splice the outputs of all the self-attention layers.

[0092] like Figure 2 As shown in the figure, when the self-attention mechanism encodes the current position, it will over-focus on its own position, so the transformer architecture adopts a multi-head attention mechanism. The one-dimensional embedded feature sequence is input into the multi-head attention layer, which is composed of multiple self-attention layers. The output of the multi-head attention layer is the concatenation of the outputs of all self-attention layers. Each self-attention layer performs different linear transformations on the input matrix and generates three matrices, namely the query matrix Q, the key matrix K, and the value matrix V.

[0093] The one-dimensional embedded feature sequence after being processed by the input processing module has no position information. A trainable position code of the same size of 768 is added to the embedded features of each small block, so that the embedded features of each small block have the spatial structure information of the spectrum graph.

[0094] Each small block is linearly projected and embedded into a high-dimensional vector space to obtain a high-dimensional feature vector for subsequent processing by the Transformer encoder module. After linear projection, classification marking is performed. Classification marking is to add a special marker at the beginning of the input sequence, such as Figure 1 E[CLS] in.

[0095] The Transformer encoder module extracts the global features and contextual dependencies of the input data through a multi-head self-attention mechanism and a feedforward neural network. The Transformer encoding process includes position encoding, multi-layer Transformer encoders, and residual connections and normalization. Position encoding: Since the Transformer cannot perceive the order of the sequence, position information needs to be added. The position encoding is added to each feature vector to ensure that the model can perceive the order of the input. Transformer encoder model: The model contains 12 layers, each of which contains a multi-head self-attention submodule and a feedforward neural network. The multi-head self-attention submodule is used to capture the dependencies between each block in the input sequence, and the feedforward neural network is used to perform nonlinear transformations on the features of each position to further extract features. Residual connections and normalization: Each submodule is followed by residual connections and normalization operations to help the model learn and converge better. This part is the core part of the Transformer-based TCM acoustic diagnosis and constitution identification model, which uses the self-attention mechanism to extract global features.

[0096] The output module includes a linear layer and a classification layer, the input end of the linear layer is connected to the output end of the Transformer encoder module, and the output end of the linear layer is connected to the input end of the classification layer; the linear layer is used to map the high-dimensional feature vector into a final feature vector, and the classification layer is used to classify the final feature vector to obtain the physical constitution type.

[0097] The output after the Transformer encoder will be passed to the linear layer of the output module. The linear layer maps the high-dimensional feature vector to the final feature vector, and the classification layer converts the output of the encoder into the final constitution type, that is, completes the classification of peaceful constitution and damp-heat constitution. Depending on the specific task, the output of the classification layer can be a classification label, regression value, or other form of result. In this embodiment, the output of the classification layer is taken as a classification label as an example. The output module maps the features processed by the Transformer encoder module to specific predicted values, which are used to reflect whether the constitution of the subject of the data source is a peaceful constitution or a damp-heat constitution.

[0098] The data set used in the present invention includes 34 subjects, with a total of 700 audios recorded, including 16 subjects with a peaceful constitution and 18 subjects with a damp and hot constitution. The age range of the two groups of subjects is 18-23 years old. The voice data is collected to generate a 128×100t Mel spectrogram. Figure 3 This is the result diagram of the speech waveform and spectrum of the pronunciation of "a" by the wet-heat subject. Figure 4 This is the result graph of the speech waveform and spectrum of the pronunciation of "a" by a peaceful subject.

[0099] The Transformer-based TCM acoustic diagnosis and constitution identification model can be roughly divided into four modules: input processing module, linear projection and embedding module, Transformer encoder module and output module. The process is roughly as follows: the mel spectrum graph processed by speech data is divided into blocks; each block is converted into an embedded feature and the position information is added. Features are extracted through multiple self-attention layers and feedforward neural networks, and the linear layer is used to map the high-dimensional feature vector to the final feature vector. The classification layer is used to classify the final feature vector to obtain the constitution type.

[0100] The performance test results of the Transformer-based TCM acoustic diagnosis constitution identification model of the present invention are shown in Table 1 in Example 1 and Table 2 of this embodiment. Two comparative experiments demonstrate the superior performance of Transformer-based constitution identification in TCM constitution identification.

[0101] Table 2 Comparison results of models with and without AudioSet pre-training

[0102] ACC / % AUC / % Sen / % Spe / % ResNet 79.26 88.26 78.35 84.48 DenseNet 80.45 88.93 78.46 84.85 DenseNet-Attention 81.69 89.56 79.21 83.97 CIT-S 82.02 88.74 78.19 85.65 CIT-P 83.33 91.16 80.25 87.03

[0103] The classification results of the Transformer-based TCM acoustic diagnosis and constitution identification model (CIT-S model) that was not pre-trained, the Transformer-based TCM acoustic diagnosis and constitution identification model (CIT-P model) that was pre-trained on the public dataset AudioSet, the ResNet model, the DenseNet model, and the DenseNet-Attention model were compared. The comparison results showed that with the increase in the number of learning rounds, the training set and the loss value gradually decreased and tended to 0, indicating that the model did not have overfitting or underfitting during the learning process. Compared with the CIT-S model, the overall classification accuracy of the CIT-P model increased by 1.31%, proving the effectiveness of the additional AudioSet dataset training.

[0104] The Transformer encoder module of this embodiment includes a multi-head attention layer and a splicing layer, and the multi-head attention layer is composed of multiple self-attention layers. The self-attention layer is used to perform different forms of linear transformations on the blocks to generate three matrices, and the splicing layer is used to splice the outputs of all self-attention layers. The Transformer encoder model contains 12 layers, each of which contains a multi-head self-attention submodule and a feedforward neural network. The multi-head self-attention submodule is used to capture the dependencies between the blocks in the input sequence, and the feedforward neural network is used to perform nonlinear transformations on the features of each position to further extract features.

[0105] Example 5

[0106] A computer device is also provided in an embodiment of the present application, which may be a server, wherein the computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer design is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection.

[0107] This embodiment also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, a method for identifying constitutions by TCM acoustic diagnosis based on Transformer is implemented. It can be understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0108] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.

[0109] The above description is only a preferred embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for identifying constitution by acoustic diagnosis in traditional Chinese medicine based on Transformer, characterized in that: include: collecting speech data of a subject, wherein the speech data includes vowels and keywords; Preprocessing the speech data to obtain a mel-spectrogram; The mel-spectrogram is input into a Transformer-based TCM acoustic diagnosis constitution identification model for constitution identification to obtain the constitution type.

2. According to the Transformer-based TCM acoustic diagnosis constitution identification method of claim 1, it is characterized in that: The preprocessing of the speech data to obtain a Mel-spectrogram includes: Performing noise reduction processing on the voice data to obtain noise-reduced voice; wherein the duration of the voice data is t seconds; Eliminate abnormal speech from the noise-reduced speech to obtain speech to be extracted, wherein the abnormal speech is speech whose speech noise is greater than a noise threshold; Extract the mel-spectrogram of the speech to be extracted.

3. According to the Transformer-based TCM acoustic diagnosis constitution identification method of claim 2, it is characterized in that: The step of extracting the mel-spectrogram of the speech to be extracted comprises: The speech to be extracted is subjected to pre-emphasis, framing and windowing processing in sequence to obtain windowed speech; wherein the windowing processing is to apply a Hamming window to each frame every 10 ms, and the length of the Hamming window is 25 ms; Performing a fast Fourier transform on the windowed speech to obtain a frequency domain signal; Inputting the frequency domain signal into a Mel filter bank for bandpass filtering to obtain a bandpass filtered signal; The signal after bandpass filtering is subjected to logarithmic operation and DCT transformation in sequence to obtain a Mel spectrum diagram.

4. According to the Transformer-based TCM acoustic diagnosis constitution identification method of claim 3, it is characterized in that: The step of inputting the frequency domain signal into a Mel filter bank for bandpass filtering to obtain a bandpass filtered signal comprises: Inputting the frequency domain signal into N Mel filters for bandpass filtering respectively to obtain N spectral values; The N spectral values ​​are added together to obtain a bandpass filtered signal.

5. According to the Transformer-based TCM acoustic diagnosis constitution identification method of claim 1, it is characterized in that: After obtaining the Mel-spectrogram, the method further includes: Randomly divide the Mel-spectrogram into M groups; Determine the input order of the M groups of mel-spectrograms.

6. According to a Transformer-based TCM acoustic diagnosis constitution identification method according to claim 1, it is characterized in that: The collecting of the subject's voice data includes: Reducing environmental noise until the environmental noise is less than a noise threshold; adjusting microphone parameters, the microphone parameters including a signal-to-noise ratio of the microphone and a distance between the microphone and the subject's lips; The subject's speech was collected using Praat software.

7. A Transformer-based TCM acoustic diagnosis and constitution identification model, characterized in that: The Transformer-based TCM acoustic diagnosis constitution identification method applied to any one of claims 1-6, the model includes an input processing module, a linear projection and embedding module, a Transformer encoder module and an output module; the input processing module is used to preprocess the input Mel spectrum map to change the input form of the Mel spectrum map; the linear projection and embedding module is used to linearly project each block and embed the linear projection result into a high-dimensional vector space; the Transformer encoder module is used to extract the global features and context dependencies of the input data, and the output module is used to map the processed features into specific mapping values.

8. The Transformer-based TCM acoustic diagnosis constitution identification model according to claim 7 is characterized in that: The input processing module includes a segmentation layer and a linear projection layer, the input end of the segmentation layer receives the Mel frequency spectrum map, and the output end of the segmentation layer is connected to the input end of the linear projection layer; the segmentation layer is used to segment the Mel frequency spectrum map into multiple image blocks, and the linear projection layer is used to expand each of the image blocks to obtain a one-dimensional embedded feature sequence.

9. The Transformer-based TCM acoustic diagnosis constitution identification model according to claim 7, characterized in that: The Transformer encoder module includes a multi-head attention layer and a splicing layer, wherein the multi-head attention layer is composed of multiple self-attention layers; the self-attention layer is used to perform different forms of linear transformations on the blocks to generate three matrices, and the splicing layer is used to splice the outputs of all the self-attention layers.

10. The Transformer-based TCM acoustic diagnosis constitution identification model according to claim 7, characterized in that: The output module includes a linear layer and a classification layer, the input end of the linear layer is connected to the output end of the Transformer encoder module, and the output end of the linear layer is connected to the input end of the classification layer; the linear layer is used to map the high-dimensional feature vector into a final feature vector, and the classification layer is used to classify the final feature vector to obtain the physical constitution type.

Images