Traditional Chinese medicine constitution identification method and device based on pulse condition and tongue condition multi-modal features

Abstract

The application discloses a traditional Chinese medicine constitution identification method and device based on pulse and tongue multi-modal features. The method comprises the following steps: obtaining original pulse signals and tongue images of individuals with different traditional Chinese medicine constitutions, obtaining a tongue pulse multi-modal image after pretreatment and feature fusion, dividing the image into a training set and a verification set after adding a category label; establishing a multi-modal traditional Chinese medicine constitution prediction model containing a fusion attention module and training the model through the training set and the verification set; obtaining a tongue pulse multi-modal image of an individual to be identified, and then determining the traditional Chinese medicine constitution of the individual through the prediction model. The application fuses multi-modal data features of pulse signals and tongue images, so that different modal data are complementary, a multi-modal traditional Chinese medicine constitution prediction model with a fusion attention mechanism is adopted, deep extraction of pulse and tongue multi-modal features is realized, corresponding traditional Chinese medicine constitution types can be more comprehensively and accurately identified, the identification result is accurate and reliable, and the device is convenient to deploy on a terminal equipment.

Description

Technical Field

[0001] The present invention relates to a method for identifying TCM constitution, and relates to the fields of artificial intelligence and TCM constitution analysis, and in particular to a method and device for identifying TCM constitution based on multimodal features of pulse and tongue. Background Art

[0002] Currently, research on TCM constitution analysis primarily focuses on manual methods. TCM constitution identification relies primarily on scales, supplemented by expert analysis. However, the former is highly subjective and time-consuming, while the latter is significantly influenced by expert subjective factors and experience. In recent years, artificial intelligence technologies, including deep learning, have been widely researched and applied in various fields, providing new methods and approaches for the development of TCM constitution identification technology.

[0003] Tongue diagnosis is a crucial component of the four diagnostic methods in Traditional Chinese Medicine (TCM). The tongue nourishes the vital essence of the internal organs, and the flow of qi and blood is reflected in the tongue. Therefore, tongue image can serve as a basis for identifying constitutions in TCM. Currently, some researchers have constructed a constitution classification model based on the Inception-v3 model, which classifies nine constitution types with a high recognition rate. Other researchers have used the Alexnet network to perform deep feature fusion on tongue images, classifying three constitutions: qi deficiency, phlegm-dampness, and damp-heat. The accuracy is significantly improved compared to traditional classification methods. Furthermore, some researchers have constructed a complex perception classification model based on convolutional neural networks to identify nine constitutions, improving recognition accuracy.

[0004] Current research on TCM constitution identification based on deep learning primarily focuses on unimodal tongue data. However, the limited nature of unimodal tongue data significantly limits the application and development of artificial intelligence in TCM constitution analysis. In the traditional TCM theory of "looking, listening, asking, and palpating," palpation is as important as observation. Palpation, also known as pulse diagnosis, is a crucial method for TCM to assess a patient's internal organs. For TCM constitutions, the characteristics of different constitution types are reflected not only in the color and condition of the tongue but also in the state and changes of the internal organs. Summary of the Invention

[0005] To address the problems presented in the previous article, the present invention provides a method and device for TCM constitution identification based on multimodal features of pulse and tongue. By acquiring the subject's pulse signal and tongue image and fusing these features, the present invention utilizes a fusion attention mechanism and a multimodal TCM constitution prediction model for training and optimization. This method accurately identifies a variety of TCM constitutions, promoting the development of multimodal approaches in TCM constitution analysis.

[0006] The technical solution adopted in the present invention is:

[0007] 1. A method for identifying TCM constitution based on multimodal features of pulse and tongue:

[0008] 1) Obtain original pulse signals and original tongue images of several individuals with different TCM constitutions, perform signal preprocessing on each original pulse signal to obtain a pulse 2D feature image, perform image preprocessing on each original tongue image to obtain a tongue standardized image, perform feature fusion on each individual's pulse 2D feature image and tongue standardized image to obtain tongue and pulse multimodal images, add category labels to each tongue and pulse multimodal image, and divide it into a training set and a validation set according to a preset ratio.

[0009] 2) Establish a multimodal TCM constitution prediction model that includes a fusion attention module, input the training set and validation set into the multimodal TCM constitution prediction model for training, and obtain a trained multimodal TCM constitution prediction model.

[0010] 3) Obtain the pulse signal and tongue image of the individual to be identified, and perform the same processing as in step 1) to obtain the multimodal image of the tongue and pulse to be identified, which is then input into the trained multimodal TCM constitution prediction model for processing. After processing, the trained multimodal TCM constitution prediction model outputs the category of the multimodal image of the tongue and pulse, thereby determining the category of the TCM constitution of the individual to be identified and realizing TCM constitution identification.

[0011] In the step 1), TCM constitutions include balanced constitution, qi deficiency constitution, yang deficiency constitution, yin deficiency constitution, phlegm-damp constitution, damp-heat constitution, blood stasis constitution, qi stagnation constitution and special constitution.

[0012] In the described step 1), the original pulse signals collected include the Cun, Guan, and Chi three pulse signals of the left hand and the right hand of the individual, namely there are six signals, and signal preprocessing is carried out to each original pulse signal, specifically first adopting Butterworth bandpass filter to carry out signal filtering processing to the Cun, Guan, and Chi three pulse signals of the left hand and the right hand of each individual, removing pulse signal noise and baseline drift, and obtaining clean periodic pulse signals; then adopting Pearson correlation coefficient method to reject irrelevant channel signals, and obtain effective periodic pulse signals; then carrying out cycle segmentation and normalization processing, obtaining the Cun, Guan, and Chi three six-dimensional twenty-four channel single-cycle pulse signals of the left hand and the right hand, and then using sampling Gramian angular field imaging (GAF, Gramian Angular Field), recurrence plot imaging (RP, recurrence plot) and Markov transfer field imaging (MTF, Markov Transition Field) respectively. The six-dimensional twenty-four-channel single-cycle pulse signals of the Cun, Guan, and Chi parts of the left and right hands were converted into two-dimensional spatial domain images by three methods. Finally, the three two-dimensional spatial domain images were fused to obtain the three-channel 2D pulse feature image.

[0013] In step 1), the original tongue image collected is an image of the tongue taken while the individual extends the tongue. Each original tongue image is subjected to image preprocessing, specifically, image segmentation and image enhancement processing to obtain a standardized tongue image. The image enhancement processing specifically utilizes preprocessing techniques such as image sharpening and image smoothing. Finally, the pulse 2D feature image of each individual and the standardized tongue image are feature fused to obtain a seven-channel multimodal tongue and pulse image.

[0014] In the step 2), the multimodal TCM constitution prediction model includes a backbone feature extraction network, a fusion attention module and a multilayer perceptron MLP (Multilayer Perceptron) connected in sequence. The fusion attention module includes a channel attention block CAB (Channel Attention Block), a spatial attention block SAB (Spatial Attention Block) and two convolutional reconstruction modules CRM (Convolutional Reconstruction Module). The input of the multimodal TCM constitution prediction model is first input into the backbone feature extraction network for processing and then output to the fusion attention module. The fusion attention module includes the channel attention block CAB for processing to obtain a first intermediate result. The first intermediate result is multiplied by the output of the backbone feature extraction network and input into the first convolutional reconstruction module CRM for processing to obtain a second intermediate result. The second intermediate result is input into the spatial attention block SAB for processing and then multiplied by the second intermediate result, and then input into the second convolutional reconstruction module CRM for processing. The final output is the output result of the multimodal TCM constitution prediction model.

[0015] The backbone feature extraction network is a Resnet50 network, consisting of 4 residual blocks, 50 convolutional layers, and 2 pooling layers. This network is used to extract key features from multimodal tongue and pulse data. The fusion attention module utilizes a channel-spatial attention architecture and introduces a frequency-domain attention mechanism to extract features from a frequency-domain perspective. Convolutional reconstruction modules are introduced at both the channel and spatial levels to trim the model to reduce model parameters and feature redundancy. The multilayer perceptron (MLP) consists of sequentially connected fully connected layers and a softmax activation function.

[0016] The channel attention block CAB of the fusion attention module first processes the input feature map using the two-dimensional discrete cosine transform DCT (Discrete Cosine Transform) to obtain a third intermediate result. The third intermediate result is processed by the fully connected layer and the Sigmoid activation function in turn, and then multiplied with the third intermediate result and output as the output of the channel attention block CAB.

[0017] The spatial attention block SAB of the fusion attention module first inputs the input feature map into the average pooling layer and the maximum pooling layer in sequence for processing, and then inputs it into the convolution layer with a kernel of 7*7 for processing, and finally processes it through the Sigmoid activation function and outputs it as the output of the spatial attention block SAB.

[0018] The convolutional reconstruction module CRM first divides the input feature map into a high-information feature map of the αC channel and a low-information feature map of the (1-α)C channel, where α is the segmentation ratio, 0≤α≤1, and C is the number of channels; then the high-information feature map and the low-information feature map are respectively input into two convolutional reconstruction units CRU (Convolutional Reconstruction Unit) for processing, and the outputs of the two convolutional reconstruction units CRU are processed in turn by the global average pooling layer and the SoftMax activation function, and the processed results are added and output as the output of the convolutional reconstruction module CRM.

[0019] In step 2), the training set is first input into the multimodal TCM constitution prediction model for training until the loss function of the multimodal TCM constitution prediction model converges. Then, the validation set is input into the multimodal TCM constitution prediction model for training, and the multimodal TCM constitution prediction model with the highest macro F1 score is selected as the trained multimodal TCM constitution prediction model. The TCM constitution identification effect of the trained multimodal TCM constitution prediction model is evaluated based on the test set.

[0020] 2. A TCM constitution identification device based on multimodal features of pulse and tongue:

[0021] The TCM constitution identification device includes a camera, an array pressure sensor, a memory, and a processor. The camera and the array pressure sensor are electrically connected to the memory, which is electrically connected to the processor. The camera is placed on the front of an individual and is used to collect tongue images of the individual. The array pressure sensor is connected to the individual's left and right hands and is used to collect pulse signals of the individual's left and right hands. The memory stores the structure and model parameters of a multimodal TCM constitution prediction model, the individual's pulse signal and tongue image, and a mobile terminal deployment architecture that allows it to run on the processor. The processor is used to read the multimodal TCM constitution prediction model, the individual's pulse signal, and tongue image stored in the memory and perform TCM constitution identification based on the multimodal characteristics of the tongue and pulse. The TCM constitution identification device can also include an optional display screen or speaker according to usage requirements. The optional display screen or speaker is used for TCM constitution identification and data visualization.

[0022] The beneficial effects of the present invention are:

[0023] 1. The present invention aims to address the situation in which existing TCM constitution identification relies on single-modal information of tongue image and simply introduces the concepts of tongue image and pulse image. It collects six-dimensional pulse signals of the three parts of the left and right hands based on an array pressure sensor, and integrates the multimodal data features of the pulse signals and tongue images, so that different modal data complement each other, and the corresponding TCM constitution type is identified more comprehensively and accurately, and the identification results are accurate and reliable.

[0024] 2. The present invention addresses the problems of being unable to fully utilize the multimodal information of pulse and tongue images in existing TCM constitution identification, poor feature extraction capability, and limited recognition accuracy. It proposes a 2D conversion and image fusion technology for pulse data, adopts a multimodal TCM constitution prediction model that integrates an attention mechanism, realizes in-depth extraction of multimodal features of pulse and tongue images, and proposes a convolutional reconstruction module to reduce the computational complexity of the network model while maintaining the recognition accuracy of TCM constitution identification, making it easy to deploy on terminal devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 The overall flow chart of the multimodal TCM constitution identification method provided by the embodiment of the present invention;

[0026] Figure 2 Pulse 2D conversion and multimodal feature fusion flow chart provided by the embodiment of the present invention;

[0027] Figure 3 A structural diagram of a multimodal TCM constitution prediction model provided by an embodiment of the invention;

[0028] Figure 4 A structural diagram of the fusion attention module provided in an embodiment of the present invention;

[0029] Figure 5 A structural diagram of the channel attention block CAB and the spatial attention block SAB provided in an embodiment of the present invention;

[0030] Figure 6 A structural diagram of a convolutional reconstruction module CRM provided in an embodiment of the present invention;

[0031] Figure 7 This is a structural block diagram of a TCM constitution identification device based on multimodal features of pulse and tongue images according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0033] like Figure 1 As shown, the TCM constitution identification method based on the multimodal characteristics of pulse and tongue is as follows:

[0034] 1) Obtaining raw pulse signals and raw tongue images from individuals with different TCM constitutions, performing signal preprocessing on each raw pulse signal to obtain a 2D pulse feature image, performing image preprocessing on each raw tongue image to obtain a standardized tongue image, performing feature fusion on each individual's 2D pulse feature image and standardized tongue image to obtain a multimodal tongue and pulse image image, adding a category label to each multimodal tongue and pulse image image and dividing it into a training set and a validation set according to a preset ratio. TCM constitutions include balanced constitution, qi deficiency constitution, yang deficiency constitution, yin deficiency constitution, phlegm-damp constitution, damp-heat constitution, blood stasis constitution, qi stagnation constitution, and special constitution.

[0035] The collected original pulse signals include the Cun, Guan, and Chi pulse signals of the left and right hands of an individual, that is, there are six signals. Signal preprocessing is performed on each original pulse signal. Specifically, the Cun, Guan, and Chi pulse signals of the left and right hands of each individual are first filtered using a Butterworth bandpass filter to remove pulse signal noise and baseline drift to obtain clean periodic pulse signals; then the Pearson correlation coefficient method is used to eliminate irrelevant channel signals to obtain effective periodic pulse signals; then cycle segmentation and normalization are performed to obtain six-dimensional, twenty-four-channel, single-cycle pulse signals of the Cun, Guan, and Chi of the left and right hands; then, the six-dimensional, twenty-four-channel, single-cycle pulse signals of the Cun, Guan, and Chi of the left and right hands are converted into two-dimensional spatial domain images using three methods, namely, sampled Gramman angle field imaging, recursive graph imaging, and Markov transfer field imaging; finally, the three two-dimensional spatial domain images are image fused to obtain a three-channel pulse 2D feature image.

[0036] The collected raw tongue images are taken while the individual extends their tongue. Each raw tongue image undergoes image preprocessing, specifically by sequentially performing image segmentation and image enhancement to obtain a standardized tongue image. Image enhancement utilizes preprocessing techniques such as image sharpening and smoothing. Finally, a seven-channel multimodal tongue and pulse image is generated by fusion of each individual's pulse 2D feature image and the standardized tongue image.

[0037] 2) Establish a multimodal TCM constitution prediction model including a fusion attention module, input the training set and the validation set into the multimodal TCM constitution prediction model for training, and obtain a trained multimodal TCM constitution prediction model. The multimodal TCM constitution prediction model includes a backbone feature extraction network, a fusion attention module, and a multilayer perceptron MLP connected in sequence. The fusion attention module includes a channel attention block CAB, a spatial attention block SAB, and two convolutional reconstruction modules CRM. The input of the multimodal TCM constitution prediction model is first input into the backbone feature extraction network for processing and then output to the fusion attention module. The fusion attention module includes the channel attention block CAB for processing to obtain a first intermediate result. The first intermediate result is then multiplied by the output of the backbone feature extraction network and input into the first convolutional reconstruction module CRM for processing to obtain a second intermediate result. The second intermediate result is input into the spatial attention block SAB for processing and then multiplied by the second intermediate result, and then input into the second convolutional reconstruction module CRM for processing. The final output is the output result of the multimodal TCM constitution prediction model.

[0038] The backbone feature extraction network is a Resnet50 network, consisting of 4 residual blocks, 50 convolutional layers, and 2 pooling layers. This network is used to extract key features from multimodal tongue and pulse data. The fusion attention module utilizes a channel-spatial attention architecture and introduces a frequency-domain attention mechanism to extract features from a frequency-domain perspective. Convolutional reconstruction modules are introduced at both the channel and spatial levels to trim the model to reduce model parameters and feature redundancy. The multilayer perceptron (MLP) consists of sequentially connected fully connected layers and a softmax activation function.

[0039] The channel attention block CAB of the fusion attention module first processes the input feature map using the two-dimensional discrete cosine transform DCT (Discrete Cosine Transform) to obtain the third intermediate result. The third intermediate result is processed by the fully connected layer and the Sigmoid activation function in turn, and then multiplied with the third intermediate result and output as the output of the channel attention block CAB.

[0040] The spatial attention block SAB of the fusion attention module first inputs the input feature map into the average pooling layer and the maximum pooling layer in sequence for processing, and then inputs it into the convolution layer with a kernel of 7*7 for processing, and finally processes it through the Sigmoid activation function and outputs it as the output of the spatial attention block SAB.

[0041] The convolutional reconstruction module CRM first divides the input feature map into a high-information feature map of the αC channel and a low-information feature map of the (1-α)C channel, where α is the segmentation ratio, 0≤α≤1, and C is the number of channels; then the high-information feature map and the low-information feature map are respectively input into two convolutional reconstruction units CRU for processing. The outputs of the two convolutional reconstruction units CRU are processed in turn by the global average pooling layer and the SoftMax activation function. The processed results are added and output as the output of the convolutional reconstruction module CRM.

[0042] First, the training set was input into the multimodal TCM constitution prediction model for training until the loss function of the multimodal TCM constitution prediction model converged. Then, the validation set was input into the multimodal TCM constitution prediction model for training. The multimodal TCM constitution prediction model with the highest macro F1 score was selected as the trained multimodal TCM constitution prediction model. The TCM constitution identification performance of the trained multimodal TCM constitution prediction model was evaluated based on the test set.

[0043] 3) Obtain the pulse signal and tongue image of the individual to be identified, and perform the same processing as in step 1) to obtain the multimodal image of the tongue and pulse to be identified, which is then input into the trained multimodal TCM constitution prediction model for processing. After processing, the trained multimodal TCM constitution prediction model outputs the category of the multimodal image of the tongue and pulse, thereby determining the category of the TCM constitution of the individual to be identified and realizing TCM constitution identification.

[0044] The specific embodiments of the present invention are as follows:

[0045] The present invention first requires the test subject to sit or rest for at least 5 minutes, and then uses a 4*6 array pressure sensor to collect the inch, guan and chi pulses of the test subject's left and right hands in sequence according to a unified pressure standard to obtain the test subject's original pulse signal; then a camera is used to collect the test subject's original tongue image. The test subject sits and the collection and shooting are performed under natural light during the day.

[0046] Then, the two original multimodal data, the original pulse signal and the original tongue image, were preprocessed, and image fusion and feature fusion processing were performed to obtain pulse and tongue multimodal images. After adding category labels to each tongue and pulse multimodal image, they were randomly assigned to training, validation and test sets in an 8:1:1 ratio. First, a Butterworth bandpass filter was used to filter the original pulse signal to remove low-frequency noise and baseline drift of the pulse signal to obtain a clean 24-channel periodic pulse signal; then the Pearson correlation coefficient method was used to eliminate irrelevant channel signals to obtain a valid periodic pulse signal; the valid periodic pulse signal was periodically segmented and normalized to obtain six-dimensional single-cycle pulse signals of the three parts of the left and right hands; three methods, sampling Gramman angle field imaging, recursive graph imaging and Markov transfer field imaging, were used to convert the single-cycle pulse signal into a two-dimensional spatial domain image, and then image fusion was performed to obtain a 144*144 pulse 2D feature image. The original tongue image is segmented to obtain a single tongue image; then, image sharpening, image smoothing and other preprocessing techniques are used to enhance the image to obtain a 144*144 three-channel standardized tongue image; finally, image feature fusion is performed to obtain the multimodal pulse and tongue image required by the model.

[0047] Then, the multimodal TCM constitution prediction model is used to extract the backbone features of the multimodal feature images of pulse and tongue, which are then input into the fusion attention module for deep feature extraction and feature screening. Finally, the multimodal fusion feature map is obtained after passing through the multi-layer perceptron MLP.

[0048] like Figure 3 As shown in the figure, the backbone feature extraction network of the multimodal TCM constitution prediction model adopts the Resnet50 network, which includes 50 convolutional layers and uses residual connections, which can effectively extract the overall features of multimodal data.

[0049] like Figure 4 As shown in the figure, the fusion attention module is mainly composed of a channel attention reconstruction block CAR, a spatial attention reconstruction block SAR and two convolution reconstruction blocks CRM. The input feature map (Input Feature) is processed by the channel attention reconstruction block CAR, the spatial attention reconstruction block SAR and the two convolution reconstruction modules CRM to output a refined feature map (Refined Feature).

[0050] like Figure 5 As shown in the figure, the upper part is the channel attention reconstruction block CAR, which performs multi-spectral pooling and maximum pooling on the multimodal feature map, introduces the frequency domain attention mechanism to divide the feature map along the channel, and assigns the corresponding frequency weight component to each channel, thereby realizing feature extraction from the perspective of the frequency domain.

[0051] The Fre function that processes the feature map using two-dimensional DCT is as follows:

[0052] Fre(F)=cat([Fre 0 ,Fre 1 ,…,Fre i ,…,Fre n-1 ])

[0053]

[0054] Among them, cat() is a splicing operation; Fre i is the multi-spectral vector of the i-th feature map after DCT processing; For two-dimensional DCT processing, u i and v i F i The two-dimensional index of the corresponding frequency component; F i is the feature vector of the i-th channel after the feature map F is divided along the channel; H and W are the length and width of the feature map respectively; is the input feature map; For the two-dimensional DCT frequency component, for each part, assign a corresponding two-dimensional DCT frequency component

[0055] like Figure 5 As shown in the figure, the lower part is the spatial attention reconstruction block SAR, which performs average pooling and maximum pooling on the multimodal feature map, and then obtains the spatial attention weight coefficient through dimension splicing and convolution processing.

[0056] like Figure 6 As shown in the figure, it is a convolutional reconstruction block CRM. This module adopts a separation and reconstruction strategy. The purpose of the separation operation is to separate the input feature map into an information-rich feature map and a feature map with less information. The reconstruction operation is to splice the information-rich feature map and the feature map with less information together to generate a more information-rich surprise feature map, thereby cropping the model to reduce model parameters and facilitate mobile deployment.

[0057] like Figure 6 As shown, the specific transformation process of the split operation Split is as follows:

[0058]

[0059] Among them, M CRU1 () and M CRU2 () are the first and second convolution reconstruction units CRU respectively; M G 、 and are the first, second and third learnable weight matrices, F up is a high-information feature map, F downis a low-information feature map, and ∪ is a concatenation operation.

[0060] like Figure 6 As shown, the reconstruction operation Reconstruction first uses global average pooling to extract global spatial information variables The SoftMax method is then used to obtain the feature importance vectors β1 and β2, and then the feature maps are merged to obtain the reconstructed feature map (Reconstruct Feature).

[0061]

[0062] Reconstruct Feature=β1F1+β2F2

[0063] Among them, F m is the output of two convolutional reconstruction units CRU; F c (i,j) is the input feature map; and are the spatial information variables of the two input feature maps respectively.

[0064] Finally, the reconstructed feature map is input into the multi-layer perceptron MLP for processing and the final recognition result is output.

[0065] During the training process of this embodiment, the cross-entropy loss function is used to calculate the difference between the predicted value and the true label, and the network parameters are updated through the back propagation algorithm to complete the model training and obtain a trained multimodal TCM constitution prediction model; using the cross-entropy loss function, the maximum epoch is set to 100, the initial learning rate is set to 5e-4, and a dynamic learning rate method is introduced to speed up the training speed to help the model overcome the overfitting problem, and the adaptive momentum optimization method AdamW (Adaptive Moment Estimation with decoupled weight decay) optimizer with decay regularization is used to update the model parameters.

[0066] Finally, the test subject's pulse signal and tongue image are input into the trained multimodal TCM constitution prediction model to identify the TCM constitution.

[0067] like Figure 7As shown, the TCM constitution identification device of the present invention includes a camera, an array pressure sensor, a memory, and a processor. The camera and the array pressure sensor are electrically connected to the memory, which is electrically connected to the processor. The camera is placed in front of the individual and is used to collect the individual's tongue image. The array pressure sensor is connected to the individual's left and right hands and is used to collect the individual's left and right pulse signals of the inch, guan, and chi parts of the left and right hands. The memory stores the structure and model parameters of the multimodal TCM constitution prediction model, the individual's pulse signal and tongue image, and a mobile terminal deployment architecture so that it can be run in the processor. The processor is used to read the multimodal TCM constitution prediction model, the individual's pulse signal, and tongue image stored in the memory and perform TCM constitution identification based on the multimodal characteristics of the tongue and pulse. The TCM constitution identification device can also include an optional display screen or speaker according to usage requirements. The optional display screen or speaker is used for TCM constitution identification and data visualization.

[0068] The above embodiments are used to illustrate the present invention rather than to limit the present invention. Any modifications and changes made to the present invention within the spirit of the present invention and the protection scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A method for identifying TCM constitution based on multimodal features of pulse and tongue, characterized in that: include: 1) Obtaining raw pulse signals and raw tongue images from several individuals with different TCM constitutions, performing signal preprocessing on each raw pulse signal to obtain a 2D pulse feature image, performing image preprocessing on each raw tongue image to obtain a standardized tongue image, performing feature fusion on each individual's 2D pulse feature image and the standardized tongue image to obtain a multimodal tongue and pulse image, adding a category label to each multimodal tongue and pulse image, and dividing it into a training set and a validation set according to a preset ratio; 2) Establish a multimodal TCM constitution prediction model that includes a fusion attention module, input the training set and validation set into the multimodal TCM constitution prediction model for training, and obtain the trained multimodal TCM constitution prediction model; 3) Obtaining the pulse signal and tongue image of the individual to be identified, and performing the same processing as in step 1) to obtain a multimodal tongue and pulse image to be identified, which is then input into a trained multimodal TCM constitution prediction model for processing. After processing, the trained multimodal TCM constitution prediction model outputs the category of the tongue and pulse multimodal image, thereby determining the TCM constitution category of the individual to be identified, thereby achieving TCM constitution identification; In the described step 1), each original pulse signal is carried out in signal preprocessing, first obtain the three six-dimensional single-cycle pulse signals of the Cun, Guan and Chi of the left hand and the right hand, then respectively use Graman angle field imaging, recursion graph imaging and Markov transfer field imaging three methods that the three six-dimensional single-cycle pulse signals of the Cun, Guan and Chi of the left hand and the right hand are converted into three two-dimensional space domain images, and the three two-dimensional space domain images are carried out image fusion to obtain pulse 2D feature images; In the step 2), the multimodal TCM constitution prediction model includes a backbone feature extraction network, a fusion attention module and a multilayer perceptron MLP connected in sequence. The fusion attention module includes a channel attention block CAB, a spatial attention block SAB and two convolutional reconstruction modules CRM. The input of the multimodal TCM constitution prediction model is first input into the backbone feature extraction network for processing and then output into the channel attention block CAB of the fusion attention module for processing to obtain a first intermediate result. The first intermediate result is multiplied by the output of the backbone feature extraction network and input into the first convolutional reconstruction module CRM for processing to obtain a second intermediate result. The second intermediate result is input into the spatial attention block SAB for processing and then multiplied with the second intermediate result, and then input into the second convolutional reconstruction module CRM for processing. The final output is the output result of the multimodal TCM constitution prediction model.

2. The method for identifying TCM constitution based on multimodal features of pulse and tongue according to claim 1, characterized in that: In the step 1), TCM constitutions include balanced constitution, qi deficiency constitution, yang deficiency constitution, yin deficiency constitution, phlegm-damp constitution, damp-heat constitution, blood stasis constitution, qi stagnation constitution and special constitution.

3. The method for identifying TCM constitution based on multimodal features of pulse and tongue according to claim 1, characterized in that: In the described step 1), the original pulse signals collected include the Cun, Guan, and Chi pulse signals of the left hand and right hand of the individual, and signal preprocessing is performed on each original pulse signal, specifically first adopting a Butterworth bandpass filter to carry out signal filtering processing on the Cun, Guan, and Chi pulse signals of the left hand and right hand of each individual, then adopting the Pearson correlation coefficient method to eliminate irrelevant channel signals, then performing cycle segmentation and normalization processing, obtaining the Cun, Guan, and Chi six-dimensional single-cycle pulse signals of the left hand and right hand, then using Graman angle field imaging, recursion graph imaging, and Markov transfer field imaging three methods respectively, the Cun, Guan, and Chi six-dimensional single-cycle pulse signals of the left hand and right hand are converted into three two-dimensional space domain images, and finally the three two-dimensional space domain images are subjected to image fusion to obtain a pulse 2D feature image.

4. The method for identifying TCM constitution based on multimodal features of pulse and tongue according to claim 1, characterized in that: In the step 1), the original tongue image collected is a tongue image taken when the individual sticks out the tongue. Each original tongue image is subjected to image preprocessing, specifically, image segmentation processing and image enhancement processing are performed on the original tongue image in sequence to obtain a standardized tongue image.

5. The method for identifying TCM constitution based on multimodal features of pulse and tongue according to claim 1, characterized in that: The channel attention block CAB of the fusion attention module first processes the input feature map using the two-dimensional discrete cosine transform DCT to obtain a third intermediate result. The third intermediate result is processed by the fully connected layer and the Sigmoid activation function in turn, and then multiplied by the third intermediate result and output as the output of the channel attention block CAB.

6. The method for identifying TCM constitution based on multimodal features of pulse and tongue according to claim 1, characterized in that: The spatial attention block SAB of the fusion attention module first inputs the input feature map into the average pooling layer and the maximum pooling layer for processing, and then inputs it into the convolution layer for processing, and finally processes it through the Sigmoid activation function and outputs it as the output of the spatial attention block SAB.

7. The method for identifying TCM constitution based on multimodal features of pulse and tongue according to claim 1, characterized in that: The convolutional reconstruction module CRM first divides the input feature map into High-information feature maps of channels and Low-information feature map of the channel, where is the split ratio, , is the number of channels; then the high-information feature map and the low-information feature map are input into two convolutional reconstruction units CRU for processing respectively. The outputs of the two convolutional reconstruction units CRU are processed by the global average pooling layer and the SoftMax activation function in turn. The processed results are added and output as the output of the convolutional reconstruction module CRM.

8. The method for identifying TCM constitution based on multimodal features of pulse and tongue according to claim 1, characterized in that: In the step 2), the training set is first input into the multimodal TCM constitution prediction model for training until the loss function of the multimodal TCM constitution prediction model converges, and then the validation set is input into the multimodal TCM constitution prediction model for training, and the multimodal TCM constitution prediction model with the highest macro F1 score is selected as the trained multimodal TCM constitution prediction model.

9. A TCM constitution identification device suitable for use in the method according to any one of claims 1 to 8, characterized in that: The system includes a camera, an array pressure sensor, a memory and a processor. The camera and the array pressure sensor are electrically connected to the memory respectively, and the memory is electrically connected to the processor. The camera is placed on the front of the individual and is used to collect the tongue image of the individual. The array pressure sensor is connected to the left and right hands of the individual and is used to collect the pulse signals of the inch, guan and chi parts of the left and right hands of the individual. The memory stores the structure of the multimodal TCM constitution prediction model and its model parameters, the individual's pulse signal and tongue image. The processor is used to read the multimodal TCM constitution prediction model, the individual's pulse signal and tongue image stored in the memory and perform TCM constitution identification based on the multimodal characteristics of the tongue and pulse.

Images