Traditional Chinese medicine constitution identification method and device based on multi-modal characteristics of pulse condition and tongue condition

Abstract

The invention discloses a traditional Chinese medicine constitution identification method and device based on pulse condition and tongue condition multi-modal characteristics. The method comprises the steps of obtaining original pulse condition signals and tongue condition images of individuals with different traditional Chinese medicine constitutions, performing preprocessing and feature fusion to obtain tongue condition and pulse condition multi-modal images, adding category labels, and dividing the tongue condition and pulse condition multi-modal images into a training set and a verification set; establishing a multi-modal traditional Chinese medicine constitution prediction model containing a fusion attention module, and training through a training set and a verification set; and obtaining a tongue condition and pulse condition multi-modal image of a to-be-identified individual, and then determining the traditional Chinese medicine constitution of the individual through the prediction model. According to the method, the multi-modal data features of the pulse condition signal and the tongue condition image are fused, so that different modal data are supplemented, the multi-modal traditional Chinese medicine constitution prediction model fusing the attention mechanism is adopted, deep extraction of the multi-modal features of the pulse condition and the tongue condition is realized, the corresponding traditional Chinese medicine constitution type can be more comprehensively and accurately identified, and the accuracy of the traditional Chinese medicine constitution prediction is improved. The identification result is accurate and reliable, and the device can be conveniently deployed on terminal equipment.

Description

Technical Field

[0001] The present invention relates to a method for identifying TCM constitution, and relates to the fields of artificial intelligence and TCM constitution analysis, and in particular to a method and device for identifying TCM constitution based on multimodal features of pulse and tongue. Background Art

[0002] At present, the research on TCM constitution analysis is mainly in the artificial field. TCM constitution identification technology is mainly based on scale identification, supplemented by expert identification, but the former is highly subjective and time-consuming, and the latter is greatly affected by expert subjective factors and experience. In recent years, artificial intelligence technology including deep learning has been widely studied and applied in various fields, and has also provided new methods and ideas for the development of TCM constitution identification technology.

[0003] Tongue diagnosis is an important part of the four clinical examinations of traditional Chinese medicine. The essence of the internal organs can be nourished by the tongue, and the rise and fall of qi and blood can be reflected on the tongue. Therefore, tongue image can be the basis for the identification of traditional Chinese medicine constitution. At present, some scholars have built a constitution classification model based on the Inception-v3 model. The model classifies nine types of constitutions and achieves a high recognition rate. Some scholars have also carried out deep feature fusion of tongue images based on the Alexnet network, and classified three types of constitutions: qi deficiency, phlegm dampness, and damp heat. The accuracy rate is greatly improved compared with traditional classification methods. In addition, some scholars have built a complex perception classification model based on convolutional neural networks to identify nine types of constitutions and improve the recognition accuracy.

[0004] At present, the research on TCM constitution identification based on deep learning mainly focuses on tongue image unimodal data. However, the single feature of tongue image unimodal data greatly limits the application and development of artificial intelligence in the field of TCM constitution analysis. In the traditional TCM theory of "observation, auscultation, inquiry and palpation", palpation is as important as observation. Palpation is pulse diagnosis, which is an important means for TCM to judge the patient's internal organs. For TCM constitution, the characteristics of different constitution types are not only reflected in the color and state of the tongue, but also reflected in the state and changes of the internal organs. Summary of the invention

[0005] In order to solve the problems existing in the background technology, the present invention provides a method and device for identifying TCM constitution based on multimodal features of pulse and tongue. The present invention obtains the pulse signal and tongue image of the tester and performs feature fusion, and uses the fusion attention mechanism and multimodal TCM constitution prediction model for training and optimization, which can accurately identify a variety of TCM constitutions and promote the development of multimodality in the field of TCM constitution analysis.

[0006] The technical solution adopted by the present invention is:

[0007] 1. A method for identifying TCM constitution based on multimodal features of pulse and tongue:

[0008] 1) Obtain original pulse signals and original tongue images of several individuals with different TCM constitutions, perform signal preprocessing on each original pulse signal to obtain a pulse 2D feature image, perform image preprocessing on each original tongue image to obtain a tongue standardized image, perform feature fusion on the pulse 2D feature image and the tongue standardized image of each individual to obtain a tongue and pulse multimodal image, add a category label to each tongue and pulse multimodal image and divide it into a training set and a validation set according to a preset ratio.

[0009] 2) Establish a multimodal TCM constitution prediction model that includes a fusion attention module, input the training set and the validation set into the multimodal TCM constitution prediction model for training, and obtain a trained multimodal TCM constitution prediction model.

[0010] 3) Obtain the pulse signal and tongue image of the individual to be identified, and perform the same processing as in step 1) to obtain the tongue and pulse multimodal image to be identified, which is then input into the trained multimodal TCM constitution prediction model for processing. After processing, the trained multimodal TCM constitution prediction model outputs the category of the tongue and pulse multimodal image, thereby determining the category of the TCM constitution of the individual to be identified, thereby realizing TCM constitution identification.

[0011] In the step 1), TCM constitutions include balanced constitution, qi deficiency constitution, yang deficiency constitution, yin deficiency constitution, phlegm-damp constitution, damp-heat constitution, blood stasis constitution, qi stagnation constitution and special constitution.

[0012] In the described step 1), the original pulse signals collected include the pulse signals of the three parts of the Cun, Guan, and Chi of the left hand and the right hand of the individual, that is, there are six signals, and signal preprocessing is performed on each original pulse signal, specifically, the three parts of the Cun, Guan, and Chi of the left hand and the right hand of each individual are firstly subjected to signal filtering processing by using a Butterworth bandpass filter to remove the pulse signal noise and baseline drift to obtain a clean periodic pulse signal; then the irrelevant channel signals are eliminated by using the Pearson correlation coefficient method to obtain an effective periodic pulse signal, and then cycle segmentation and normalization are performed to obtain the six-dimensional twenty-four-channel single-period pulse signals of the three parts of the Cun, Guan, and Chi of the left hand and the right hand, and then sampling Gramian Angular Field (GAF, Gramian Angular Field), recurrence plot imaging (RP, recurrence plot) and Markov Transition Field imaging (MTF, Markov Transition Field) are respectively used. The six-dimensional twenty-four-channel single-cycle pulse signals of the left and right hands were converted into two-dimensional spatial domain images by three methods, and finally the three two-dimensional spatial domain images were fused to obtain the three-channel pulse 2D feature image.

[0013] In the step 1), the original tongue image collected is a tongue image of the tongue taken when the individual sticks out the tongue, and each original tongue image is subjected to image preprocessing, specifically, the original tongue image is subjected to image segmentation processing and image enhancement processing in sequence to obtain a tongue image standardization image. The image enhancement processing is specifically processed using preprocessing techniques such as image sharpening and image smoothing. Finally, the pulse 2D feature image of each individual and the tongue image standardization image are feature fused to obtain a seven-channel tongue image and pulse multimodal image.

[0014] In the step 2), the multimodal TCM constitution prediction model includes a backbone feature extraction network, a fusion attention module and a multilayer perceptron MLP (Multilayer Perceptron) connected in sequence, the fusion attention module includes a channel attention block CAB (Channel Attention Block), a spatial attention block SAB (Spatial Attention Block) and two convolutional reconstruction modules CRM (Convolutional Reconstruction Module), the input of the multimodal TCM constitution prediction model is first input into the backbone feature extraction network for processing and then output to the fusion attention module. The fusion attention module includes the channel attention block CAB for processing to obtain a first intermediate result, the first intermediate result is multiplied by the output of the backbone feature extraction network and then input into the first convolutional reconstruction module CRM for processing to obtain a second intermediate result, the second intermediate result is input into the spatial attention block SAB for processing and then multiplied by the second intermediate result, and then input into the second convolutional reconstruction module CRM for processing, and the final output is the output result of the multimodal TCM constitution prediction model.

[0015] The backbone feature extraction network is specifically a Resnet50 backbone network, including 4 residual blocks, 50 convolutional layers and 2 pooling layers, which are used to extract the main features of tongue and pulse multimodal data. The fusion attention module adopts a channel-space attention architecture and introduces a frequency domain attention mechanism to extract features from the frequency domain. The convolution reconstruction module is introduced at the channel and spatial levels to trim the model to reduce model parameters and feature redundancy. The multilayer perceptron MLP includes fully connected layers and Softmax activation functions connected in sequence.

[0016] The channel attention block CAB of the fusion attention module first processes the input feature map using a two-dimensional discrete cosine transform DCT (Discrete Cosine Transform) to obtain a third intermediate result. The third intermediate result is processed by a fully connected layer and a Sigmoid activation function in turn, and then multiplied by the third intermediate result and output as the output of the channel attention block CAB.

[0017] The spatial attention block SAB of the fusion attention module first inputs the input feature map into the average pooling layer and the maximum pooling layer for processing, and then inputs it into the convolution layer with a kernel of 7*7 for processing, and finally processes it through the Sigmoid activation function and outputs it as the output of the spatial attention block SAB.

[0018] The convolutional reconstruction module CRM first divides the input feature map into a high-information feature map of the αC channel and a low-information feature map of the (1-α)C channel, where α is the segmentation ratio, 0≤α≤1, and C is the number of channels; then the high-information feature map and the low-information feature map are respectively input into two convolutional reconstruction units CRU (Convolutional Reconstruction Unit) for processing, and the outputs of the two convolutional reconstruction units CRU are processed in turn by the global average pooling layer and the SoftMax activation function, and the processed results are added and output as the output of the convolutional reconstruction module CRM.

[0019] In the step 2), the training set is first input into the multimodal TCM constitution prediction model for training until the loss function of the multimodal TCM constitution prediction model converges, and then the validation set is input into the multimodal TCM constitution prediction model for training, and the multimodal TCM constitution prediction model with the highest macro F1 score is selected as the trained multimodal TCM constitution prediction model. The TCM constitution identification effect of the trained multimodal TCM constitution prediction model is evaluated according to the test set.

[0020] 2. A TCM constitution identification device based on multimodal features of pulse and tongue:

[0021] The TCM constitution identification device includes a camera, an array pressure sensor, a memory and a processor, wherein the camera and the array pressure sensor are electrically connected to the memory respectively, and the memory is electrically connected to the processor, the camera is placed on the front of the individual and is used to collect the tongue image of the individual, the array pressure sensor is connected to the left and right hands of the individual and is used to collect the pulse signals of the three parts of the left and right hands of the individual, the memory stores the structure of the multimodal TCM constitution prediction model and its model parameters, the pulse signal and tongue image of the individual, and the mobile terminal deployment architecture so that it can be run in the processor, and the processor is used to read the multimodal TCM constitution prediction model stored in the memory, the pulse signal and tongue image of the individual, and perform TCM constitution identification based on the multimodal characteristics of the tongue and pulse. The TCM constitution identification device can also include an optional display screen or a speaker according to the use requirements, and the optional display screen or speaker is used for TCM constitution identification and data visualization.

[0022] The beneficial effects of the present invention are:

[0023] 1. Aiming at the existing situation that TCM constitution identification relies on single-modal information of tongue and simply introduces the concepts of tongue and pulse, the present invention collects six-dimensional pulse signals of the three parts of the left and right hands based on an array pressure sensor, and integrates the multimodal data features of the pulse signals and tongue images, so that different modal data complement each other, and the corresponding TCM constitution type is identified more comprehensively and accurately, and the identification result is accurate and reliable.

[0024] 2. Aiming at the problems that the existing TCM constitution identification cannot fully utilize the multimodal information of pulse and tongue, has poor feature extraction ability and limited recognition accuracy, the present invention proposes 2D conversion and image fusion technology of pulse data, adopts a multimodal TCM constitution prediction model integrating attention mechanism, realizes the deep extraction of multimodal features of pulse and tongue, and proposes a convolution reconstruction module to reduce the computational amount of the network model while maintaining the recognition accuracy of TCM constitution identification, which is convenient for deployment on terminal devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 An overall flow chart of the multimodal TCM constitution identification method provided by an embodiment of the present invention;

[0026] Figure 2 Pulse 2D conversion and multimodal feature fusion flow chart provided by the embodiment of the present invention;

[0027] Figure 3 A structural diagram of a multimodal TCM constitution prediction model provided for an embodiment of the invention;

[0028] Figure 4 A structural diagram of a fusion attention module provided in an embodiment of the present invention;

[0029] Figure 5 A structural diagram of a channel attention block CAB and a spatial attention block SAB provided in an embodiment of the present invention;

[0030] Figure 6 A structural diagram of a convolutional reconstruction module CRM provided in an embodiment of the present invention;

[0031] Figure 7 It is a structural block diagram of a TCM constitution identification device based on multimodal features of pulse and tongue images according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0033] like Figure 1 As shown, the TCM constitution identification method based on the multimodal characteristics of pulse and tongue is as follows:

[0034] 1) Obtain the original pulse signals and original tongue images of several individuals with different TCM constitutions, perform signal preprocessing on each original pulse signal to obtain a pulse 2D feature image, perform image preprocessing on each original tongue image to obtain a tongue standardized image, perform feature fusion on each individual's pulse 2D feature image and tongue standardized image to obtain a tongue pulse multimodal image, add a category label to each tongue pulse multimodal image and divide it into a training set and a validation set according to a preset ratio. TCM constitutions include peaceful constitution, qi deficiency constitution, yang deficiency constitution, yin deficiency constitution, phlegm dampness constitution, damp-heat constitution, blood stasis constitution, qi stagnation constitution and special constitution.

[0035] The collected original pulse signals include the pulse signals of the Cun, Guan, and Chi parts of the left and right hands of an individual, that is, there are six signals. Signal preprocessing is performed on each original pulse signal. Specifically, the Cun, Guan, and Chi parts of the left and right hands of each individual are firstly subjected to signal filtering processing using a Butterworth bandpass filter to remove the pulse signal noise and baseline drift to obtain a clean periodic pulse signal; then the Pearson correlation coefficient method is used to eliminate irrelevant channel signals to obtain an effective periodic pulse signal, and then period segmentation and normalization are performed to obtain the six-dimensional twenty-four-channel single-period pulse signals of the Cun, Guan, and Chi parts of the left and right hands, and then the six-dimensional twenty-four-channel single-period pulse signals of the Cun, Guan, and Chi parts of the left and right hands are converted into two-dimensional spatial domain images using three methods, namely, sampled Gramman angular field imaging, recursive graph imaging, and Markov transfer field imaging, and finally the three two-dimensional spatial domain images are subjected to image fusion to obtain a three-channel pulse 2D feature image.

[0036] The original tongue image collected is the tongue image of the individual when the tongue is extended. Each original tongue image is subjected to image preprocessing, specifically, the original tongue image is subjected to image segmentation processing and image enhancement processing in sequence to obtain a tongue image standardization image. The image enhancement processing is specifically processed using preprocessing techniques such as image sharpening and image smoothing. Finally, the pulse 2D feature image of each individual and the tongue image standardization image are feature fused to obtain a seven-channel tongue image and pulse multimodal image.

[0037] 2) Establish a multimodal TCM constitution prediction model including a fusion attention module, input the training set and the validation set into the multimodal TCM constitution prediction model for training, and obtain a trained multimodal TCM constitution prediction model. The multimodal TCM constitution prediction model includes a backbone feature extraction network, a fusion attention module and a multilayer perceptron MLP connected in sequence, the fusion attention module includes a channel attention block CAB, a spatial attention block SAB and two convolutional reconstruction modules CRM, the input of the multimodal TCM constitution prediction model is first input into the backbone feature extraction network for processing and then output to the fusion attention module, the fusion attention module includes a channel attention block CAB for processing to obtain a first intermediate result, the first intermediate result is multiplied by the output of the backbone feature extraction network and then input into the first convolutional reconstruction module CRM for processing to obtain a second intermediate result, the second intermediate result is input into the spatial attention block SAB for processing and then multiplied by the second intermediate result, and then input into the second convolutional reconstruction module CRM for processing, and the final output is the output result of the multimodal TCM constitution prediction model.

[0038] The backbone feature extraction network is specifically a Resnet50 backbone network, including 4 residual blocks, 50 convolutional layers and 2 pooling layers, which are used to extract the main features of tongue and pulse multimodal data. The fusion attention module adopts a channel-space attention architecture and introduces a frequency domain attention mechanism to extract features from the frequency domain. The convolution reconstruction module is introduced at the channel and spatial levels to trim the model to reduce model parameters and feature redundancy. The multilayer perceptron MLP includes fully connected layers and Softmax activation functions connected in sequence.

[0039] The channel attention block CAB of the fusion attention module first processes the input feature map using the two-dimensional discrete cosine transform DCT (Discrete Cosine Transform) to obtain the third intermediate result. The third intermediate result is processed by the fully connected layer and the Sigmoid activation function in turn, and then multiplied with the third intermediate result and output as the output of the channel attention block CAB.

[0040] The spatial attention block SAB of the fused attention module first inputs the input feature map into the average pooling layer and the maximum pooling layer for processing, and then inputs it into the convolution layer with a kernel of 7*7 for processing, and finally processes it through the Sigmoid activation function and outputs it as the output of the spatial attention block SAB.

[0041] The convolutional reconstruction module CRM first divides the input feature map into a high-information feature map of the αC channel and a low-information feature map of the (1-α)C channel, where α is the segmentation ratio, 0≤α≤1, and C is the number of channels; then the high-information feature map and the low-information feature map are respectively input into two convolutional reconstruction units CRU for processing. The outputs of the two convolutional reconstruction units CRU are processed by the global average pooling layer and the SoftMax activation function in turn, and the processed results are added and output as the output of the convolutional reconstruction module CRM.

[0042] First, the training set is input into the multimodal TCM constitution prediction model for training until the loss function of the multimodal TCM constitution prediction model converges. Then the validation set is input into the multimodal TCM constitution prediction model for training, and the multimodal TCM constitution prediction model with the highest macro F1 score is selected as the trained multimodal TCM constitution prediction model. The TCM constitution identification effect of the trained multimodal TCM constitution prediction model is evaluated based on the test set.

[0043] 3) Obtain the pulse signal and tongue image of the individual to be identified, and perform the same processing as in step 1) to obtain the tongue and pulse multimodal image to be identified, which is then input into the trained multimodal TCM constitution prediction model for processing. After processing, the trained multimodal TCM constitution prediction model outputs the category of the tongue and pulse multimodal image, thereby determining the category of the TCM constitution of the individual to be identified, thereby realizing TCM constitution identification.

[0044] The specific embodiments of the present invention are as follows:

[0045] The present invention first requires the tester to sit or rest for at least 5 minutes, and uses a 4*6 array pressure sensor to collect the inch, guan, and chi pulses of the tester's left and right hands in sequence according to a unified pressure standard to obtain the tester's original pulse signal; then a camera is used to collect the tester's original tongue image, and the tester sits down to collect and shoot under natural light during the day.

[0046] Then, the two original multimodal data, the original pulse signal and the original tongue image, are preprocessed, and image fusion and feature fusion processing are performed to obtain pulse and tongue multimodal images. After adding category labels to each tongue and pulse multimodal image, they are randomly assigned to training set, verification set and test set in a ratio of 8:1:1. First, a Butterworth bandpass filter is used to filter the original pulse signal, remove the low-frequency noise and baseline drift of the pulse signal, and obtain a clean 24-channel periodic pulse signal; then the Pearson correlation coefficient method is used to eliminate irrelevant channel signals to obtain an effective periodic pulse signal; the effective periodic pulse signal is periodically segmented and normalized to obtain a six-dimensional single-cycle pulse signal of the three parts of the left and right hands; three methods of sampling Gramman angle field imaging, recursive graph imaging and Markov transfer field imaging are used to convert the single-cycle pulse signal into a two-dimensional spatial domain image, and then image fusion is performed to obtain a 144*144 pulse 2D feature image. The original tongue image is segmented to obtain a single tongue image; then image enhancement is performed using preprocessing techniques such as image sharpening and image smoothing to obtain a 144*144 three-channel tongue standardized image; finally, image feature fusion is performed to obtain the pulse and tongue multimodal image required by the model.

[0047] Then, the multimodal TCM constitution prediction model is used to extract the backbone features of the multimodal feature images of pulse and tongue, which are then input into the fusion attention module for deep feature extraction and feature screening. Finally, the multimodal fusion feature map is obtained after passing through the multi-layer perceptron MLP.

[0048] like Figure 3 As shown in the figure, the backbone feature extraction network of the multimodal TCM constitution prediction model adopts the Resnet50 network, which includes 50 convolutional layers and uses residual connections, which can effectively extract the overall characteristics of multimodal data.

[0049] like Figure 4 As shown in the figure, the fusion attention module is mainly composed of a channel attention reconstruction block CAR, a spatial attention reconstruction block SAR and two convolution reconstruction blocks CRM. The input feature map (Input Feature) is processed by the channel attention reconstruction block CAR, the spatial attention reconstruction block SAR and two convolution reconstruction modules CRM to output a refined feature map (Refined Feature).

[0050] like Figure 5 As shown in the figure, the upper part is the channel attention reconstruction block CAR, which performs multi-spectral pooling and maximum pooling on the multimodal feature map, introduces the frequency domain attention mechanism to divide the feature map along the channel, and assigns a corresponding frequency weight component to each channel, thereby realizing feature extraction from the perspective of the frequency domain.

[0051] The Fre function that processes the feature map using two-dimensional DCT is as follows:

[0052] Fre(F)=cat([Fre 0 ,Fre 1 ,…,Fre i ,…,Fre n-1 ])

[0053]

[0054] Among them, cat() is a concatenation operation; Fre i is the multi-spectral vector of the i-th feature map after DCT processing; For two-dimensional DCT processing, u i and v i F i The two-dimensional index of the corresponding frequency component; F i is the feature vector of the ith channel after the feature map F is divided along the channel; H and W are the length and width of the feature map respectively; is the input feature map; For the two-dimensional DCT frequency component, for each part, a corresponding two-dimensional DCT frequency component is assigned

[0055] like Figure 5 As shown in the figure, the lower part is the spatial attention reconstruction block SAR, which performs average pooling and maximum pooling on the multimodal feature map, and then obtains the spatial attention weight coefficient through dimensional splicing and convolution processing.

[0056] like Figure 6 As shown in the figure, it is a convolutional reconstruction block CRM. This module adopts a separation and reconstruction strategy. The purpose of the separation operation is to separate the input feature map into an information-rich feature map and a feature map with less information. The reconstruction operation is to splice the information-rich feature map with the feature map with less information together to generate a surprise feature map with richer information, thereby cropping the model to reduce model parameters and facilitate mobile terminal deployment.

[0057] like Figure 6 As shown, the specific transformation process of the separation operation Split is as follows:

[0058]

[0059] Among them, M CRU1 () and M CRU2 () are the first and second convolutional reconstruction units CRU respectively; M G , and are the first, second and third learnable weight matrices, respectively, and F up is a high-information feature map, F downis a low-information feature map, and ∪ is a concatenation operation.

[0060] like Figure 6 As shown, the reconstruction operation Reconstruction first uses global average pooling to extract global spatial information variables Then the SoftMax method is used to obtain the feature importance vectors β1 and β2, and then the feature maps are merged to obtain the reconstructed feature map (Reconstruct Feature).

[0061]

[0062] Reconstruct Feature=β1F1+β2F2

[0063] Among them, F m is the output of two convolutional reconstruction units CRU; F c (i,j) is the input feature map; and They are the spatial information variables of the two input feature maps respectively.

[0064] Finally, the reconstructed feature map is input into the multi-layer perceptron MLP for processing and the final recognition result is output.

[0065] During the training process of this embodiment, the cross entropy loss function is used to calculate the difference between the predicted value and the true label, and the network parameters are updated through the back propagation algorithm to complete the model training and obtain a trained multimodal TCM constitution prediction model; using the cross entropy loss function, the maximum epoch is set to 100, the initial learning rate is set to 5e-4, and the dynamic learning rate method is introduced to speed up the training speed to help the model overcome the overfitting problem, and the adaptive momentum optimization method AdamW (Adaptive Moment Estimation with decoupled weight decay) optimizer with decay regularization is used to update the model parameters.

[0066] Finally, the test subject's pulse signal and tongue image are input into the trained multimodal TCM constitution prediction model for TCM constitution identification.

[0067] like Figure 7As shown, the TCM constitution identification device of the present invention includes a camera, an array pressure sensor, a memory and a processor, the camera and the array pressure sensor are electrically connected to the memory respectively, the memory is electrically connected to the processor, the camera is placed on the front of the individual and is used to collect the tongue image of the individual, the array pressure sensor is connected to the left hand and the right hand of the individual and is used to collect the pulse signals of the three parts of the left hand and the right hand of the individual, the memory stores the structure of the multimodal TCM constitution prediction model and its model parameters, the pulse signal and the tongue image of the individual, and the mobile terminal deployment architecture so that it can be run in the processor, and the processor is used to read the multimodal TCM constitution prediction model stored in the memory, the pulse signal and the tongue image of the individual and perform TCM constitution identification based on the multimodal characteristics of the tongue and pulse. The TCM constitution identification device can also include an optional display screen or a speaker according to the use requirements, and the optional display screen or speaker is used for TCM constitution identification and data visualization.

[0068] The above embodiments are used to explain the present invention rather than to limit the present invention. Any modification and change made to the present invention within the spirit of the present invention and the protection scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A method for identifying TCM constitution based on multimodal features of pulse and tongue, characterized in that: include: 1) obtaining original pulse signals and original tongue images of several individuals with different TCM constitutions, performing signal preprocessing on each original pulse signal to obtain a pulse 2D feature image, performing image preprocessing on each original tongue image to obtain a tongue standardized image, performing feature fusion on the pulse 2D feature image of each individual and the tongue standardized image to obtain a tongue pulse multimodal image, adding a category label to each tongue pulse multimodal image and dividing it into a training set and a validation set according to a preset ratio; 2) Establishing a multimodal TCM constitution prediction model including a fusion attention module, inputting the training set and the validation set into the multimodal TCM constitution prediction model for training, and obtaining a trained multimodal TCM constitution prediction model; 3) Obtain the pulse signal and tongue image of the individual to be identified, and perform the same processing as in step 1) to obtain the tongue and pulse multimodal image to be identified, which is then input into the trained multimodal TCM constitution prediction model for processing. After processing, the trained multimodal TCM constitution prediction model outputs the category of the tongue and pulse multimodal image, thereby determining the category of the TCM constitution of the individual to be identified, thereby realizing TCM constitution identification.

2. The method for identifying TCM constitution based on multimodal features of pulse and tongue according to claim 1, characterized in that: In the step 1), TCM constitutions include balanced constitution, qi deficiency constitution, yang deficiency constitution, yin deficiency constitution, phlegm-damp constitution, damp-heat constitution, blood stasis constitution, qi stagnation constitution and special constitution.

3. The method for identifying TCM constitution based on multimodal features of pulse and tongue according to claim 1, characterized in that: In the described step 1), the collected original pulse signals include the Cun, Guan, and Chi pulse signals of the left hand and right hand of the individual, and signal preprocessing is performed on each original pulse signal, specifically, the Cun, Guan, and Chi pulse signals of the left hand and right hand of each individual are first subjected to signal filtering processing using a Butterworth bandpass filter, and then irrelevant channel signals are eliminated using a Pearson correlation coefficient method, and then cycle segmentation and normalization processing are performed to obtain the Cun, Guan, and Chi six-dimensional single-cycle pulse signals of the left hand and right hand, and then the six-dimensional twenty-four-channel single-cycle pulse signals of the left hand and right hand are converted into two-dimensional space domain images using three methods of sampled Gramman angle field imaging, recursive graph imaging, and Markov transfer field imaging, and finally the three two-dimensional space domain images are subjected to image fusion to obtain a pulse 2D feature image.

4. The method for identifying TCM constitution based on multimodal features of pulse and tongue according to claim 1, characterized in that: In the step 1), the original tongue image collected is a tongue image taken when the individual sticks out the tongue, and each original tongue image is subjected to image preprocessing, specifically, the original tongue image is subjected to image segmentation processing and image enhancement processing in sequence to obtain a standardized tongue image.

5. The method for identifying TCM constitution based on multimodal features of pulse and tongue according to claim 1, characterized in that: In the step 2), the multimodal TCM constitution prediction model includes a backbone feature extraction network, a fusion attention module and a multilayer perceptron MLP connected in sequence, the fusion attention module includes a channel attention block CAB, a spatial attention block SAB and two convolutional reconstruction modules CRM, the input of the multimodal TCM constitution prediction model is first input into the backbone feature extraction network for processing and then output to the fusion attention module. The fusion attention module includes the channel attention block CAB for processing to obtain a first intermediate result, the first intermediate result is multiplied by the output of the backbone feature extraction network and then input into the first convolutional reconstruction module CRM for processing to obtain a second intermediate result, the second intermediate result is input into the spatial attention block SAB for processing and then multiplied by the second intermediate result, and then input into the second convolutional reconstruction module CRM for processing, and the final output is the output result of the multimodal TCM constitution prediction model.

6. The method for identifying TCM constitution based on multimodal features of pulse and tongue according to claim 5, characterized in that: The channel attention block CAB of the fusion attention module first processes the input feature map using a two-dimensional discrete cosine transform DCT to obtain a third intermediate result. The third intermediate result is processed by a fully connected layer and a Sigmoid activation function in turn, and then multiplied by the third intermediate result and output as the output of the channel attention block CAB.

7. The method for identifying TCM constitution based on multimodal features of pulse and tongue according to claim 5, characterized in that: The spatial attention block SAB of the fusion attention module first inputs the input feature map into the average pooling layer and the maximum pooling layer for processing, and then inputs the input feature map into the convolution layer for processing, and finally processes the feature map through the Sigmoid activation function and outputs the feature map as the output of the spatial attention block SAB.

8. The method for identifying TCM constitution based on multimodal features of pulse and tongue according to claim 5, characterized in that: The convolutional reconstruction module CRM first divides the input feature map into a high-information feature map of the αC channel and a low-information feature map of the (1-α)C channel, where α is the segmentation ratio, 0≤α≤1, and C is the number of channels; then the high-information feature map and the low-information feature map are respectively input into two convolutional reconstruction units CRU for processing, and the processed outputs of the two convolutional reconstruction units CRU are processed by the global average pooling layer and the SoftMax activation function in turn, and the processed results are added and output as the output of the convolutional reconstruction module CRM.

9. The method for identifying TCM constitution based on multimodal features of pulse and tongue according to claim 1, characterized in that: In the step 2), the training set is first input into the multimodal TCM constitution prediction model for training until the loss function of the multimodal TCM constitution prediction model converges, and then the validation set is input into the multimodal TCM constitution prediction model for training, and the multimodal TCM constitution prediction model with the highest macro F1 score is selected as the trained multimodal TCM constitution prediction model.

10. A TCM constitution identification device applicable to the method according to any one of claims 1 to 9, characterized in that: The invention comprises a camera, an array pressure sensor, a memory and a processor. The camera and the array pressure sensor are electrically connected to the memory respectively, and the memory is electrically connected to the processor. The camera is placed on the front of an individual and is used to collect the tongue image of the individual. The array pressure sensor is connected to the left hand and the right hand of the individual and is used to collect the pulse signals of the inch, guan and chi parts of the left hand and the right hand of the individual. The memory stores the structure of a multimodal TCM constitution prediction model and its model parameters, the pulse signal and the tongue image of the individual. The processor is used to read the multimodal TCM constitution prediction model, the pulse signal and the tongue image of the individual stored in the memory and perform TCM constitution identification based on the multimodal characteristics of the tongue and pulse.

Images