Multi-spectral TCM tongue image information analysis device and method

Abstract

The present invention relates to the field of information analysis, and discloses a multi-spectral Chinese medicine tongue image information analysis device and method, which improves the accuracy of tongue image feature extraction and classification decision-making through methods such as multimodal fusion and deep learning. The multi-spectral Chinese medicine tongue image information analysis method comprises: obtaining the reaction of the tongue surface in different spectral bands through dynamic spectral acquisition technology, obtaining multi-spectral tongue image processing data through preprocessing, extracting features from the processed data using a multi-parameter adaptive neural fuzzy inference system model, obtaining recognition results of features such as tongue color, tongue shape, and tongue coating, converting the multi-spectral tongue image processing data into a graph structure, constructing a graph neural network model to capture the relationship between pixel points, and extracting relationship features. Through multimodal fusion technology, the feature vectors of different modes are fused into a comprehensive tongue image feature vector, and the fused feature vector is classified and decided using a preset classifier to obtain the Chinese medicine tongue image information analysis result.

Description

Technical Field

[0001] The present invention relates to the field of information analysis, and in particular to a multi-spectral Chinese medicine tongue image information analysis device and method. Background Art

[0002] Tongue diagnosis, as an important part of TCM visual diagnosis, has a long history and a profound theoretical basis. Traditional TCM tongue diagnosis mainly relies on the doctor's intuitive observation and empirical judgment, and evaluates the patient's health status and disease tendency by analyzing the tongue color, tongue coating, tongue shape and other characteristics. However, this method has the problems of strong subjectivity and experience dependence. Different doctors may have different interpretations of the same tongue image, which affects the accuracy and reliability of tongue diagnosis.

[0003] With the progress of science and technology, especially the development of spectral technology and image processing technology, new possibilities have been provided for the modernization and objectivity of tongue diagnosis in traditional Chinese medicine. As a non-destructive, rapid and non-invasive detection technology, spectroscopy can obtain the response of the tongue in different spectral bands, thereby revealing tongue image information more comprehensively. At the same time, image processing technology can perform high-precision and quantitative analysis of tongue images, improving the accuracy and repeatability of tongue diagnosis.

[0004] Although spectral technology and image processing technology have broad application prospects in TCM tongue diagnosis, the existing technology still has some shortcomings:

[0005] Data acquisition and processing: Traditional tongue diagnosis data collection methods are limited by environmental and equipment conditions and may not accurately capture subtle changes in tongue images. In addition, there may be information loss or distortion during data processing.

[0006] Feature extraction and recognition: Existing tongue feature extraction methods are mostly based on traditional image processing technology, which has limited ability to recognize complex and changeable tongue features and is difficult to fully explore the deep information in the tongue image.

[0007] Accuracy of classification decisions: Due to the complexity and diversity of tongue features, existing classification algorithms may not be able to accurately distinguish the subtle differences between different tongue images, resulting in low accuracy of classification decisions.

[0008] User interaction and feedback: Existing systems often lack effective user interaction mechanisms, making it impossible for doctors to adjust tongue diagnosis parameters according to actual needs or provide feedback to optimize system performance.

[0009] In order to solve the above problems, this study proposed a multi-spectral Chinese medicine tongue image information analysis device and method. Summary of the invention

[0010] The present invention provides a multi-spectral TCM tongue image information analysis device and method, which improves the accuracy of tongue image feature extraction and classification decision-making through methods such as multimodal fusion and deep learning.

[0011] The first aspect of the present invention provides a multi-spectral Chinese medicine tongue image information analysis method, the multi-spectral Chinese medicine tongue image information analysis method comprising: acquiring multi-spectral tongue image data, and pre-processing the multi-spectral tongue image data to obtain multi-spectral tongue image processing data; constructing a multi-parameter adaptive neural fuzzy inference system model based on the multi-spectral tongue image processing data, training and optimizing the multi-parameter adaptive neural fuzzy inference system model for the recognition ability of tongue image features by adjusting the fuzzy rule library, extracting features from the multi-spectral tongue image processing data based on the trained and optimized multi-parameter adaptive neural fuzzy inference system model to obtain a feature vector of the multi-spectral tongue image processing data; converting the multi-spectral tongue image processing data into a graph; A structure is provided in which each pixel is regarded as a node, and the connection relationship between the nodes is determined by the spatial distance and color similarity between the pixels. A graph neural network model is constructed based on the converted graph structure, and the graph neural network model is trained so that the model can learn the relationship between the nodes and the feature representation of the nodes. The trained graph neural network model is used to capture the relationship between the pixels to obtain the relational features of the multispectral tongue image processing data; multimodal fusion is performed based on the multispectral tongue image processing data feature vector and the multispectral tongue image processing data feature vector to obtain a comprehensive tongue image feature vector; a preset classifier is used to perform classification decision on the fused comprehensive tongue image feature vector to obtain the multispectral Chinese medicine tongue image information analysis result.

[0012] Optionally, in a first implementation of the first aspect of the present invention, the multispectral tongue image data is acquired and preprocessed to obtain multispectral tongue image processing data, including: based on the dynamic spectral acquisition data, by dynamically adjusting the settings of the filter or the spectrometer, the reaction of the tongue surface in different spectral bands is captured at different time points; the tongue images in different spectral bands are feature-fused to create a comprehensive feature representation to obtain multispectral tongue image data; and / or nonlinear feature transformation technology is applied, including kernel method or manifold learning, to reveal the intrinsic structure and relationship of the tongue surface data to obtain multispectral tongue image data. The invention relates to a method for preparing a multispectral tongue image data; performing image registration based on the multispectral tongue image data to ensure pixel correspondence between different spectral images, automatically aligning images of different spectra using an image registration algorithm, and fusing them to form a multispectral tongue image data cube, and performing noise removal and enhancement based on the multispectral tongue image data cube; extracting spectral features related to the physiological state of the tongue from the multispectral tongue image data cube, including reflectivity, absorbance, and transmittance, performing dimension reduction processing on the spectral features using principal component analysis or independent component analysis, and then performing standardization and normalization to obtain pre-processed multispectral tongue image processing data.

[0013] Optionally, in a second implementation of the first aspect of the present invention, a multi-parameter adaptive neural fuzzy inference system model is constructed based on the multi-spectral tongue image processing data, the recognition ability of the multi-parameter adaptive neural fuzzy inference system model for tongue image features is optimized by adjusting the fuzzy rule base, and feature extraction is performed on the multi-spectral tongue image processing data based on the multi-parameter adaptive neural fuzzy inference system model after training and optimization to obtain a feature vector of the multi-spectral tongue image processing data, including: designing an initial fuzzy rule base, the fuzzy rule base is defined based on tongue color, tongue coating, and tongue shape features, and for each fuzzy rule in the fuzzy rule base, we define an output function:

[0014] O_i=w_i\cdotf_i(S,T,F);

[0015] Among them, (O_i) is the output of the (i)th rule, (w_i) is the weight of the rule, (f_i(S,T,F)) is the fuzzy inference function based on the rule, (S,T,F) are the input values ​​of tongue color, tongue coating, and tongue shape respectively; (f_i(S,T,F)) is the inference function based on the (i)th fuzzy rule, which is determined according to the input value and the rules in the fuzzy rule base;

[0016] The final output is the weighted sum of all rule outputs:

[0017] O=\sum_{i}O_i;

[0018] A multi-parameter parallel processing mechanism is designed, and modern parallel computing technology is used to allow multiple parameters to perform fuzzy reasoning in the fuzzy rule library at the same time; a neural network and a fuzzy rule library are combined to construct an adaptive multi-parameter adaptive neural fuzzy reasoning system model; the multi-parameter adaptive neural fuzzy reasoning system model is trained according to the multi-spectral tongue image processing data to obtain a trained multi-parameter adaptive neural fuzzy reasoning system model; the multi-parameter adaptive neural fuzzy reasoning system model after training and optimization is used to extract features from the multi-spectral tongue image processing data to obtain recognition results including tongue color, tongue shape, and tongue coating features, and the extracted recognition results are encoded to generate feature vectors of the multi-spectral tongue image processing data.

[0019] Optionally, in a third implementation method of the first aspect of the present invention, the multispectral tongue image processing data is converted into a graph structure, wherein each pixel point is regarded as a node, and the connection relationship between the nodes is determined by the spatial distance and color similarity between the pixels. Based on the converted graph structure, a graph neural network model is constructed, and the graph neural network model is trained so that it can learn the relationship between the nodes and the feature representation of the nodes. The trained graph neural network model is used to capture the relationship between the pixels to obtain the relationship characteristics of the multispectral tongue image processing data, including: converting the multispectral tongue image processing data into a graph structure, wherein each pixel point is regarded as a node, and a threshold value Q is preset. When the spatial distance between two pixels is less than the threshold value Q, it is considered that there is a connection between the two nodes; constructing a graph neural network model to define node characteristics, including the color value, brightness, and other characteristics of the pixels. degree, contrast, and construct a graph neural network model according to the converted graph structure, define convolution operations in the model to aggregate node neighbor information; and / or use superpixel segmentation technology to divide the tongue image into multiple superpixel areas, each superpixel as a node, and edges are established according to the similarity and adjacency relationship between superpixels; train the graph neural network model based on the historical labeled data in the multispectral tongue image processing data, and during the training process, dynamically adjust the graph structure according to the feedback of the model, and the dynamic adjustment includes deleting or adding edges to train the graph neural network model; input the real-time data in the multispectral tongue image processing data into the trained graph neural network model, and obtain the feature representation of each node through the forward propagation of the graph neural network model, and these feature representations capture the relationship between pixel points to obtain the relationship characteristics of the multispectral tongue image processing data.

[0020] Optionally, in a fourth implementation of the first aspect of the present invention, the multimodal fusion is performed based on the multispectral tongue image processing data feature vector and the multispectral tongue image processing data feature vector to obtain a comprehensive tongue image feature vector, including: assigning different weights to the feature vectors of each modality based on the importance of the multispectral tongue image processing data feature vector and the multispectral tongue image processing data feature vector, and then performing weighted summation; and / or connecting the feature vectors of the multispectral tongue image processing data feature vector and the multispectral tongue image processing data feature vector in series to form a longer feature vector; and / or using a tensor to represent the multimodal data, learning the interaction relationship between the modalities through tensor decomposition or a tensor network, setting the multimodal data to a tensor (\mathcal{T}), wherein the feature vector of each modality constitutes a mode of the tensor, and a third-order tensor (\mathcal{T}\in\mathbb{R}^{I\timesJ\timesK}) is decomposed into:

[0021] \mathcal{T}=\mathcal{G}\times_1\mathbf{A}\times_2\mathbf{B}\times_3\mathbf{C};

[0022] Among them, (\mathcal{G}) is the core tensor, which captures the interaction between modalities; (\mathbf{A},\mathbf{B},\mathbf{C}) is a factor matrix, which is related to the eigenvector of each modality; and / or a deep learning model is constructed, which has multiple input branches, each branch corresponds to the eigenvector of a modality, and in the deep layer of the fusion network of the deep learning model, the features of different modalities are fused together through the fusion layer, and the fusion network is jointly trained to optimize the parameters of the fusion layer and the parameters of the feature extractors of each modality, using an end-to-end training method or pre-training the feature extractors of each modality separately, and then jointly fine-tuning; to obtain a comprehensive tongue image feature vector.

[0023] Optionally, in a fifth implementation of the first aspect of the present invention, the use of a preset classifier to make classification decisions on the fused comprehensive tongue image feature vector to obtain a multi-spectral Chinese medicine tongue image information analysis result includes: selecting one or more classifiers, the classifiers including support vector machines, random forests, and neural networks, and optimizing the classifiers according to the characteristics of the multi-spectral tongue image data, including adjusting the kernel function or parameters of the SVM and optimizing the architecture of the neural network; using an ensemble learning method to improve the accuracy of classification by combining the prediction results of multiple classifiers, designing an ensemble strategy for dynamic weight adjustment, and optimizing the classification results according to the performance of each classifier in the validation set. Dynamically adjust its weight in the final decision based on its performance; input the selected and reduced feature vector into a preset classifier for classification decision; and / or design a multi-level classification strategy, first roughly classify the tongue image, and then finely classify each category to obtain the multi-spectral TCM tongue image information analysis results; post-process the multi-spectral TCM tongue image information analysis results, filter out low-confidence multi-spectral TCM tongue image information analysis results through a confidence threshold, or perform multiple rounds of classification to improve accuracy; and / or design a result interpretation module to provide TCM theoretical explanations and suggestions for each multi-spectral TCM tongue image information analysis result.

[0024] Optionally, in a sixth implementation of the first aspect of the present invention, an interactive interface is developed to allow users to intuitively view and interpret the results of the multi-spectral Chinese medicine tongue image information analysis; a feedback mechanism is provided to allow users to correct or supplement the results of the multi-spectral Chinese medicine tongue image information analysis, and the classifier and fusion strategy are continuously optimized through user feedback to form a closed-loop learning system; and / or a dynamic feedback mechanism is introduced to enable the system to be continuously optimized based on actual classification results and expert feedback. When experts raise objections to the results of the multi-spectral Chinese medicine tongue image information analysis, the system automatically records these feedbacks and adjusts the model parameters or adds new training samples accordingly to improve future classification performance; and / or for the same patient, a tongue image history record database is established to track the changes in the tongue image over time, and by analyzing the long-term change trend of the tongue image, provide doctors with deeper insights into health conditions; and / or an interactive tongue image analysis tool is developed to allow doctors to manually adjust the weights and thresholds of tongue image features to more intuitively explore the impact of different feature combinations on the results of the multi-spectral Chinese medicine tongue image information analysis.

[0025] The second aspect of the present invention provides a multi-spectral Chinese medicine tongue image information analysis device, the multi-spectral Chinese medicine tongue image information analysis device comprising: an acquisition module, used to acquire multi-spectral tongue image data, and pre-process the multi-spectral tongue image data to obtain multi-spectral tongue image processing data; a first processing module, used to construct a multi-parameter adaptive neural fuzzy inference system model based on the multi-spectral tongue image processing data, and train and optimize the multi-parameter adaptive neural fuzzy inference system model to recognize tongue image features by adjusting the fuzzy rule base, and extract features from the multi-spectral tongue image processing data based on the multi-parameter adaptive neural fuzzy inference system model after training and optimization to obtain a feature vector of the multi-spectral tongue image processing data; a second processing module, used to construct a multi-parameter adaptive neural fuzzy inference system model based on the multi-spectral tongue image processing data The data is converted into a graph structure, in which each pixel point is regarded as a node, and the connection relationship between the nodes is determined by the spatial distance and color similarity between the pixels. Based on the converted graph structure, a graph neural network model is constructed, and the graph neural network model is trained to enable it to learn the relationship between the nodes and the feature representation of the nodes. The trained graph neural network model is used to capture the relationship between the pixels to obtain the multispectral tongue image processing data relationship characteristics; a setting module is used to perform multimodal fusion based on the multispectral tongue image processing data feature vector and the multispectral tongue image processing data feature vector to obtain a comprehensive tongue image feature vector; an allocation module is used to use a preset classifier to make classification decisions on the fused comprehensive tongue image feature vector to obtain the multispectral Chinese medicine tongue image information analysis result.

[0026] The third aspect of the present invention provides a multi-spectral Chinese medicine tongue and facial image information analysis device, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor calls the instructions in the memory so that the multi-spectral Chinese medicine tongue and facial image information analysis device executes the above-mentioned multi-spectral Chinese medicine tongue and facial image information analysis method.

[0027] A fourth aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the above-mentioned multi-spectral TCM tongue image information analysis method.

[0028] In the technical solution provided by the present invention, the reaction of the tongue surface in different spectral bands is obtained by dynamic spectral acquisition technology, and multi-spectral tongue image processing data is obtained after preprocessing. Then, the processed data is subjected to feature extraction using a multi-parameter adaptive neural fuzzy inference system model to obtain recognition results of features such as tongue color, tongue shape, and tongue coating. At the same time, the method also converts the multi-spectral tongue image processing data into a graph structure, constructs a graph neural network model to capture the relationship between pixel points, and further extracts relational features. Through multimodal fusion technology, feature vectors of different modalities are fused into a comprehensive tongue image feature vector. Finally, a preset classifier is used to make classification decisions on the fused feature vector to obtain the analysis results of traditional Chinese medicine tongue image information. In addition, the present invention also includes functions such as developing an interactive interface, providing a feedback mechanism, and tracking tongue image changes to form a complete, closed-loop learning system.

[0029] Beneficial effects:

[0030] Improve diagnostic accuracy: By comprehensively using multispectral technology and deep learning algorithms, the present invention can more accurately capture subtle changes in tongue images, thereby improving the accuracy and reliability of TCM tongue diagnosis.

[0031] Enhanced objectivity: The present invention reduces the subjective judgment component in traditional tongue diagnosis by quantitatively analyzing the characteristics of tongue images, making the tongue diagnosis results more objective.

[0032] Advantages of multimodal fusion: By fusing feature information of multiple modalities such as multispectral, neural network and fuzzy reasoning, the present invention can more comprehensively reflect the complexity and diversity of tongue images and improve the robustness of classification decisions.

[0033] Interactive interface and feedback mechanism: The interactive interface and feedback mechanism provided by the present invention allow doctors and users to intuitively view and interpret tongue diagnosis results, and make corrections or supplements according to actual needs, thereby continuously optimizing system performance.

[0034] Continuous learning and optimization: By introducing a dynamic feedback mechanism, the present invention can continuously optimize according to actual classification results and expert feedback, and continuously improve the classification performance and accuracy of the system.

[0035] Health status tracking and prediction: Through long-term tracking and analysis of the tongue images of the same patient, the present invention helps doctors to have a deeper understanding of the changing trends of the patient's health status and provide strong support for prevention and treatment. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A schematic diagram of an embodiment of a multi-spectral TCM tongue image information analysis method in an embodiment of the present invention;

[0037] Figure 2It is another embodiment schematic diagram of the multi-spectral TCM tongue image information analysis method in the embodiment of the present invention;

[0038] Figure 3 A schematic diagram of an embodiment of a multi-spectral Chinese medicine tongue image information analysis device in an embodiment of the present invention;

[0039] Figure 4 It is a schematic diagram of an embodiment of a multi-spectral Chinese medicine tongue image information analysis device in an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The embodiment of the present invention provides a multi-spectral Chinese medicine tongue image information analysis device and method, which improves the accuracy of tongue image feature extraction and classification decision-making through methods such as multimodal fusion and deep learning. The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described here can be implemented in an order other than that illustrated or described here. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0041] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 An embodiment of the multi-spectral TCM tongue image information analysis method in the embodiment of the present invention includes:

[0042] 101. Acquire multispectral tongue image data, and preprocess the multispectral tongue image data to obtain multispectral tongue image processed data;

[0043] It is understandable that the execution subject of the present invention may be a multi-spectral TCM tongue and facial image information analysis device, or a terminal or a server, which is not limited here. The embodiment of the present invention is described by taking a server as the execution subject as an example.

[0044] It should be noted that in the multi-spectral TCM tongue image information analysis, multi-spectral tongue image data is obtained. Equipment settings: Use a spectrometer, set its band range to 400-1000nm, and collect spectral data at intervals of 10nm. Ensure that the equipment is calibrated to ensure the accuracy of the data. Data collection: Select a healthy adult volunteer as the subject. Ask the subject to stick out his tongue and place it in front of the acquisition window of the spectrometer. Start the spectrometer and measure the tongue reflectance spectrum for each band. Measure each band 3 times to take the average value.

[0045] Original data record (the following data is an example):

[0046] Band (nm) Reflectivity 1 (%) Reflectivity 2 (%) Reflectivity 3 (%) Average reflectivity (%) 400 10.5 10.3 10.7 10.5 410 9.8 9.9 9.7 9.8 ... ... ... ... ... 1000 5.2 5.3 5.1 5.2

[0051] Preprocess the multispectral tongue image data. Data cleaning: remove obviously abnormal data points, such as outliers caused by equipment failure or subject movement. Data smoothing: use Savitzky-Golay filter to smooth the data to reduce noise. Data normalization: normalize the reflectance data to the range of [0,1] for subsequent analysis. Normalization formula: Normalized reflectance = (original reflectance - minimum value) / (maximum value - minimum value). Preprocessed data (based on the original data example):

[0052] Band (nm) Average reflectivity (%) Normalized reflectivity 400 10.5 0.875 410 9.8 0.817 ... ... ... 1000 5.2 0.000

[0057] Through the above steps, we have successfully acquired multispectral tongue image data and preprocessed it to obtain normalized reflectance data. These data can be used for subsequent feature extraction, pattern recognition, disease diagnosis and other tasks.

[0058] 102. Constructing a multi-parameter adaptive neural fuzzy inference system model based on the multi-spectral tongue image processing data, training and optimizing the multi-parameter adaptive neural fuzzy inference system model to recognize tongue image features by adjusting the fuzzy rule base, performing feature extraction on the multi-spectral tongue image processing data based on the multi-parameter adaptive neural fuzzy inference system model after training and optimization, and obtaining a feature vector of the multi-spectral tongue image processing data;

[0059] It should be noted that the previously preprocessed multispectral tongue image data, such as reflectance, absorbance and other features, are used. Assume that we have 100 sample data, each sample contains reflectance data in different spectral bands. ANFIS model construction: Determine input and output: The input is multispectral tongue image processing data (such as normalized reflectance), and the output is the feature classification or health status assessment of the tongue image. Select appropriate fuzzy logic operators and membership functions. Initialize the fuzzy rule base, and set the initial rules based on expert knowledge or preliminary data analysis.

[0060] Training and optimization of multi-parameter adaptive neural fuzzy inference system model, data division: 100 samples are divided into training set (80 samples) and test set (20 samples). Training process: Use the training set to train the ANFIS model, and adjust the model parameters through the back propagation algorithm and the least squares method. Set the training parameters such as the number of iterations and learning rate. After 1000 iterations, the model reaches a stable state. Performance evaluation: Use the test set to evaluate the performance of the model, such as accuracy, recall, etc. Set the accuracy of the model on the test set to 90%.

[0061] Feature extraction, application of trained model: Apply the trained and optimized ANFIS model to the entire data set (including training set and test set). The model will output the tongue image feature classification or health status assessment for each sample. Feature vector extraction: For each sample, extract the feature vector output by the model. These feature vectors can include the membership and rule activation degree of tongue image features such as reflectance and absorbance in different spectral bands.

[0062] We obtained a feature vector containing 10 features, representing the tongue image characteristics in different spectral bands.

[0063] Specific data example, sample data (partial display):

[0064] Sample 1: The reflectivity in the 400nm band is 0.85, 410nm is 0.83, ..., 1000nm is 0.52.

[0065] Sample 2: The reflectivity in the 400nm band is 0.87, 410nm is 0.82, ..., 1000nm is 0.50.

[0066] ... (and so on, a total of 100 samples)

[0067] ANFIS model output (partial display):

[0068] For sample 1, the eigenvector output by the model is: [0.75, 0.80, ..., 0.20] (10 eigenvalues ​​in total)

[0069] For sample 2, the eigenvector output by the model is: [0.78, 0.82, ..., 0.18] (10 eigenvalues ​​in total)

[0070] These feature vectors represent the model's recognition results of tongue image features in different spectral bands.

[0071] 103. According to the multispectral tongue image processing data, the data is converted into a graph structure, wherein each pixel is regarded as a node, and the connection relationship between the nodes is determined by the spatial distance and color similarity between the pixels. Based on the converted graph structure, a graph neural network model is constructed, and the graph neural network model is trained to enable the model to learn the relationship between the nodes and the feature representation of the nodes. The relationship between the pixels is captured by the trained graph neural network model to obtain the relationship characteristics of the multispectral tongue image processing data;

[0072] It should be noted that the pre-processed multispectral tongue image data includes tongue images of various spectral bands. We selected a tongue image under ultraviolet light, a visible light, and an infrared light, and the resolution of each image is 640x480 pixels.

[0073] Graph structure conversion: Each pixel of each tongue image is considered as a node, so there are 640x480=256,000 nodes in total. The connection relationship between nodes is determined by the spatial distance and color similarity between pixels. For example, a threshold can be set so that when the spatial distance between two pixels is less than a certain value and the color similarity is higher than another value, they are connected. Color similarity can be measured by calculating the distance between two pixels in a color space (such as RGB or HSV).

[0074] Graph neural network model construction and training,Model construction: Use graph convolutional network (GCN) or graph attention network (GAT) as the basis of the graph neural network model. The input of the model is the graph structure constructed above, where the node features can include the color value of the pixel, spectral band information, etc. The output of the model is the feature representation of each node, which will capture the relationship between the pixels.

[0075] Model training: Since this is an unsupervised learning task, we can use reconstruction loss (such as mean square error MSE) as the training target, that is, let the model learn how to reconstruct the features of the current node based on the information of the neighboring nodes. Set training parameters such as learning rate and batch size, and use an optimizer (such as Adam) for model training. We trained for 10 epochs, each of which contained 100 iterations.

[0076] Feature extraction and analysis,Feature extraction: After training is completed, the trained graph neural network model is used to extract features from each node in the graph structure. The extracted features may include the embedding vector of the node, the strength of the relationship between nodes, etc.

[0077] Data analysis: Visualize the extracted features, such as using t-SNE or PCA to reduce the dimension and then draw a scatter plot to observe the clustering between different pixels. Different areas or abnormal points on the tongue surface can be identified by calculating the similarity between features.

[0078] Specific data example: Number of nodes: 256,000 (based on a resolution of 640x480); Number of connections: Each node is connected to an average of 10 nodes that are spatially close and similar in color; Training parameters: Learning rate = 0.001, batch size = 32, number of iterations = 1000 times / epoch, total number of epochs = 10; Feature dimension: The feature vector extracted from each node is 128 dimensions; Accuracy improvement: The features extracted through graph neural networks have improved the accuracy of tongue image classification tasks by 10% compared to traditional features.

[0079] 104. Perform multimodal fusion based on the multispectral tongue image processing data feature vector and the multispectral tongue image processing data feature vector to obtain a comprehensive tongue image feature vector;

[0080] It should be noted that in the analysis of multispectral TCM tongue image information, multimodal fusion is a key step, which can effectively integrate the tongue image features from different spectral bands, so as to obtain a more comprehensive and accurate description of the tongue image features. Multimodal fusion feature vector extraction: For each band of image, use an appropriate algorithm (such as CNN) to extract the feature vector. Set the feature vector dimension of each band to 512 dimensions. Feature fusion method: Select a fusion method, such as weighted averaging, stacking, or fusion through a machine learning algorithm. In this example, we choose to use the weighted average method for fusion, that is, weighted averaging of the values ​​on each feature dimension.

[0081] Fusion process: For each sample, the feature vectors of the three bands (visible light, infrared light, and ultraviolet light) are weighted averaged. Assuming the feature vectors of the three bands are (V_{vis}), (V_{ir}), and (V_{uv}), the fused feature vector (V_{fused}) can be calculated as: (V_{fused} = \frac{V_{vis} + V_{ir} + V_{uv}}{3}) (In practical applications, the weights can be adjusted according to the importance of the bands).

[0082] Results: Through the above fusion process, we obtained a comprehensive tongue image feature vector (V_{fused}) that fused information from multiple spectral bands. This feature vector contains information from different bands and can therefore describe the characteristics of the tongue image more comprehensively.

[0083] Specific data example: Number of samples: 100 samples; Number of bands: 3 (visible light, infrared light, ultraviolet light); Feature vector dimension: The feature vector of each band is 512-dimensional; Fusion method: weighted average method; Fusion result: A 512-dimensional comprehensive tongue image feature vector (V_{fused}) is obtained for subsequent tongue image analysis or disease diagnosis.

[0084] 105. Use the preset classifier to classify and decide the fused comprehensive tongue image feature vector to obtain the multi-spectral TCM tongue image information analysis result;

[0085] It should be noted that in the analysis of multi-spectral TCM tongue image information, using a preset classifier to make classification decisions on the fused comprehensive tongue image feature vector is an important part of the analysis process.

[0086] The multimodal fusion method described above was used to fuse the different spectral feature vectors of each sample into a comprehensive tongue image feature vector. The dimension of the fused feature vector was set to 512 dimensions.

[0087] Classifier selection and training, Classifier selection: Select a suitable classifier, such as support vector machine (SVM), random forest or neural network. In this example, we choose SVM as the classifier. Classifier training: Use the labeled tongue image data as the training set to train the SVM classifier. We have 80 labeled samples for training and the remaining 20 samples for testing. Adjust the parameters of SVM through cross-validation and other techniques to achieve the best classification performance.

[0088] Classification decision and evaluation,Classification decision: Use the trained SVM classifier to classify the fused comprehensive tongue image feature vector. The classifier classifies each feature vector into the corresponding tongue image category, such as "healthy", "damp heat", "yin deficiency", etc. Result evaluation: Use the test set (20 samples) to evaluate the performance of the classifier. Calculate the classifier's accuracy, recall rate, F1 score and other indicators. Assume that our classifier achieves 90% accuracy on the test set.

[0089] Specific data example: number of samples: 100 (80 training samples, 20 test samples); feature vector dimension: 512 dimensions (after fusion); classifier type: support vector machine (SVM); classification categories: there are 5 categories (such as "healthy", "damp heat", "yin deficiency", "yang deficiency", "qi stagnation"); classifier performance: test set accuracy = 90%, recall rate = 85%, F1 score = 0.88.

[0090] In an embodiment of the present invention, multiple steps are included, including data acquisition and preprocessing, feature extraction, multimodal fusion, and classification decision-making. First, the method collects multispectral tongue image data through a spectrometer, and performs preprocessing operations such as cleaning, smoothing, and normalization on these data to improve data quality. Next, a multi-parameter adaptive neural fuzzy inference system (ANFIS) model is used to extract features from the preprocessed data to obtain a feature vector of the tongue image. Then, the relationship between pixels is further captured through a graph neural network model to obtain relationship features. These feature vectors are then integrated through multimodal fusion technology to form a comprehensive tongue image feature vector that integrates information from different spectral bands. Finally, a preset classifier (such as SVM) is used to make a classification decision on the fused feature vector, thereby obtaining the analysis result of the tongue image information. Through the collection and analysis of multispectral data, the details of the tongue image that are difficult to observe under the traditional single spectrum can be captured, thereby improving the accuracy and reliability of TCM tongue diagnosis; multispectral data provides the reflection and absorption characteristics of the tongue surface in different spectral bands, which provide a richer information basis for TCM diagnosis; through the use of ANFIS model and graph neural network model, the intelligent extraction and analysis of tongue image features are realized, reducing the diagnostic errors caused by human factors; multimodal fusion technology can effectively integrate the feature vectors of different spectral bands to form a more comprehensive and representative tongue image feature vector, which helps to improve the performance of the classifier; this method can provide TCM with an objective basis for tongue diagnosis and assist doctors in disease diagnosis and treatment plan formulation; this technical process is not only applicable to TCM tongue diagnosis, but can also be extended to other medical fields or biometric recognition fields, and has broad application prospects. In summary, the multispectral TCM tongue image information analysis method provides a more scientific, accurate and efficient analysis method for TCM tongue diagnosis by combining multispectral data collection, intelligent feature extraction and classification decision-making, which helps to improve the level and quality of TCM diagnosis.

[0091] See also Figure 2 Another embodiment of the multi-spectral Chinese medicine tongue image information analysis method in the embodiment of the present invention includes:

[0092] 201. Acquire multispectral tongue image data, and preprocess the multispectral tongue image data to obtain multispectral tongue image processed data;

[0093] Specifically, based on the dynamic spectral acquisition data, by dynamically adjusting the settings of the filter or the spectrometer, the reaction of the tongue surface in different spectral bands is captured at different time points; the tongue surface images in different spectral bands are feature fused to create a comprehensive feature representation to obtain multispectral tongue surface image data; and / or nonlinear feature transformation technology, including kernel method or manifold learning, is applied to reveal the intrinsic structure and relationship of the tongue surface data to obtain multispectral tongue surface image data; image registration is performed based on the multispectral tongue surface image data to ensure the pixel correspondence between different spectral images, and images of different spectra are automatically aligned and fused using an image registration algorithm to form a multispectral tongue surface image data cube, and noise removal and enhancement are performed based on the multispectral tongue surface image data cube; spectral features related to the physiological state of the tongue surface, including reflectance, absorbance, and transmittance, are extracted from the multispectral tongue surface image data cube, and the spectral features are reduced in dimension using principal component analysis or independent component analysis methods, and then standardized and normalized to obtain pre-processed multispectral tongue surface image processing data;

[0094] It should be noted that in the dynamic spectral data collection, the equipment preparation is: a hyperspectral imaging spectrometer is used, the spectral range covers 400-1000nm, and the wavelength resolution is better than 2.5nm. The acquisition speed is set to 128FPS for the full spectrum. Spectral acquisition: a volunteer is selected to collect tongue images under constant light conditions indoors. The settings of the spectrometer are dynamically adjusted to capture the reactions of the tongue surface in different spectral bands from 400nm to 1000nm at intervals of 10nm. Five images are collected in each band to ensure the stability of the data.

[0095] Multispectral tongue image data fusion and feature transformation: Image fusion: The tongue images in different spectral bands are fused. Specifically, the images of each band are grayed and then fused using the weighted average method to obtain a comprehensive feature representation, namely the multispectral tongue image data. Feature transformation: Kernel principal component analysis (KPCA) is used as a nonlinear feature transformation technology to process the fused multispectral tongue image data. KPCA reveals the intrinsic structure and relationship of the tongue data and obtains the multispectral tongue image data in the new feature space.

[0096] Image registration and fusion, Image registration: Use feature-based image registration algorithms, such as SIFT (Scale Invariant Feature Transform), to automatically align images of different spectra. Accurate image registration is achieved through steps such as extracting key points, matching descriptors, and estimating transformation models. Image fusion: Fuse the registered images of different spectra to form a multi-spectral tongue image data cube. Specifically, pixel-level fusion methods such as weighted average or Laplacian pyramid fusion can be used.

[0097] Noise removal and enhancement: noise removal and enhancement processing is performed on the multispectral tongue image data cube. Median filtering or Gaussian filtering can be used to remove noise in the image; at the same time, histogram equalization or contrast stretching can be used to enhance the contrast and clarity of the image.

[0098] Spectral feature extraction and preprocessing, spectral feature extraction: extract spectral features related to the physiological state of the tongue surface from the multispectral tongue image data cube, including reflectance, absorptivity and transmittance. These features can reflect the tissue structure and biochemical composition information of the tongue surface. Feature dimension reduction and preprocessing: use principal component analysis (PCA) or independent component analysis (ICA) methods to reduce the dimension of the extracted spectral features to reduce data redundancy and computational complexity. Then, standardize and normalize the features after dimension reduction to obtain the preprocessed multispectral tongue image processing data.

[0099] Specific data example: 1 volunteer, 25 years old, male. Spectral images of 61 bands were collected at intervals of 10nm in the range of 400-1000nm. 5 images were collected in each band, totaling 305 original images. After image fusion, a multispectral tongue image data map represented by comprehensive features was obtained. After applying KPCA for feature transformation, a multispectral tongue image data matrix in the new feature space was obtained, with a dimension of 61x61 (number of bands x number of principal components). After image registration, a multispectral tongue image data cube was formed, with a size of 640x480x61 (width x height x number of bands). After noise removal and enhancement processing, the image quality was significantly improved. Spectral features such as reflectance, absorbance and transmittance were extracted, and PCA dimensionality reduction processing was performed. Finally, a 20-dimensional pre-processed multispectral tongue image processing data vector was obtained.

[0100] 202. Constructing a multi-parameter adaptive neural fuzzy inference system model based on the multi-spectral tongue image processing data, training and optimizing the multi-parameter adaptive neural fuzzy inference system model to recognize tongue image features by adjusting the fuzzy rule base, performing feature extraction on the multi-spectral tongue image processing data based on the multi-parameter adaptive neural fuzzy inference system model after training and optimization, and obtaining a feature vector of the multi-spectral tongue image processing data;

[0101] Specifically, we design an initial fuzzy rule base, which is defined based on the characteristics of tongue color, tongue coating, and tongue shape. For each fuzzy rule in the fuzzy rule base, we define an output function:

[0102] O_i=w_i\cdotf_i(S,T,F);

[0103] Among them, (O_i) is the output of the (i)th rule, (w_i) is the weight of the rule, (f_i(S,T,F)) is the fuzzy inference function based on the rule, (S,T,F) are the input values ​​of tongue color, tongue coating, and tongue shape respectively; (f_i(S,T,F)) is the inference function based on the (i)th fuzzy rule, which is determined according to the input value and the rules in the fuzzy rule base;

[0104] The final output is the weighted sum of all rule outputs:

[0105] O=\sum_{i}O_i;

[0106] Design a multi-parameter parallel processing mechanism, use modern parallel computing technology to allow multiple parameters to perform fuzzy reasoning in the fuzzy rule base at the same time; combine neural networks and fuzzy rule bases to build an adaptive multi-parameter adaptive neural fuzzy reasoning system model; train the multi-parameter adaptive neural fuzzy reasoning system model based on the multi-spectral tongue image processing data to obtain a trained multi-parameter adaptive neural fuzzy reasoning system model; use the multi-parameter adaptive neural fuzzy reasoning system model after training and optimization to extract features from the multi-spectral tongue image processing data to obtain recognition results including tongue color, tongue shape, and tongue coating features, encode the extracted recognition results, and generate feature vectors of the multi-spectral tongue image processing data;

[0107] It should be noted that the fuzzy sets and their membership functions for tongue color (S), tongue coating (T), and tongue shape (F) are determined. For example, tongue color can be divided into light red, red, crimson, etc., tongue coating can be divided into thin white, yellow greasy, thick greasy, etc., and tongue shape can be divided into fat, thin, and tooth marks, etc.

[0108] Based on TCM theory and expert experience, fuzzy rules are designed. For example: Rule 1: If the tongue is red, the tongue coating is yellow and greasy, and the tongue is fat, it may indicate internal dampness and heat. Rule 2: If the tongue is light red, the tongue coating is thin and white, and the tongue shape is normal, it may indicate a healthy state.

[0109] Define the output function (O_i = w_i \cdot f_i(S, T, F)) for each fuzzy rule. Where (w_i) is the weight of the rule and (f_i(S, T, F)) is the fuzzy inference function based on the rule.

[0110] Multi-parameter parallel processing mechanism uses modern parallel computing technology (such as GPU acceleration, multi-threading, etc.) to achieve parallel processing of multiple parameters such as tongue color, tongue coating, and tongue shape. Parallel algorithms are designed to enable multiple fuzzy reasoning processes to be carried out simultaneously to improve processing efficiency.

[0111] Construct an adaptive neuro-fuzzy inference system model, combine neural network and fuzzy rule base, and build an adaptive multi-parameter adaptive neuro-fuzzy inference system (ANFIS) model. In the ANFIS model, the weights and biases of the neural network are adjusted through training to adapt to different tongue image data.

[0112] Train the ANFIS model and prepare multispectral tongue image processing data as a training set, including tongue color, tongue coating, tongue shape and other features. Use the training set to train the ANFIS model and adjust the model parameters through optimization methods such as the back propagation algorithm. During the training process, monitor the performance and error of the model until a satisfactory training effect is achieved.

[0113] Feature extraction and encoding: Use the trained and optimized ANFIS model to extract features from multispectral tongue image processing data. The extracted features include tongue color, tongue shape, tongue coating, etc. These features are encoded, for example, using one-hot encoding (One-Hot Encoding) or label encoding (Label Encoding). Generate feature vectors of multispectral tongue image processing data for subsequent classification, diagnosis and other tasks.

[0114] Specific data example:

[0115] The training set contains 1000 samples, each of which includes multispectral tongue image data with features such as tongue color, tongue coating, and tongue shape. The fuzzy rule base contains 10 rules, each of which corresponds to a specific TCM syndrome. The ANFIS model uses a 3-layer neural network structure with 20 hidden layer nodes and the output layer is the classification result of TCM syndrome. After training, the accuracy of the ANFIS model on the test set reached more than 90%. For a new tongue image sample, the feature vector extracted by the ANFIS model is: [0.1, 0.8, 0.1, ..., 0] (indicating features such as reddish tongue color, yellow and greasy tongue coating, and slightly fat tongue shape), which can be used for subsequent TCM syndrome classification and diagnosis tasks.

[0116] 203. According to the multispectral tongue image processing data, the data is converted into a graph structure, wherein each pixel is regarded as a node, and the connection relationship between the nodes is determined by the spatial distance and color similarity between the pixels. Based on the converted graph structure, a graph neural network model is constructed, and the graph neural network model is trained to enable the model to learn the relationship between the nodes and the feature representation of the nodes. The trained graph neural network model is used to capture the relationship between the pixels, and the relationship characteristics of the multispectral tongue image processing data are obtained;

[0117] Specifically, the multispectral tongue image processing data is converted into a graph structure, each pixel is regarded as a node, a threshold Q is preset, and when the spatial distance between two pixels is less than the threshold Q, it is considered that there is a connection between the two nodes; a graph neural network model is constructed, node features are defined, including the color value, brightness, and contrast of the pixel points, and a graph neural network model is constructed according to the converted graph structure, and a convolution operation is defined in the model to aggregate the neighbor information of the node; and / or the tongue image is divided into multiple superpixel areas using superpixel segmentation technology, each superpixel is regarded as a node, and edges are established according to the similarity and adjacency relationship between superpixels; the graph neural network model is trained according to the historical marked data in the multispectral tongue image processing data, and during the training process, the graph structure is dynamically adjusted according to the feedback of the model, and the dynamic adjustment includes deleting or adding edges, and the graph neural network model is trained; the real-time data in the multispectral tongue image processing data is input into the trained graph neural network model, and the feature representation of each node is obtained through the forward propagation of the graph neural network model, and these feature representations capture the relationship between the pixels, and the relationship characteristics of the multispectral tongue image processing data are obtained;

[0118] It should be noted that the multispectral tongue image processing data is converted into a graph structure, in which each pixel is a node. For example, if the size of a tongue image is 640x480, there will be 307,200 nodes in the graph. Set a spatial distance threshold Q, for example, Q = 5 pixels. Calculate the spatial distance between each pair of pixels. If the spatial distance between two pixels is less than Q, it is considered that there is a connection between the two nodes, that is, add an edge to the graph.

[0119] Build a graph neural network model and define node features, including the color value of the pixel (the value of the three RGB channels), brightness (the average value of the pixel value), and contrast (the difference with the surrounding pixels). These features will be used as the input of the graph neural network. Build a graph neural network model and use the graph convolution layer to aggregate the neighbor information of the node. For example, you can use Graph Convolutional Network (GCN) or Graph Attention Network (GAT) as the basic model.

[0120] Superpixel segmentation and graph structure construction, use superpixel segmentation technology (such as SLIC algorithm) to divide the tongue image into multiple superpixel regions. For example, a 640x480 image can be divided into 1000 superpixel regions. Each superpixel is a node, and the edge is established based on the similarity and adjacency between superpixels. For example, the color histogram similarity between two superpixels can be calculated. If the similarity is above a certain threshold and they are spatially adjacent, an edge is added to the graph.

[0121] Graph neural network model training and optimization: Prepare historical labeled data, such as 1,000 labeled tongue images, for training the graph neural network model. During the training process, dynamically adjust the graph structure based on the feedback from the model. For example, you can delete or add edges to optimize the performance of the model. This can be achieved by monitoring the performance on the validation set and adjusting the connection relationship of the graph accordingly. The optimized graph neural network model is trained.

[0122] Real-time data processing and feature extraction: input the real-time data in the multispectral tongue image processing data into the trained graph neural network model. Through the forward propagation of the graph neural network model, the feature representation of each node is obtained. These feature representations capture the relationship between pixels (or superpixel areas) and form the relational features of the multispectral tongue image processing data.

[0123] Specific data example: Image size: 640x480 pixels; spatial distance threshold Q: 5 pixels; node feature dimension: color value (3 dimensions) + brightness (1 dimension) + contrast (1 dimension) = 5 dimensions; number of superpixel segmentations: 1000 superpixel areas; training data set size: 1000 labeled tongue images; graph neural network model type: Graph Convolutional Network (GCN) or Graph Attention Network (GAT); dynamic adjustment strategy during training: based on the performance on the validation set, adjust the connection relationship of the edges in the graph after each iteration to optimize the model performance.

[0124] After the above steps, we can obtain a graph neural network model that can capture the relationship between pixels (or superpixel regions) in multispectral tongue image data and extract the corresponding relationship features for subsequent analysis.

[0125] 204. Perform multimodal fusion based on the multispectral tongue image processing data feature vector and the multispectral tongue image processing data feature vector to obtain a comprehensive tongue image feature vector;

[0126] Specifically, according to the multispectral tongue image processing data feature vector and the importance of the multispectral tongue image processing data feature vector, different weights are assigned to the feature vectors of each modality, and then weighted summation is performed; and / or the multispectral tongue image processing data feature vector and the feature vector of the multispectral tongue image processing data feature vector are connected in series to form a longer feature vector; and / or tensors are used to represent multimodal data, and the interaction relationship between modalities is learned through tensor decomposition or tensor network, and the multimodal data is set as a tensor (\mathcal{T}), wherein the feature vector of each modality constitutes a mode of the tensor, and a third-order tensor (\mathcal{T}\in\mathbb{R}^{I\timesJ\timesK}) is decomposed into:

[0127] \mathcal{T}=\mathcal{G}\times_1\mathbf{A}\times_2\mathbf{B}\times_3\mathbf{C};

[0128] Among them, (\mathcal{G}) is the core tensor, which captures the interaction between the modalities; (\mathbf{A},\mathbf{B},\mathbf{C}) is a factor matrix, which is related to the feature vector of each modality; and / or a deep learning model is constructed, which has multiple input branches, each branch corresponds to the feature vector of a modality, in the deep layer of the fusion network of the deep learning model, the features of different modalities are fused together through the fusion layer, the fusion network is jointly trained, the parameters of the fusion layer and the parameters of the feature extractors of each modality are optimized, an end-to-end training method is adopted or the feature extractors of each modality are pre-trained separately, and then jointly fine-tuned; a comprehensive tongue image feature vector is obtained;

[0129] It should be noted that the dataset: a multispectral tongue image dataset containing 1,000 samples is used, and each sample contains images of multiple modalities such as visible light, infrared light, and ultraviolet light. Feature extraction: feature extraction is performed on the image of each modality to obtain the respective feature vectors. For example, visible light images can extract features such as color and texture, and infrared images can extract features such as temperature distribution.

[0130] Multimodal feature fusion method: Weighted summation: Different weights are assigned to each modality according to its importance. For example, the weight of the visible light modality is 0.5, the weight of the infrared light modality is 0.3, and the weight of the ultraviolet light modality is 0.2. The weighted summation of the feature vectors of each modality is performed to obtain the fused feature vector.

[0131] Feature vector concatenation: The feature vectors of each mode are directly concatenated to form a longer feature vector. For example, if the length of the visible light feature vector is 100, the length of the infrared light feature vector is 50, and the length of the ultraviolet light feature vector is 80, then the length of the concatenated feature vector is 230.

[0132] Tensor representation and decomposition: Multimodal data is represented as a third-order tensor ((\mathcal{T}\in\mathbb{R}^{I\times J\times K})), where I, J, and K represent the eigenvector dimensions of the visible light, infrared light, and ultraviolet light modes, respectively. Using tensor decomposition methods such as Tucker decomposition, the tensor (\mathcal{T}) is decomposed into a core tensor (\mathcal{G}) and factor matrices (\mathbf{A}), (\mathbf{B}), and (\mathbf{C}). The core tensor (\mathcal{G}) is used to capture the interaction between the modes, and the factor matrix is ​​used to correlate with the eigenvector of each mode.

[0133] Deep learning model fusion: Build a deep learning model, such as a convolutional neural network (CNN), with three input branches corresponding to the feature vectors of visible light, infrared light, and ultraviolet light modalities. In the deep layer of the model, the features of different modalities are fused together through fusion layers (such as fully connected layers or convolutional layers). Use an end-to-end training method to optimize the parameters of the fusion layer and the parameters of the feature extractors of each modality. Alternatively, pre-train the feature extractors of each modality separately, and then perform joint fine-tuning.

[0134] Results and Evaluation,After processing by the above multimodal feature fusion method, we can obtain a comprehensive tongue image feature vector.,This feature vector integrates the information of multiple modalities and can more comprehensively reflect the characteristics of the tongue image.,In order to evaluate the fusion effect, the fused feature vector can be evaluated,using indicators such as classification accuracy and recall rate.,For example, in the tongue image classification task, the fused feature vector is used for classification,and compared with the feature vector of a single modality to verify the effectiveness of the fusion method.

[0135] Specific data examples: Data set size: 1000 samples; Number of modalities: 3 (visible light, infrared light, ultraviolet light); Feature vector dimension: 100 dimensions for visible light, 50 dimensions for infrared light, and 80 dimensions for ultraviolet light; Weighted sum weight distribution: 0.5 for visible light, 0.3 for infrared light, and 0.2 for ultraviolet light; Tensor decomposition dimension: I=100, J=50, K=80 (corresponding to the feature vector dimension); Deep learning model structure: Convolutional neural network (CNN) with three input branches and one fusion layer; Evaluation indicators: Classification accuracy, recall rate, etc.

[0136] 205. Using a preset classifier to make a classification decision on the fused comprehensive tongue image feature vector, and obtaining a multi-spectral TCM tongue image information analysis result;

[0137] Specifically, one or more classifiers are selected, including support vector machines, random forests, and neural networks. The classifiers are optimized according to the characteristics of multispectral tongue image data, including adjusting the kernel function or parameters of SVM and optimizing the architecture of the neural network; an integrated learning method is used to improve the accuracy of classification by combining the prediction results of multiple classifiers, and an integrated strategy for dynamic weight adjustment is designed to dynamically adjust the weight of each classifier in the final decision according to its performance on the validation set; the selected and dimensionally reduced feature vectors are input into a preset classifier for classification decision; and / or a multi-level classification strategy is designed to first perform a rough classification of the tongue image, and then perform a fine classification of each category to obtain a multispectral TCM tongue image information analysis result; the multispectral TCM tongue image information analysis result is post-processed, and a confidence threshold is used to filter the multispectral TCM tongue image information analysis results with low confidence, or multiple rounds of classification are performed to improve the accuracy; and / or a result interpretation module is designed to provide TCM theoretical explanations and suggestions for each multispectral TCM tongue image information analysis result;

[0138] It should be noted that the dataset: a dataset containing 1,000 multispectral tongue image samples is used, of which 800 samples are used for training and 200 samples are used for testing. Feature extraction: multi-dimensional features such as color, texture, and shape are extracted from each sample to form a high-dimensional feature vector. Feature selection and dimensionality reduction: principal component analysis (PCA) and other methods are used to reduce the dimensionality of the feature vector to reduce redundant information and improve classification efficiency.

[0139] Classifier selection and optimization,Classifier selection: Support vector machine (SVM), random forest and neural network are selected as candidate classifiers. SVM optimization: Adjust the kernel function of SVM to radial basis function (RBF), and determine the optimal parameters C and gamma through cross-validation. Neural network optimization: Design a neural network architecture with input layer, hidden layer and output layer. Determine the optimal number of hidden layer nodes and activation function type through experiments.

[0140] Ensemble learning and dynamic weight adjustment,Ensemble learning: integrate the prediction results of the three classifiers SVM, random forest and neural network. Dynamic weight adjustment: dynamically adjust the weight of each classifier in the final decision according to its accuracy on the validation set. For example, if the accuracy of SVM on the validation set is 90%, random forest is 85%, and neural network is 88%, then their weights can be set to 0.4, 0.3 and 0.3 respectively.

[0141] Multi-level classification strategy, coarse classification: first use SVM to roughly classify the tongue image, such as into healthy, damp heat, cold dampness, etc. Fine classification: for each major category, use random forest or neural network for fine classification, such as further dividing the damp heat category into mild damp heat, moderate damp heat and severe damp heat.

[0142] Post-processing and result interpretation,Post-processing: Set a confidence threshold (such as 0.8) and filter out classification results below the threshold to improve accuracy. For the filtered results, multiple rounds of classification can be performed to further improve accuracy. Result interpretation: Design a result interpretation module to provide explanations and suggestions for each classification result based on TCM theory. For example, for tongue images in the "damp heat" category, the interpretation module may recommend that patients add light foods to their diet and avoid greasy and spicy foods.

[0143] Specific data example:

[0144] Dataset size: 1000 samples (800 training samples, 200 test samples); Feature dimension: After PCA dimensionality reduction, the feature vector dimension is reduced from the original 1000 dimensions to 50 dimensions; SVM parameters: C=100, gamma=0.01 (determined by cross-validation); Neural network architecture: input layer-50 nodes, hidden layer-100 nodes (activation function is ReLU), output layer-number of classification categories; Classifier accuracy: SVM-90%, random forest-85%, neural network-88% (performance on the validation set); Ensemble learning weights: SVM-0.4, random forest-0.3, neural network-0.3 (dynamically adjusted according to the accuracy of the validation set); Confidence threshold: 0.8 (used to filter low-confidence results).

[0145] 206. Develop an interactive interface that allows users to intuitively view and interpret the results of multi-spectral TCM tongue image information analysis;

[0146] Provide a feedback mechanism so that users can correct or supplement the analysis results of multi-spectral TCM tongue and facial information, and continuously optimize the classifier and fusion strategy through user feedback to form a closed-loop learning system;

[0147] And / or introduce a dynamic feedback mechanism so that the system can be continuously optimized according to the actual classification effect and expert feedback. When experts object to the analysis results of multi-spectral TCM tongue and facial information, the system automatically records these feedbacks and adjusts the model parameters or adds new training samples accordingly to improve future classification performance;

[0148] And / or for the same patient, a tongue image history database is established to track the changes in tongue image over time, and by analyzing the long-term trend of tongue image changes, doctors can be provided with deeper insights into health conditions;

[0149] and / or develop an interactive tongue image analysis tool that allows doctors to manually adjust the weights and thresholds of tongue image features to more intuitively explore the impact of different feature combinations on the results of multispectral TCM tongue image information analysis;

[0150] It should be noted that an intuitive and easy-to-use interactive interface is designed to allow users to upload multispectral tongue images. The interface displays the results of multispectral TCM tongue information analysis, including tongue classification, health status assessment, etc. It provides a result interpretation function to present complex analysis results to users in a concise and clear manner in the form of charts or text.

[0151] Feedback mechanism implementation: set feedback button in the interface to allow users to modify or supplement the analysis results. Collect and record user feedback data, such as user confirmation, modification or supplementary information on classification results. Use user feedback data to continuously optimize classifiers and fusion strategies. For example, adjust the kernel function parameters of SVM or the weights of neural networks based on user feedback.

[0152] Dynamic feedback and continuous optimization: When experts raise objections to the results of the multispectral TCM tongue image analysis, the system automatically records these feedbacks. Automatically adjust model parameters or add new training samples based on expert feedback. For example, if an expert points out that a tongue image should be classified as "damp heat" rather than "damp cold", the system will add the tongue image and its correct classification to the training set and retrain the model. Through continuous iteration and optimization, the accuracy and generalization ability of the classifier are improved.

[0153] A tongue image history record database is established for each patient to store the results of previous tongue image analysis. The long-term change trend of tongue image, such as changes in color, texture and other characteristics, is analyzed. Based on the change trend of tongue image, doctors are provided with deeper insights into health conditions and diagnostic recommendations.

[0154] Development of an interactive tongue analysis tool, an interactive tongue analysis tool was developed that allows doctors to manually adjust the weights and thresholds of tongue features. Doctors can use this tool to intuitively explore the impact of different feature combinations on the results of multi-spectral TCM tongue image information analysis. The tool provides a real-time preview function, allowing doctors to instantly see changes in analysis results when adjusting parameters.

[0155] Specific data examples: Interactive interface: The user uploaded 1,000 multispectral tongue images for testing, and the interface successfully displayed the classification results and health assessment reports of each image. Feedback mechanism: Among the 1,000 images, the user proposed corrections to the classification results of 50 of them. The system optimized the classifier based on these feedbacks, which increased the classification accuracy by 2%. Dynamic feedback and continuous optimization: Experts objected to the classification results of 10 of the images. The system automatically recorded these feedbacks, adjusted the model parameters and added new training samples accordingly. After retraining, the accuracy of the model on the validation set increased by 1.5%. Tongue history database: A tongue history database was established for 50 patients, tracking the changes in their tongue images within 3 months. By analyzing these changes, the doctor provided targeted health advice to 10 of the patients. Interactive tongue analysis tool: The doctor used this tool to adjust the feature weights and thresholds of 200 tongue images. Through the adjustment, the doctor found that the weight changes of certain specific features had a significant impact on the classification results, thereby improving the accuracy of the diagnosis.

[0156] In the embodiment of the present invention, the whole process from data acquisition, preprocessing, feature extraction, multimodal fusion to classification decision-making, result display and feedback optimization is covered. Specifically, the technology obtains multispectral tongue image data through dynamic spectral acquisition technology, and uses advanced image processing and machine learning algorithms to preprocess and extract features of the data. Subsequently, by constructing a multi-parameter adaptive neural fuzzy inference system model and a graph neural network model, the complex features and their relationships in the tongue image are deeply explored. Next, multimodal fusion technology is used to effectively integrate the features under different spectral bands to form a comprehensive tongue image feature vector. Finally, the optimized classifier is used to make classification decisions on the feature vector, and the results are displayed to the user through an interactive interface, while collecting user feedback to continuously optimize the model performance. Through multispectral data acquisition and advanced data processing technology, this technology can capture the subtle changes of tongue images in different spectral bands, thereby providing more accurate tongue image information, which is helpful for the accurate diagnosis of TCM tongue diagnosis; using machine learning algorithms such as adaptive neural fuzzy inference system and graph neural network model, this technology can automatically extract key features in tongue images and learn the relationship between features, realizing intelligent analysis of tongue image information; through multimodal fusion technology, this technology can integrate information in different spectral bands to form a comprehensive tongue image feature vector, providing richer diagnostic basis for TCM tongue diagnosis; the interactive interface allows users to intuitively view and interpret the analysis results, and provides a feedback mechanism so that users can correct or supplement the results. This interactive design not only improves the user experience, but also continuously optimizes the model performance through user feedback, forming a closed-loop learning system; establishing a tongue image history record database to track the changes of tongue images over time, which helps doctors analyze the long-term change trend of tongue images and provide patients with deeper insights into health status and personalized treatment recommendations; it is not only applicable to the field of TCM tongue diagnosis, but can also be extended to other medical image analysis and biometric recognition fields, with broad application prospects. In summary, this technology provides a comprehensive, accurate and intelligent information analysis method for TCM tongue diagnosis by comprehensively using multispectral data acquisition, advanced data processing and machine learning algorithms, and interactive interface design, which helps to improve the diagnostic accuracy and efficiency of TCM tongue diagnosis and provide strong support for TCM clinical practice.

[0157] The above describes the multi-spectral TCM tongue and face image information analysis method in the embodiment of the present invention. The following describes the multi-spectral TCM tongue and face image information analysis device in the embodiment of the present invention. Figure 3 In one embodiment of the present invention, a multi-spectral Chinese medicine tongue image information analysis device comprises:

[0158] An acquisition module 301 is used to acquire multispectral tongue image data and preprocess the multispectral tongue image data to obtain multispectral tongue image processed data;

[0159] The first processing module 302 is used to construct a multi-parameter adaptive neural fuzzy inference system model based on the multi-spectral tongue image processing data, train and optimize the multi-parameter adaptive neural fuzzy inference system model to recognize tongue image features by adjusting the fuzzy rule base, and extract features from the multi-spectral tongue image processing data based on the multi-parameter adaptive neural fuzzy inference system model after training and optimization to obtain a feature vector of the multi-spectral tongue image processing data;

[0160] The second processing module 303 is used to convert the multispectral tongue image processing data into a graph structure, wherein each pixel point is used as a node, and the connection relationship between the nodes is determined by the spatial distance and color similarity between the pixels. Based on the converted graph structure, a graph neural network model is constructed, and the graph neural network model is trained to enable it to learn the relationship between the nodes and the feature representation of the nodes. The trained graph neural network model is used to capture the relationship between the pixels to obtain the relationship features of the multispectral tongue image processing data.

[0161] Setting module 304, for performing multimodal fusion according to the multispectral tongue image processing data feature vector and the multispectral tongue image processing data feature vector to obtain a comprehensive tongue image feature vector;

[0162] The allocation module 305 is used to use a preset classifier to make classification decisions on the fused comprehensive tongue image feature vector to obtain the multi-spectral TCM tongue image information analysis results.

[0163] In the embodiment of the present invention, through multispectral data acquisition and advanced data processing technology, the device can capture the subtle changes of tongue images in different spectral bands, thereby providing more accurate tongue image information, which is helpful for the accurate diagnosis of TCM tongue diagnosis; combined with machine learning algorithms such as ANFIS and GNN, the device can automatically extract key features in tongue images and learn the relationship between features, realizing intelligent analysis of tongue image information. This reduces the interference of human factors and improves the objectivity and accuracy of the analysis; through multimodal fusion technology, the device can integrate information from different processing modules to form a comprehensive tongue image feature vector. This step effectively utilizes the richness of multispectral data and provides a more comprehensive diagnostic basis for TCM tongue diagnosis; the design of the device has good scalability and customizability. By adjusting model parameters and adding new processing modules, the system performance can be further optimized to adapt to different clinical needs; the application of the device has promoted the modernization process of TCM tongue diagnosis and provided strong support for TCM clinical practice. Through objective and quantitative tongue image analysis, it is helpful to improve the accuracy and scientificity of TCM diagnosis.

[0164] above Figure 3The multi-spectral Chinese medicine tongue and facial image information analysis device in the embodiment of the present invention is described in detail from the perspective of modular functional entities. The multi-spectral Chinese medicine tongue and facial image information analysis device in the embodiment of the present invention is described in detail from the perspective of hardware processing.

[0165] Figure 4 4 is a schematic diagram of the structure of a multi-spectral Chinese medicine tongue and face image information analysis device provided by an embodiment of the present invention. The multi-spectral Chinese medicine tongue and face image information analysis device 400 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 410 (for example, one or more processors) and a memory 420, and one or more storage media 430 (for example, one or more mass storage devices) storing application programs 433 or data 432. Among them, the memory 420 and the storage medium 430 can be short-term storage or permanent storage. The program stored in the storage medium 430 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the multi-spectral Chinese medicine tongue and face image information analysis device 400. Furthermore, the processor 410 may be configured to communicate with the storage medium 430 to execute a series of instruction operations in the storage medium 430 on the multi-spectral Chinese medicine tongue and face image information analysis device 400.

[0166] The multi-spectral TCM tongue image information analysis device 400 may also include one or more power supplies 440, one or more wired or wireless network interfaces 450, one or more input and output interfaces 460, and / or one or more operating systems 431, such as Windows Serve, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will appreciate that Figure 4 The structure of the multi-spectral Chinese medicine tongue and facial image information analysis device shown does not constitute a limitation of the multi-spectral Chinese medicine tongue and facial image information analysis device, and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0167] The present invention also provides a multi-spectral Chinese medicine tongue and facial image information analysis device, which includes a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes the steps of the multi-spectral Chinese medicine tongue and facial image information analysis method in the above-mentioned embodiments.

[0168] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the multi-spectral Chinese medicine tongue image information analysis method.

[0169] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0170] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the whole or part of the solution of the present invention, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0171] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-spectral Chinese medicine tongue image information analysis method, characterized in that: The multi-spectral TCM tongue image information analysis method comprises: Acquiring multispectral tongue image data and preprocessing the multispectral tongue image data to obtain multispectral tongue image processing data, including: Based on the dynamic spectrum acquisition data, the reaction of the tongue surface in different spectral bands can be captured at different time points by dynamically adjusting the filter or the setting of the spectrometer; The tongue surface images in different spectral bands are fused to create a comprehensive feature representation and obtain multispectral tongue surface image data; and / or applying nonlinear feature transformation techniques, including kernel methods or manifold learning, to reveal the intrinsic structure and relationship of tongue surface data to obtain multispectral tongue surface image data; Performing image registration based on the multispectral tongue image data to ensure pixel correspondence between different spectral images, automatically aligning images of different spectra using an image registration algorithm, and fusing them to form a multispectral tongue image data cube, and performing noise removal and enhancement based on the multispectral tongue image data cube; Extracting spectral features related to the physiological state of the tongue from the multispectral tongue image data cube, including reflectance, absorptivity, and transmittance, and performing dimension reduction processing on the spectral features using principal component analysis or independent component analysis, and then performing standardization and normalization to obtain pre-processed multispectral tongue image processing data; A multi-parameter adaptive neural fuzzy inference system model is constructed based on the multi-spectral tongue image processing data, and the recognition ability of the multi-parameter adaptive neural fuzzy inference system model for tongue image features is optimized by adjusting the fuzzy rule base. Feature extraction is performed on the multi-spectral tongue image processing data based on the multi-parameter adaptive neural fuzzy inference system model after training and optimization to obtain a feature vector of the multi-spectral tongue image processing data, including: Design the initial fuzzy rule base, which is defined based on tongue color, tongue coating, and tongue shape. For each fuzzy rule in the fuzzy rule base, we define an output function: O_i=w_i\cdotf_i(S,T,F); Among them, (O_i) is the output of the (i)th rule, (w_i) is the weight of the rule, (f_i(S,T,F)) is the fuzzy inference function based on the rule, (S,T,F) are the input values ​​of tongue color, tongue coating, and tongue shape respectively; (f_i(S,T,F)) is the inference function based on the (i)th fuzzy rule, which is determined according to the input value and the rules in the fuzzy rule base; The final output is the weighted sum of all rule outputs: O=\sum_{i}O_i; Design a multi-parameter parallel processing mechanism, using modern parallel computing technology to allow multiple parameters to perform fuzzy reasoning in the fuzzy rule base simultaneously; Combining neural network and fuzzy rule base, an adaptive multi-parameter adaptive neural fuzzy inference system model is constructed; Training a multi-parameter adaptive neural fuzzy inference system model according to the multi-spectral tongue image processing data to obtain a trained multi-parameter adaptive neural fuzzy inference system model; Using the trained and optimized multi-parameter adaptive neural fuzzy inference system model to extract features from the multi-spectral tongue image processing data, obtaining recognition results including tongue color, tongue shape, and tongue coating features, encoding the extracted recognition results, and generating feature vectors of the multi-spectral tongue image processing data; According to the multispectral tongue image processing data, the data is converted into a graph structure, wherein each pixel is regarded as a node, and the connection relationship between the nodes is determined by the spatial distance and color similarity between the pixels. Based on the converted graph structure, a graph neural network model is constructed, and the graph neural network model is trained to enable it to learn the relationship between the nodes and the feature representation of the nodes. The trained graph neural network model is used to capture the relationship between the pixels, and the relationship features of the multispectral tongue image processing data are obtained, including: According to the multispectral tongue image processing data, the data is converted into a graph structure, each pixel point is regarded as a node, a threshold value Q is preset, and when the spatial distance between two pixel points is less than the threshold value Q, it is considered that there is a connection between the two nodes; Build a graph neural network model and define node features, including the color value, brightness, and contrast of the pixel points. Build a graph neural network model based on the converted graph structure and define convolution operations in the model to aggregate the neighbor information of the node. and / or using superpixel segmentation technology to divide the tongue image into multiple superpixel regions, each superpixel being a node, and edges being established based on similarities and adjacency relationships between superpixels; Training a graph neural network model based on the historical labeled data in the multispectral tongue image processing data; during the training process, dynamically adjusting the graph structure according to the feedback of the model, the dynamic adjustment includes deleting or adding edges, and training to obtain a graph neural network model; Input the real-time data in the multispectral tongue image processing data into the trained graph neural network model, and obtain the feature representation of each node through the forward propagation of the graph neural network model. These feature representations capture the relationship between the pixel points, and obtain the multispectral tongue image processing data relationship features; According to the multi-spectral tongue image processing data feature vector and the multi-spectral tongue image processing data feature vector, multi-modal fusion is performed to obtain a comprehensive tongue image feature vector, including: According to the multi-spectral tongue image processing data feature vector and the importance of the multi-spectral tongue image processing data feature vector, different weights are assigned to the feature vectors of each modality, and then weighted summation is performed; and / or connecting the multi-spectral tongue image processing data feature vector and the feature vector of the multi-spectral tongue image processing data feature vector in series to form a longer feature vector; And / or use tensors to represent multimodal data, and learn the interaction between modes through tensor decomposition or tensor networks. Set the multimodal data as a tensor (\mathcal{T}), where the eigenvector of each mode constitutes a mode of the tensor. A third-order tensor (\mathcal{T}\in\mathbb{R}^{I\timesJ\timesK}) is decomposed into: \mathcal{T}=\mathcal{G}\times_1\mathbf{A}\times_2\mathbf{B}\times_3\mathbf{C}; Among them, (\mathcal{G}) is the core tensor that captures the interaction between modes; (\mathbf{A},\mathbf{B},\mathbf{C}) is the factor matrix associated with the eigenvector of each mode; And / or construct a deep learning model with multiple input branches, each branch corresponding to a feature vector of a modality, fuse the features of different modalities together through a fusion layer in the deep layer of the fusion network of the deep learning model, jointly train the fusion network, optimize the parameters of the fusion layer and the parameters of the feature extractors of each modality, adopt an end-to-end training method or pre-train the feature extractors of each modality separately, and then perform joint fine-tuning; Obtain comprehensive tongue image feature vector; The preset classifier is used to make classification decisions on the fused comprehensive tongue image feature vector to obtain the multi-spectral Chinese medicine tongue image information analysis results.

2. The multi-spectral TCM tongue image information analysis method according to claim 1 is characterized in that: The preset classifier is used to classify and decide the fused comprehensive tongue image feature vector to obtain the multi-spectral TCM tongue image information analysis result, including: Select one or more classifiers, including support vector machines, random forests, and neural networks, and optimize the classifiers according to the characteristics of multispectral tongue image data, including adjusting the kernel function or parameters of SVM and optimizing the architecture of the neural network; Use ensemble learning methods to improve classification accuracy by combining the prediction results of multiple classifiers, and design an ensemble strategy with dynamic weight adjustment to dynamically adjust the weight of each classifier in the final decision based on its performance on the validation set; Input the selected and dimension-reduced feature vectors into a preset classifier for classification decision; and / or design a multi-level classification strategy, first roughly classify the tongue image, then finely classify each category, and obtain the multi-spectral TCM tongue image information analysis results; Post-process the multi-spectral TCM tongue and face information analysis results, filter low-confidence multi-spectral TCM tongue and face information analysis results through confidence thresholds, or perform multiple rounds of classification to improve accuracy; And / or design a result interpretation module to provide TCM theoretical explanations and suggestions for each multi-spectral TCM tongue image information analysis result.

3. The multi-spectral TCM tongue image information analysis method according to claim 1 or 2, characterized in that: Also includes: Develop an interactive interface that allows users to intuitively view and interpret the results of multi-spectral TCM tongue image analysis; Provide a feedback mechanism so that users can correct or supplement the analysis results of multi-spectral TCM tongue and facial information, and continuously optimize the classifier and fusion strategy through user feedback to form a closed-loop learning system; And / or introduce a dynamic feedback mechanism so that the system can be continuously optimized according to the actual classification effect and expert feedback. When experts object to the analysis results of multi-spectral TCM tongue and facial information, the system automatically records these feedbacks and adjusts the model parameters or adds new training samples accordingly to improve future classification performance; And / or for the same patient, a tongue image history database is established to track the changes in tongue image over time, and by analyzing the long-term trend of tongue image changes, doctors can be provided with deeper insights into health conditions; And / or develop an interactive tongue image analysis tool that allows doctors to manually adjust the weights and thresholds of tongue image features to more intuitively explore the impact of different feature combinations on the analysis results of multispectral TCM tongue image information.

4. A multi-spectral Chinese medicine tongue image information analysis device, characterized in that: The multi-spectral TCM tongue image information analysis method according to any one of claims 1 to 3 is adopted, wherein the multi-spectral TCM tongue image information analysis device comprises: An acquisition module, used for acquiring multi-spectral tongue image data, and pre-processing the multi-spectral tongue image data to obtain multi-spectral tongue image processing data; A first processing module is used to construct a multi-parameter adaptive neural fuzzy inference system model based on the multi-spectral tongue image processing data, train and optimize the multi-parameter adaptive neural fuzzy inference system model to recognize tongue image features by adjusting the fuzzy rule base, and extract features from the multi-spectral tongue image processing data based on the multi-parameter adaptive neural fuzzy inference system model after training and optimization to obtain a feature vector of the multi-spectral tongue image processing data; The second processing module is used to convert the multispectral tongue image processing data into a graph structure, wherein each pixel point is used as a node, and the connection relationship between the nodes is determined by the spatial distance and color similarity between the pixels. Based on the converted graph structure, a graph neural network model is constructed, and the graph neural network model is trained to enable it to learn the relationship between the nodes and the feature representation of the nodes. The trained graph neural network model is used to capture the relationship between the pixels to obtain the relationship characteristics of the multispectral tongue image processing data. A setting module is used to perform multimodal fusion according to the multispectral tongue image processing data feature vector and the multispectral tongue image processing data feature vector to obtain a comprehensive tongue image feature vector; The allocation module is used to use the preset classifier to make classification decisions on the fused comprehensive tongue image feature vector to obtain the multi-spectral Chinese medicine tongue image information analysis results.

5. A multi-spectral Chinese medicine tongue image information analysis device, characterized in that: The multi-spectral TCM tongue image information analysis device comprises: a memory and at least one processor, wherein the memory stores instructions; The at least one processor calls the instructions in the memory to enable the multi-spectral Chinese medicine tongue and facial image information analysis device to execute the multi-spectral Chinese medicine tongue and facial image information analysis method as described in any one of claims 1-3.

6. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by the processor, the multi-spectral Chinese medicine tongue image information analysis method as described in any one of claims 1-3 is implemented.

Images