Two-stage graph convolution Mama network-based tongue picture identification method for deficiency symptom patient

Abstract

The invention provides a false symptom patient tongue picture recognition method based on a two-stage graph convolution Mama network, and the method comprises the steps: carrying out the superpixel segmentation of a tongue picture image through employing an SLIC algorithm, and dividing the tongue picture image into B sub-graphs; local aggregation and enhancement of spectral and spatial features in sub-graphs are carried out through a two-stage graph convolution layer and a DSFR module, and an enhanced node feature matrix is obtained; obtaining absolute position information of each sub-graph through an RAPE module, and fusing the absolute position information with the enhanced node feature matrix to obtain a feature embedding sequence of the sub-graphs; performing global spatial feature learning based on a Mama network by taking the feature embedding sequence of the sub-graph as input to obtain a model output feature graph; and performing category mapping on the tongue picture category through MLP to obtain a tongue picture recognition result. According to the method, the classification precision in a complex environment is improved, and the problems of high calculation complexity and information redundancy of a multi-branch model are avoided.

Description

Technical Field

[0001] The present invention relates to the technical field of image classification, and in particular to a method for identifying tongue images of patients with deficiency syndromes. Background Art

[0002] Hyperspectral image classification technology has important application value in the identification of tongue images in patients with deficiency syndromes in Traditional Chinese Medicine (TCM). It combines hyperspectral imaging technology with artificial intelligence models to provide a quantitative, objective, and efficient diagnostic tool for TCM tongue diagnosis. TCM tongue image is an important basis for diagnosing deficiency syndromes. The tongue image of patients with deficiency syndromes is usually characterized by a lighter tongue color, thin or no tongue coating, and a thin tongue. Traditional visual observation methods are difficult to standardize and quantify. Hyperspectral imaging technology can accurately extract subtle features of the tongue surface by capturing the spectral and spatial information of tongue images in the visible and near-infrared ranges, providing a scientific data basis for identifying tongue images of deficiency syndromes. In addition, by combining machine learning or deep learning algorithms, hyperspectral image classification can further explore key spectral features in tongue image data, significantly improving the accuracy and objectivity of deficiency syndrome identification.

[0003] In the task of tongue image recognition in people with deficiency syndromes, the color, shape, texture and other characteristics of the tongue image are closely related to different types of deficiency syndromes, and the tongue body is often interfered with by other parts of the oral cavity such as teeth, lips and tongue coating. These factors bring significant challenges to tongue image recognition. In addition, there are large differences in tongue image characteristics among different populations. For example, factors such as age, gender and health status may affect the appearance of the tongue, making the characteristics of tongue images in people with deficiency syndromes more complex and diverse. These background interferences and the complexity of tongue images result in low accuracy and robustness of traditional image recognition methods in tongue image recognition, making it difficult to meet the needs of actual applications.

[0004] Patent publication number CN118038192A discloses a few-shot graph convolutional tongue image classification method. This method primarily extracts features from tongue images using a local spatial attention module, dividing the image into regions such as the tongue tip, tongue surface, tongue root, left tongue margin, and right tongue margin. A composite CNN and Transformer network is then used to extract features from each region. While capable of extracting features from local regions, the method is relatively inadequate in modeling global spatial dependencies, relying primarily on the structure of a graph convolutional network to achieve global feature fusion.

[0005] Therefore, faced with complex background environments and diverse tongue image characteristics, more efficient deep learning models are needed to accurately identify and classify the tongue images of people with deficiency syndromes. These models not only need high-precision classification capabilities, but also strong robustness to cope with different interference factors and individual differences. In this context, the tongue image recognition method for people with deficiency syndromes based on deep learning and hyperspectral imaging technology has demonstrated superior performance. It can further improve recognition accuracy and model generalization capabilities by automatically learning the semantic information of the image. Especially in the case of small sample data and complex features, deep learning methods can provide more accurate classification results, thereby providing support for the design and application of TCM deficiency syndrome diagnosis systems. Summary of the Invention

[0006] In response to the technical problem that existing tongue image recognition models have poor recognition accuracy when faced with small sample data and complex features, the present invention proposes a tongue image recognition method for patients with deficiency syndrome based on a two-stage graph convolutional Mamba network, which improves the classification accuracy in complex environments, ensures the extraction of local and global features, and effectively avoids the high computational complexity and information redundancy problems of multi-branch models.

[0007] In order to achieve the above object, the technical solution of the present invention is achieved as follows:

[0008] A tongue image recognition method for deficiency syndrome patients based on a two-stage graph convolutional Mamba network, the steps are as follows:

[0009] S1: Collect tongue images of people with deficiency syndromes, screen the collected tongue images, and create a standardized tongue image dataset through image annotation and format conversion;

[0010] S2: Use the SLIC algorithm to perform superpixel segmentation on the tongue image and divide the tongue image into B sub-images;

[0011] S3: Locally aggregate and enhance the spectral and spatial features within the subgraph through a two-stage graph convolutional layer and DSFR module to obtain an enhanced node feature matrix;

[0012] S4: The absolute position information of each subgraph is obtained through the RAPE module and fused with the enhanced node feature matrix to obtain the feature embedding sequence of the subgraph. The feature embedding sequence of the subgraph is used as input to perform global spatial feature learning based on the Mamba network to obtain the output feature map of the Mamba network model.

[0013] S5: Based on the feature map output by the Mamba network model, the tongue image categories are mapped through MLP to obtain the tongue image recognition results.

[0014] Furthermore, the implementation method of step S2 is as follows: using the SLIC algorithm to perform superpixel segmentation on the standardized tongue image dataset, dividing the tongue image into K continuous superpixel regions; constructing each superpixel region into a subgraph G b =(V b ,E b ), where V b is a node set, E b is an edge set; the 8-neighborhood connection strategy is used to construct the subgraph G according to the neighborhood relationship of pixel space b The adjacency matrix A b ; In subgraph G b Initialize the spectral features of each node as vector x i , and combine the spectral features of all nodes into the node feature matrix of the subgraph |V b | indicates the number of nodes.

[0015] Furthermore, the method for locally aggregating and enhancing spectral and spatial features within a sub-graph through the two-stage graph convolution layer and DSFR module in step S3 is as follows: in each layer of graph convolution, in each sub-graph, graph convolution is used to locally aggregate spectral and spatial features:

[0016]

[0017] in, Represents the node feature matrix output by the b-th subgraph at layer l, the initial input is the adjacency matrix containing self-loops, I is the identity matrix, is the adjacency matrix The degree matrix of is the weight matrix of the l-1th layer, Sigmoid(·) is the Sigmoid activation function;

[0018] In each layer of DSFR module, the node feature matrix Perform downsampling subgraph feature reconstruction to obtain the DSFR module output node feature matrix After two stacked graph convolution layers and the calculation of the DSFR module, the enhanced node feature matrix representation is obtained

[0019] Furthermore, the node feature matrix The method for reconstructing downsampled sub-graph features is:

[0020] The score of each node in the subgraph is calculated through a set of trainable weight matrices W:

[0021]

[0022] Scores for each node b Perform normalization and calculate the scores based on the normalized scores gb Select the N key nodes with the highest scores:

[0023] N=min(num_nodes,|V b |)

[0024] Among them, num_nodes represents the maximum number of key nodes that you want to retain;

[0025] According to the selected N key nodes, the adjacency matrix A b Perform down-sampling sub-graph feature reconstruction:

[0026] A' b =subgraph(node_idx,A b )

[0027] Among them, A b 'To reconstruct the adjacency matrix, node_idx is the key node index after selection, and subgraph() represents the operation of constructing the subgraph adjacency matrix based on the new node index;

[0028] Reconstruct the adjacency matrix A b The key node features in 'are enhanced to obtain the node feature matrix of each layer of DSFR module

[0029] The pair reconstructs the adjacency matrix A b The method for enhancing the features of the key nodes in ' is:

[0030] Map the normalized scores of N key nodes through the Sigmoid function:

[0031] scores yb =Sigmoid(scores Agb )

[0032] Among them, scores yb For mapping scores, scores Agb To reconstruct the adjacency matrix A b 'The normalized scores of the N key nodes in;

[0033] Combine the mapping score with the reconstructed adjacency matrix A b 'The node features of the key nodes in are multiplied element by element to obtain the enhanced node feature matrix of each layer

[0034]

[0035] in, ⊙ represents element-by-element multiplication, Represents the reconstructed adjacency matrix A b 'Node feature matrix of the N key nodes in .

[0036] Furthermore, the method for obtaining the feature embedding sequence of the subgraph in step S4 is as follows: given a subgraph G b The coordinates of the center of gravity are (x i ,y i ), calculate the subgraph Gb and the center of the input tongue image (x c ,y c ) is the normalized Euclidean distance between:

[0037]

[0038] Where H and W are the height and width of the input tongue image respectively;

[0039] Introduce direction vector v i , used to describe the spatial distribution direction of the subgraph:

[0040] v i =(cosθ,sinθ)

[0041]

[0042] The absolute position information of each sub-graph is obtained by encoding the normalized Euclidean distance and direction vector:

[0043] P i =d i ·v i

[0044] In each subgraph, the absolute position information is combined with the enhanced node features of the subgraph Fusion is performed to obtain subgraph G b The feature embedding sequence of

[0045]

[0046] Where σ(·) represents the Relu activation function and ⊙ represents element-wise multiplication.

[0047] The feature embedding sequence of the final B subgraphs is represented as E=E1,E2,...E b ...,E B .

[0048] Furthermore, the method for global spatial feature learning based on the Mamba network is as follows: normalizing the feature embedding sequence E of B subgraphs, generating a feature sequence x and a feature sequence z through linear transformation, performing bidirectional separable convolution and bidirectional SSM on the feature sequence x in sequence to obtain multi-level spatial information, and fusing the multi-level spatial information with the feature sequence z to obtain the output feature map of the Mamba network model.

[0049] Furthermore, the method of performing bidirectional separable convolution and bidirectional SSM on the feature sequence x in sequence to obtain multi-level spatial information is as follows: performing bidirectional depth-wise separable convolution on the feature sequence x to capture global features:

[0050] x f =Forward DW Conv1d(x)

[0051] x b =Backward DW Conv1d(x)

[0052] Among them, x f is the feature after forward convolution, x b is the feature after reverse convolution, Forward DW Conv1d(·) is the forward convolution operation, and Backward DW Conv1d(·) is the reverse convolution operation;

[0053] Feature x after forward convolution f And the feature x after reverse convolution b Perform forward SSM and reverse SSM calculations on the input to obtain the forward SSM output feature sequence y f And the reverse SSM output feature sequence y b .

[0054] Furthermore, the forward SSM and reverse SSM are calculated as follows:

[0055] h t+1 =(I+AΔt)h t +Bu t

[0056] y t =Ch t +Du t

[0057] Where Δt represents the time step, I is the identity matrix, and h t is the hidden state, u t is the current value of the input feature sequence, y t is the current value of the output feature sequence, A, B, C, and D are learnable parameter matrices;

[0058] The method for fusing multi-level spatial information with the feature sequence z is:

[0059] After activating the feature sequence z, it is output with the forward SSM feature sequence y f And the reverse SSM output feature sequence y b Perform element-wise multiplication:

[0060] H f =y f ⊙σz

[0061] H b =y b ⊙σz

[0062] Among them, H f is the forward fusion feature, H b is the reverse fusion feature, ⊙ is the element multiplication, and σ· is the ReLU activation function; the comprehensive feature is obtained by addition fusion:

[0063] H=Linear(H f +H b )

[0064] Among them, Linear(·) is a linear operation;

[0065] The residual connection is used to add the fused comprehensive features to the activated feature sequence z to obtain the output feature map of the Mamba network model:

[0066] H out =H+σ(z).

[0067] Furthermore, the method of performing category mapping on tongue image categories through MLP to obtain tongue image recognition results is as follows:

[0068] Model output feature H out First, the first layer of MLP linear transformation is performed:

[0069] Z s =σ(H out W1+b1)

[0070] Among them, W1 and b1 are weight matrix and bias term respectively;

[0071] After further mapping through the second layer of MLP, the classification results are obtained:

[0072]

[0073] W2 and b2 are the weight matrix and bias term respectively.

[0074] The beneficial effects of the present invention are:

[0075] 1. The present invention improves the classification accuracy of the model in complex environments, reduces the network model's reliance on training samples to a certain extent, and helps to deal with various complex interference problems around the tongue, thereby improving the generalization ability and robustness of the network.

[0076] 2. The two-stage sequential structure ensures the extraction of local and global features, effectively avoiding the high computational complexity and information redundancy problems of the multi-branch model.

[0077] 3. In general, GCN effectively extracts local features of tongue images by aggregating spectral-spatial features within subgraphs. The Mamba network captures global contextual information and improves classification performance by modeling the spatial relationship between superpixels. In addition, the present invention designs a down-sampled subgraph feature reconstruction module (DSFR, Down sampled Subgraph Feature Reconstruction), which dynamically selects key nodes and reduces redundant information, thereby improving the expressive power of the model. At the same time, the introduced region-relation aware absolute position information embedding module (RAPE, Region-Relation Aware Absolute Positional Encoding) further enhances the model's understanding of spatial relationships, ensuring that the tongue image recognition task for deficiency syndrome populations can still maintain a high recognition accuracy under small sample data. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0079] Figure 1 Flow chart of the method of the present invention.

[0080] Figure 2 Flowchart of the local spectral feature aggregation module within a subgraph.

[0081] Figure 3 Flowchart of the global spatial feature learning module between subgraphs.

[0082] Figure 4 This is the DSFR structure diagram. DETAILED DESCRIPTION

[0083] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without creative work are within the scope of protection of the present invention.

[0084] A tongue image recognition method for deficiency syndrome patients based on two-stage graph convolutional Mamba network, Figure 1 As shown, the steps include:

[0085] S1: Collect tongue images of people with deficiency syndromes, screen the collected tongue images, and create a standardized tongue image dataset through image annotation and format conversion.

[0086] Tongue image data from patients was collected at a hospital and manually processed to create a dataset for network training. Tongue images of normal tongues and those of patients with deficiency syndromes were collected using a professional digital camera at a fixed time period and indoor lighting, with a fixed focal length and sensitivity. The collected tongue photos were manually screened to remove blurry and incomplete images, and uniform histogram equalization was performed on images with inconsistent brightness. The processed tongue image data was manually annotated and formatted, and the dataset was divided into training and validation sets at an 8:2 ratio. The tongue image samples in the dataset were resized to the same size as the network input (224×224) and used as input for the network.

[0087] S2: Use the SLIC algorithm to perform superpixel segmentation on the tongue image and divide the tongue image into B sub-images.

[0088] Specifically, the input tongue image is represented as Where H and W are the height and width of the image, respectively, and K is the number of spectral bands. The SLIC algorithm is used for superpixel segmentation, and the tongue image is divided into B continuous superpixel regions. Each superpixel region is represented by S b ; In this example, B is 200.

[0089] Furthermore, each superpixel region Sb is constructed as a subgraph G b =(V b ,E b ), where V b is a set of nodes, each node corresponds to a pixel in each superpixel, E b is an edge set, which represents the spatial adjacency relationship between nodes.

[0090] Furthermore, the 8-neighborhood connection strategy is used to construct the subgraph G according to the neighborhood relationship of pixel space. bThe adjacency matrix

[0091]

[0092] Among them, A b,ij is the adjacency matrix A b Elements in .

[0093] This adjacency structure ensures that GCN can efficiently propagate and aggregate information within each subgraph, thereby capturing local spectral and spatial features.

[0094] Furthermore, in each subgraph G b In the example, the spectral feature of each node is initialized as a vector And combine the spectral features of all nodes into the node feature matrix of the subgraph:

[0095]

[0096] Among them, X b For subgraph G b The node feature matrix of .

[0097] Node feature matrix X b As the input of graph convolution (GCN), it ensures that the model can effectively aggregate spectral and spatial features in a local range.

[0098] S3: Locally aggregate and enhance the spectral and spatial features within the subgraph through a two-stage graph convolutional layer (GCN) and a down-sampled subgraph feature reconstruction module (DSFR) to obtain an enhanced node feature matrix.

[0099] Specifically, such as Figure 2 As shown in Figure 2, in each layer of graph convolution, within each subgraph, graph convolution (GCN) is used to perform local aggregation of spectral and spatial features, and the calculation formula is:

[0100]

[0101] in, Represents the node feature matrix output by the b-th subgraph at layer l, the initial input is the adjacency matrix containing self-loops, I is the identity matrix, that is, the self-loop, is the adjacency matrix The degree matrix of is the weight matrix of the l-1 layer. In each layer of graph convolution, the features of the node are propagated and fused through its neighboring nodes to capture the spectral and spatial information within the superpixel area.

[0102] Within the superpixel region of the tongue image, interfering pixels may be mixed with the target object. The participation of all pixels in feature propagation leads to the accumulation of redundant information, increased computational complexity, and interference with the model's learning of key features. Therefore, relying solely on GCN feature aggregation cannot effectively address this redundant information challenge. To address this issue, the present invention designs a downsampled subgraph feature reconstruction module (DSFR) to dynamically select key nodes within the subgraph during feature aggregation.

[0103] Specifically, after graph convolution (GCN), the subgraph G b The midpoint feature matrix is Among them C l is the output feature dimension of the GCN layer l.

[0104] like Figure 4 As shown, in each layer of the down-sampling sub-graph feature reconstruction module (DSFR), first, a set of trainable weight matrices Calculate the score of each node in the subgraph:

[0105]

[0106] Among them, W is the weight matrix, which is used to learn the importance of each node in the feature aggregation process. Through continuous training, W adaptively adjusts the weight to accurately reflect the importance of the node.

[0107] Further, the scores of each node b Perform normalization:

[0108]

[0109] Where ε is a small constant to prevent the denominator from being zero, scores gb is the normalized score.

[0110] Further, according to the normalized scores gb Select the N key nodes with the highest scores:

[0111] N=min(num_nodes,|V b |)

[0112] Among them, num_nodes represents the maximum number of key nodes that we want to retain, |V b | is the subgraph G b The number of nodes in the.

[0113] These key nodes represent the key nodes in the subgraph, and redundant or interfering nodes are screened out to ensure that the model focuses on key features.

[0114] Furthermore, in order to maintain the integrity of the subgraph structure, the adjacency matrix A is constructed based on the selected N key nodes. b Perform down-sampling sub-graph feature reconstruction:

[0115] A' b =subgraph(node_idx,A b )

[0116] in, To reconstruct the adjacency matrix, n × n = N, node_idx is the selected key node index, and subgraph() represents the subgraph adjacency matrix operation based on the new node index, i.e., downsampling. This step ensures that the subgraph topology reflects the actual relationships between nodes while retaining important nodes.

[0117] Furthermore, the reconstructed adjacency matrix A b 'The features of the key nodes in are enhanced: the normalized scores are mapped through the Sigmoid function:

[0118] scores yb =Sigmoid(scores Agb )

[0119] Among them, scores yb For mapping scores, scores Agb To reconstruct the adjacency matrix A b 'The normalized scores of the N key nodes in .

[0120] Mapping scores yb and reconstruct the adjacency matrix A b 'The node features in are multiplied element by element to obtain the enhanced node feature matrix of each layer

[0121]

[0122] in, ⊙ represents element-by-element multiplication, Represents the reconstructed adjacency matrix A b 'Node feature matrix of the N key nodes in .

[0123] After the last layer of down-sampling subgraph feature reconstruction module (DSFR), the enhanced node feature matrix of the last layer is expressed as In this embodiment, GCN and DSFR have two layers in total, L=2.

[0124] The enhanced node feature matrix of the final layer is obtained Used for fusion with absolute position information embedding.

[0125] S4: The absolute position information of each subgraph is obtained through the RAPE module and fused with the enhanced node feature matrix to obtain the feature embedding sequence of the subgraph; the feature embedding sequence of the subgraph is used as input, and global spatial feature learning is performed based on the Mamba network to obtain the model output feature map.

[0126] Traditional position information encoding cannot fully consider the relative relationships of subgraphs in global space. To address this, this paper designs a regional relation-based absolute position embedding (RAPE) module. By embedding the absolute positions of subgraphs, it captures the spatial proximity and directionality between subgraphs, ensuring that the model can more accurately model the spatial dependencies between subgraphs.

[0127] Specifically, given a subgraph G b The coordinates of the center of gravity are (x i ,y i ), calculate the subgraph G b and the center of the input tongue image (x c ,y c ) is the normalized Euclidean distance between:

[0128]

[0129] Among them, the normalized Euclidean distance d i Reflects the relative position of the sub-image to the center of the image;

[0130] Furthermore, we introduce the direction vector v i , used to describe the spatial distribution direction of the subgraph:

[0131] v i =(cosθ,sinθ)

[0132]

[0133] Furthermore, by normalizing the Euclidean distance d i And direction vector are encoded to obtain the absolute position information of each sub-image:

[0134] P i =d i ·v i

[0135] Furthermore, in each subgraph, the absolute position information is fused with the enhanced node features of the subgraph to obtain the subgraph G b The feature embedding sequence E b : To enhance the model's spatial perception:

[0136]

[0137] Among them, E b Represents subgraph G b The feature embedding sequence of , σ() represents the Relu activation function.

[0138] After embedding the absolute position information, the Mamba network is then used to realize global spatial feature learning between sub-graphs to make up for the deficiency of graph convolution (GCN) that can only capture local features.

[0139] After embedding the absolute position information, the embedded feature sequence is passed through the Mamba network for further global feature learning. At this stage, the present invention combines bidirectional depthwise separable convolution with a state-space model to capture the complex dependencies in the feature sequence and achieve the fusion of multi-level spatial information.

[0140] Specifically, if Figure 3 As shown, first embed the sequence of B subgraphs Normalization is performed, where C2 is the output feature dimension. The feature embedding sequence E is normalized (Norm) and the feature sequence x and feature sequence z are generated through linear transformation:

[0141] x=Linear1(Norm(E))

[0142] z=Linear2(Norm(E))

[0143] The feature sequence x is used for subsequent bidirectional deep convolution and state-space model calculations, and the feature sequence z is used as the activation path for feature enhancement. Linear1(·) and Linear2(·) are linear operations used to generate two different feature sequences.

[0144] Furthermore, a bidirectional depth-wise separable convolution (Forward DWConv1d and Backward DWConv1d) is performed on the feature sequence x to capture global features:

[0145] x f =Forward DW Conv1d(x)

[0146] x b =Backward DW Conv1d(x)

[0147] Among them, x f is the feature after forward convolution, x bThe forward convolution operation is performed in the forward direction, and the backward convolution operation is performed in the backward direction. The bidirectional convolution design ensures that the model can capture both left-to-right and right-to-left feature patterns, effectively covering long-range dependencies in the feature sequence.

[0148] The State Space Model (SSM) is used to capture long-range dependencies between feature sequences. The SSM integrates global spatial information by simulating state transitions and the evolution of feature sequences. This module incorporates the dynamic representation of the state space, enabling the model to model complex spatial dependencies between subgraphs.

[0149] Furthermore, the feature x after forward convolution f And the feature x after reverse convolution b Perform forward SSM and reverse SSM on the input separately:

[0150] y f =Forward SSM(x f )

[0151] y b =Backward SSM(x b )

[0152] Among them, y f Output feature sequence and y for forward SSM b is the reverse SSM output feature sequence, Forward SSM(·) is the forward SSM calculation, and Backward SSM(·) is the reverse SSM calculation.

[0153] At this stage, the introduction of SSM can further strengthen the model's modeling of the temporal and spatial dependencies between sub-graph features, providing a more robust representation for subsequent feature fusion.

[0154] SSM models the input feature sequence of each layer as a process of state evolution over time. The dynamic changes of features can be described by the following state transition equation:

[0155] h t+1 =Ah t +Bu t

[0156] y t =Ch t +Du t

[0157] Among them, h t is the hidden state, indicating the state at time t, u tis the current value of the input feature sequence, y t is the current value of the output feature sequence, and A, B, C, and D are the learnable parameter matrices in the state-space model.

[0158] The state transition equations model the dependencies between hidden states, while the output equations connect hidden states to the output sequence. This design captures complex dependencies between subgraphs and enables efficient information propagation through recursive modeling. In image processing tasks, signals are typically discrete rather than continuous. To enable the model to process discrete feature sequences, the state space equations above need to be discretized.

[0159] After discretization, the state transfer equation can be expressed as:

[0160] h t+1 =(I+AΔt)h t +Bu t

[0161] y t =Ch t +Du t

[0162] Where Δt represents the time step and I is the identity matrix. This discretization form ensures the stability and accuracy of the model when processing sub-graph features.

[0163] Furthermore, in order to fully utilize the features output by the SSM module and the information carried by the feature sequence z in the activation path, the present invention designs a strategy of element-by-element multiplication and addition fusion.

[0164] After activating the feature sequence z, it is output with the forward SSM feature sequence y f And the reverse SSM output feature sequence y b Perform element-wise multiplication:

[0165] H f =y f ⊙σz

[0166] H b =y b ⊙σz

[0167] Among them, H f is the forward fusion feature, H b The above operation dynamically injects the information carried by the activated feature sequence z into the forward SSM output feature and the reverse SSM output feature, thereby enhancing the model's ability to identify important features and filtering out irrelevant noise information.

[0168] Furthermore, the comprehensive feature H is obtained by additive fusion:

[0169] H=Linear(H f +H b )

[0170] Among them, Linear(·) is a linear operation.

[0171] Furthermore, a residual connection is used to add the fused comprehensive features to the activated feature sequence z to ensure that the model retains key information during feature propagation:

[0172] H out =H+σ(z)

[0173] in, Output feature map of the model, d is the output feature dimension

[0174] S5: Based on the model output feature map, the tongue image categories are mapped through MLP to obtain the tongue image recognition results.

[0175] In the final stage of the model, the present invention adopts a two-layer MLP to classify superpixel features and maps the classification results to the pixels covered by each superpixel to achieve efficient and consistent prediction results.

[0176] Specifically, the model output feature H out First, the first layer of MLP linear transformation is performed:

[0177] Z s =σ(H out W1+b1)

[0178] in, and b1 are the weight matrix and bias term respectively, σ is the ReLU activation function, and h is the hidden layer dimension.

[0179] Furthermore, the classification results are obtained by further mapping through the second layer of MLP:

[0180]

[0181] in, and b2 are the weight matrix and bias term respectively, and c is the number of categories in the dataset.

[0182] For each pixel p i , if the super pixel it belongs to is s, then the prediction result of the pixel is The classification result of the superpixel Decide:

[0183]

[0184] This paper compares and verifies other existing models from the perspectives of accuracy, accuracy, and training time. The results are shown in Table 1:

[0185] Table 1

[0186] Model Training accuracy / % Verification accuracy / % Training time / h ResNet50 95.6 91.6 0.58 VGG16 98.9 90.7 1.47 AlexNet 99.2 90.4 0.55 GoogleNet 99.5 92.9 0.75 The present invention 99.9 94.7 0.49

[0187] Training accuracy: The training accuracy of the present invention reaches 99.9%, which is the highest compared with ResNet50 (95.6%), VGG16 (98.9%), AlexNet (99.2%), and GoogleNet (99.5%). This shows that the present invention has a better data fitting effect during the model training process and can learn data features more accurately.

[0188] Verification accuracy: The verification accuracy of the present invention is 94.7%, which is higher than ResNet50 (91.6%), VGG16 (90.7%), and AlexNet (90.4%), although slightly higher than GoogleNet (92.9%). This shows that the present invention can also maintain high accuracy on new data, the model has strong generalization ability, and can more reliably perform predictions and judgments in actual application scenarios.

[0189] In terms of training time, the proposed method takes only 0.49 hours to train, which is shorter than ResNet50 (0.58 hours), VGG16 (1.47 hours), GoogleNet (0.75 hours), and AlexNet (0.55 hours). This means that the proposed method takes less time to train, saving significant time and costs, improving R&D efficiency, and enabling faster deployment. It also reduces computing resource consumption and training costs.

[0190] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A tongue image recognition method for deficiency syndrome patients based on a two-stage graph convolutional Mamba network, characterized in that: The steps are: S1: Collect tongue images of people with deficiency syndromes, screen the collected tongue images, and create a standardized tongue image dataset through image annotation and format conversion; S2: Use the SLIC algorithm to perform superpixel segmentation on the tongue image and divide the tongue image into B sub-images; S3: Locally aggregate and enhance the spectral and spatial features within the subgraph through a two-stage graph convolutional layer and DSFR module to obtain an enhanced node feature matrix; S4: The absolute position information of each subgraph is obtained through the RAPE module and fused with the enhanced node feature matrix to obtain the feature embedding sequence of the subgraph. The feature embedding sequence of the subgraph is used as input to perform global spatial feature learning based on the Mamba network to obtain the output feature map of the Mamba network model. S5: Based on the feature map output by the Mamba network model, the tongue image categories are mapped through MLP to obtain the tongue image recognition results.

2. The tongue image recognition method for deficiency syndrome patients based on the two-stage graph convolutional Mamba network according to claim 1 is characterized in that: The implementation method of step S2 is as follows: use the SLIC algorithm to perform superpixel segmentation on the standardized tongue image dataset, divide the tongue image into K continuous superpixel regions; construct each superpixel region into a subgraph G b =(V b ,E b ), where V b is a node set, E b is an edge set; the 8-neighborhood connection strategy is used to construct the subgraph G according to the neighborhood relationship of pixel space b The adjacency matrix A b ; In subgraph G b Initialize the spectral features of each node as vector x i , and combine the spectral features of all nodes into the node feature matrix of the subgraph |V b | indicates the number of nodes.

3. The tongue image recognition method for deficiency syndrome patients based on the two-stage graph convolutional Mamba network according to claim 2 is characterized in that: The method described in step S3 for local aggregation and enhancement of spectral and spatial features within a sub-graph through a two-stage graph convolution layer and a DSFR module is as follows: in each layer of graph convolution, within each sub-graph, graph convolution is used to perform local aggregation of spectral and spatial features: in, Represents the node feature matrix output by the b-th subgraph at layer l, the initial input is the adjacency matrix containing self-loops, I is the identity matrix, is the adjacency matrix The degree matrix of is the weight matrix of the l-1th layer, Sigmoid(·) is the Sigmoid activation function; In each layer of DSFR module, the node feature matrix Perform downsampling subgraph feature reconstruction to obtain the DSFR module output node feature matrix After two stacked graph convolution layers and the calculation of the DSFR module, the enhanced node feature matrix representation is obtained 4. The tongue image recognition method for deficiency syndrome patients based on the two-stage graph convolutional Mamba network according to claim 3 is characterized in that: The node feature matrix The method for reconstructing downsampled sub-graph features is: The score of each node in the subgraph is calculated through a set of trainable weight matrices W: Scores for each node b Perform normalization and calculate the scores based on the normalized scores gb Select the N key nodes with the highest scores: N=min(number_of_nodes,|V b |) Among them, num_nodes represents the maximum number of key nodes that you want to retain; According to the selected N key nodes, the adjacency matrix A b Perform down-sampling sub-graph feature reconstruction: A' b =subgraph(node_idx,A b ) Among them, A b 'To reconstruct the adjacency matrix, node_idx is the key node index after selection, and subgraph() represents the operation of constructing the subgraph adjacency matrix based on the new node index; Reconstruct the adjacency matrix A b The key node features in 'are enhanced to obtain the node feature matrix of each layer of DSFR module 5. The tongue image recognition method for deficiency syndrome patients based on two-stage graph convolutional Mamba network according to claim 4 is characterized in that: The pair reconstructs the adjacency matrix A b The method for enhancing the features of the key nodes in ' is: Map the normalized scores of N key nodes through the Sigmoid function: scores yb =Sigmoid(scores Agb ) Among them, scores yb For mapping scores, scores Agb To reconstruct the adjacency matrix A b 'The normalized scores of the N key nodes in; Combine the mapping score with the reconstructed adjacency matrix A b The node features of the key nodes in 'are multiplied element by element to obtain the enhanced node feature matrix of each layer in, ⊙ represents element-by-element multiplication, Represents the reconstructed adjacency matrix A b 'Node feature matrix of the N key nodes in .

6. The tongue image recognition method for deficiency syndrome patients based on a two-stage graph convolutional Mamba network according to any one of claims 3 to 5, characterized in that: The method for obtaining the feature embedding sequence of the subgraph in step S4 is: given a subgraph G b The coordinates of the center of gravity are (x i ,y i ), calculate the subgraph G b and the center of the input tongue image (x c ,y c ) is the normalized Euclidean distance between: Where H and W are the height and width of the input tongue image respectively; Introduce direction vector v i , used to describe the spatial distribution direction of the subgraph: v i =(cosθ,sinθ) The absolute position information of each sub-graph is obtained by encoding the normalized Euclidean distance and direction vector: P i =d i ·v i In each subgraph, the absolute position information is combined with the enhanced node features of the subgraph Fusion is performed to obtain subgraph G b The feature embedding sequence of: Among them, σ() represents the Relu activation function, and ⊙ represents element multiplication; The feature embedding sequence of the final B subgraphs is represented as E=E1,E2,...E b ...,E B .

7. The tongue image recognition method for deficiency syndrome patients based on a two-stage graph convolutional Mamba network according to any one of claims 2 to 5, characterized in that: The method for global spatial feature learning based on the Mamba network is as follows: normalizing the feature embedding sequence E of B subgraphs, generating feature sequences x and z through linear transformation, performing bidirectional separable convolution and bidirectional SSM on the feature sequence x in sequence to obtain multi-level spatial information, and fusing the multi-level spatial information with the feature sequence z to obtain the output feature map of the Mamba network model.

8. The tongue image recognition method for deficiency syndrome patients based on two-stage graph convolutional Mamba network according to claim 7, characterized in that: The method of performing bidirectional separable convolution and bidirectional SSM on the feature sequence x in sequence to obtain multi-level spatial information is as follows: Perform bidirectional depth-wise separable convolution on the feature sequence x to capture global features: x f =Forward DW Conv1d(x) x b =Backward DW Conv1d(x) Among them, x f is the feature after forward convolution, x b is the feature after reverse convolution, Forward DW Conv1d(·) is the forward convolution operation, and Backward DW Conv1d(·) is the reverse convolution operation; Feature x after forward convolution f And the feature x after reverse convolution b Perform forward SSM and reverse SSM calculations on the input to obtain the forward SSM output feature sequence y f And the reverse SSM output feature sequence y b .

9. The tongue image recognition method for deficiency syndrome patients based on the two-stage graph convolutional Mamba network according to claim 8, characterized in that In, The forward SSM and reverse SSM are calculated as follows: h t+1 =(I+AΔt)h t +Bu t y t =Ch t +You t Where Δt represents the time step, I is the identity matrix, and h t is the hidden state, u t is the current value of the input feature sequence, y t is the current value of the output feature sequence, A, B, C, and D are learnable parameter matrices; The method for fusing multi-level spatial information with the feature sequence z is: After activating the feature sequence z, it is output with the forward SSM feature sequence y f And the reverse SSM output feature sequence y b Perform element-wise multiplication: H f =y f ⊙z H b =y b ⊙z Among them, H f is the forward fusion feature, H b is the reverse fusion feature, ⊙ is the element multiplication, and σ· is the ReLU activation function; the comprehensive feature is obtained by addition fusion: H=Linear(H f +H b ) Among them, Linear(·) is a linear operation; The residual connection is used to add the fused comprehensive features to the activated feature sequence z to obtain the output feature map of the Mamba network model: H out =H+σ(z)。 10. The tongue image recognition method for deficiency syndrome patients based on two-stage graph convolutional Mamba network according to claim 9, characterized in that: The method of performing category mapping on tongue image categories by MLP and obtaining tongue image recognition results is as follows: the model output feature H out First, the first layer of MLP linear transformation is performed: WITH s =σ(H out W1+b1) Among them, W1 and b1 are weight matrix and bias term respectively; After further mapping through the second layer of MLP, the classification results are obtained: W2 and b2 are the weight matrix and bias term respectively.

Images