Tongue image segmentation method based on double-space pyramid deep convolutional network

Abstract

The application discloses a tongue image segmentation method based on a double-space pyramid deep convolutional network, and comprises the following steps: step one, capturing a tongue image by using a camera and a mobile phone to improve the generalization of a data set; step two, performing background removal processing on the collected image to obtain a reference image without a background; and step three, constructing a deep convolutional network Deep-DSP based on a double-space pyramid. The double-space pyramid deep convolutional network Deep-DSP is constructed, so that the accuracy of tongue image segmentation is effectively improved. The design of the network structure allows better learning of features in a multiple coding state, so that the tongue area can be more accurately segmented, and the accuracy of traditional Chinese medicine tongue diagnosis is improved. The combination of the camera and the mobile phone for image acquisition, the background removal processing and the manual image cropping processing effectively enhance the generalization of the data set. This makes the model more adaptable to different illumination conditions, tongue abnormalities and shooting devices, and improves the universality of the model for different tongue images.

Description

Technical Field

[0001] This invention belongs to the field of image segmentation technology, specifically relating to a tongue image segmentation method based on a dual spatial pyramid depthwise convolutional network. Background Technology

[0002] In traditional Chinese medicine diagnosis, experienced doctors can glean insights into a patient's underlying symptoms by examining images of their tongue. With the continuous development of computer science and artificial intelligence, neural networks are increasingly being applied to image processing and feature extraction. In image processing, there are two types of segmentation: graph-based segmentation and clustering-based segmentation.

[0003] For graph-based image segmentation, graph theory maps the graph to a weighted indirect graph, treating pixels as nodes and viewing the image segmentation problem as the partitioning of a graph's vertices. Therefore, the optimal segmentation is obtained according to the minimum cut criterion. The accuracy of image segmentation has been improved by modifying the cut criterion. Representative methods include normalized cut algorithms, graph cut algorithms, and grasping cut algorithms. However, graph-based algorithms still require user interaction when handling complex texture features in images.

[0004] Clustering algorithms are implemented by iterating through pixels and calculating pixel similarity in color, brightness, and texture. Other features are then clustered into the same superpixels to obtain the final image segmentation result. Compared to the original K-means algorithm, spectral clustering incorporates a graph into the clustering process, resulting in a more accurate optimal solution. Mean-shift clustering segments pixels with the same modulus into the same region. Another clustering method, Simple Linear Iterative Clustering (SLIC), converts the color image into a 5-dimensional feature vector composed of Lab color space and XY color coordinates. The Lab color space includes the brightness channel L, color channel a, and color channel b. While clustering-based methods achieve good segmentation results, their time and space complexity is too high for practical applications.

[0005] With the development of computers and artificial intelligence, deep learning has been increasingly applied to image segmentation, bringing many breakthroughs to the field. Long et al. proposed the Fully Convolutional Network (FCN), a pioneer in using deep learning to solve image segmentation problems. FCN includes a forward inference process and a backward learning process, transforming coarse segmentation results into denser ones. Similar to FCN, a network called SegNet also combines an encoder and a decoder. Subsequently, Olaf et al. proposed U-Net, which is currently popular in segmentation tasks across all domains. U-Net features a symmetric downsampling architecture, upsampling and connecting the encoded image, and decoding, combining semantic and positional information. Zhao et al. proposed the Pyramid Scene Parsing Network (PSPNet), which uses a Residual Network (ResNet) as a feature extractor and an extended network to extract different patterns from the input image. A key feature of this network is the application of a pyramid pooling module for multi-scale convolution. Chen et al. focused on dilated convolutions, proposing the DeepLab series of deep convolutional networks for image segmentation and continuously improving the DeepLab network architecture, achieving good segmentation results. In recent years, researchers have tended to introduce Generative Adversarial Networks (GANs) into image segmentation, obtaining good results. However, to improve image segmentation performance, previous network structures have become increasingly complex, significantly increasing the time and space complexity of network training. Furthermore, current image segmentation applications mainly focus on target images with clear texture information; segmenting target regions with unclear color and texture features remains a challenging task.

[0006] Therefore, a tongue image segmentation method based on a dual-space pyramid deep convolutional network is proposed. By constructing an intelligent tongue diagnosis system that combines traditional Chinese medicine with neural networks, the diagnostic accuracy is automatically improved. Summary of the Invention

[0007] The purpose of this invention is to provide a tongue image segmentation method based on a dual-space pyramid depth convolutional network to solve the problems mentioned in the background art.

[0008] To achieve the above objectives, the present invention provides the following technical solution: a tongue image segmentation method based on a dual-space pyramid depthwise convolutional network.

[0009] Step 1: Image Acquisition: To improve the generalization of the dataset, images of the tongue were captured using a camera and a mobile phone. Standard tongue images were captured by the camera, and random tongue images were captured by the mobile phone. The images were scaled to the same pixel size. A total of 500 images of the front of the tongue and 56 images of the back of the tongue were collected as the training set.

[0010] Step 2: Image Processing: Remove the background from the acquired image to obtain a background-free reference image; this reference image is then used in the training of the neural network, along with the original image.

[0011] Step 3: Constructing a Deep-DSP based on dual spatial pyramids: The Deep-DSP deep convolutional network based on dual spatial pyramids includes two encoders, two spatial pyramid blocks, and two decoders. After the image is input into encoder 1 and reduced to 1 / 4 of its original size, it is processed by an intermediate stream and then simultaneously input into spatial pyramid block 1 and encoder 2.

[0012] The image input to encoder 2 is reduced to 1 / 4 of its input size, and then processed by spatial pyramid block 2 before being input to decoder 1. Decoder 1 enlarges the input image by 4 times and merges it with the output image of spatial pyramid block 1 through a merging module before inputting it into decoder 2. The output image of decoder 2 is then merged with the input image that has not been processed by encoder 1 through a second merging module before being input to the final output layer to obtain the binary image output by the network. Finally, the output image is obtained by dot product with the original image.

[0013] Step 4: Model Training: Use the 430 anterior tongue images and 50 dorsum tongue images with different features from Step 1 as the training set; use the remaining 70 anterior tongue images and 6 dorsum tongue images as the test set; train the Deep-DSP constructed in Step 3; adjust the image size to 320×320, and use an adaptive optimizer to train the proposed model, setting the loss function between the output image and the reference image to L1 loss; initialize the learning rate to 1e-3, decrease it by 1e-6 every 100 iterations, and extract the results after 100 cycles;

[0014] Step 5: Input any tongue image into the trained Deep-DSP model, and output a tongue image with the background removed.

[0015] Preferably, the collected images cover a variety of tongue abnormalities, including coatings and cracks.

[0016] Preferably, in step two, the image is manually cropped using Photoshop to ensure the accuracy of the tongue region; multiple operations are performed on the tongue image to add or crop the selected tongue region; the pixel values ​​of the selected tongue region and redundant background region are retained to [0,0,0], and the obtained image is regarded as the corresponding reference image.

[0017] Preferably, encoder 1 includes a 3×3 conventional convolutional layer, a batch norm layer, and a modified linear unit layer.

[0018] T j+1 =Sr j (T j )=r(b( 1 c 3×3 (T j ,Ke j )));where Sr j This represents the j-th stage encompassing a regular convolutional layer, r represents a modified linear unit, and b represents a batch norm layer. 1 c 3×3 Represents a 3×3 conventional convolutional layer, K ej Represents the convolution kernel, T j T represents the input image at stage j. j+1 This represents the input image of stage j+1, which is also the output image of stage j.

[0019] Preferably, encoder 2 includes a 3×3 convolutional layer, which has a function similar to pooling layer and has training capability, including BN layer and ReLU layer. Compared with encoder 1, encoder 2 needs to process images with a length and width that are 1 / 4 of the images processed by encoder 1.

[0020] Sp j This represents a convolutional layer with a stride of 2 (represented as...). 1 c 2 3×3 The j-th stage of ) for I'∈R m×n×c ,I∈R 2m×2n×c If I'= 1 c 2 3×3 (I,Ke j ),(Ke j ∈R 3×3×c×c The corresponding elements (x, y, z) located at I' (x∈1:m, y∈1:n, z∈1:c) satisfy:

[0021] Each encoder consists of one backbone layer and two auxiliary layers. The backbone layer comprises two parts, each consisting of three regular convolutional modules. The two auxiliary layers are connected to the first and second halves of the backbone layer, respectively. Because the auxiliary layers have shallower convolutions, they focus more on the original information than the backbone layer. By increasing the connections in the auxiliary layers, the influence of the original information on the encoding is enhanced. Encoder T t =Cat(Sp2·Sr1·Sr0·T) 0 ,Sp0·T 0); where Cat represents the merging module, the first input of Cat represents the main layer, the second input represents the auxiliary layer, and T 0 T represents the input image. t This represents the output image.

[0022] Preferably, the intermediate stream includes a 3×3 convolutional layer, a batch norm layer, and a ReLU layer;

[0023] T t′ =Sm2·Sm1·Sm0·T 0′ ;

[0024] Where Sm j’ The intermediate flow stage is represented by T, calculated as follows: j′+1 =Sm j′ (T j′ )=r(b( 3 c 3×3 (T j′ ,Ke j′ )));

[0025] Preferably, the spatial pyramid block includes a 1×1 convolutional part, three 3×3 non-convolutional parts, and a pooling part;

[0026] The 1×1 convolutional part includes 1×1 convolutional layers, batch norm (BN) layers, and integer positive linear activation function (ReLU) layers;

[0027] Convolution output image T out1 =r(b(c 1×1 (T in ,M1)));where T in ∈R m×n×c This represents the input encoded image, where m, n, and c are the image height, width, and number of channels, respectively, and T. out1 ∈R m×n×c This represents the output image from the first part; c 1×1 Let M1 ∈ R be a 1×1 convolutional layer. c×c The matrix representing the kernel parameters; for I, I'∈R m×n×c If I' = c 1×1 (I), the corresponding channel transpose vectors I(x,y), I'(x,y)∈R 1×c (x∈1:m,y∈1:n) satisfies: I′(x,y)=I(x,y)M1; r represents the ReLU layer; if I'=r(I), the corresponding elements (x,y,z) (x∈1:m,y∈1:n,z∈1:c) from I' and I satisfy: b represents the BN layer; if I' = b(I),

[0028] The corresponding elements (x, y, z) from I' and I (x∩1:m, y∈1:n, z∈1:c) satisfy: Where mean(I) z D(I) represents the average value of the z-channel pixels in image I. z ) represents the variance of the z-channel pixels in image I, and ε is an additional value in the denominator to maintain numerical stability, set to ε = 1 × 10⁻⁶. -5 ;

[0029] The 3×3 regular convolutional part consists of a 3×3 regular convolutional layer, a BN layer, and a ReLU layer;

[0030] Regular convolution output image T outi =r(b( d c 3×3 (T in ,K i )));where T outi ∈R m×n×c This represents the output image of the i-th part, where i = 3, 4, or 5. d c 3×3 This represents a 3×3 dilated convolutional layer with a dilation rate of d. To ensure that the processed image is not distorted, the padding amount is equal to the dilation rate; K i ∈R 3×3×c×c This represents the kernel parameter matrix from the i-th part; if I' = d c 3×3 (I,K i The channel map I'(:,:,z2)(z2∈1:c) of the corresponding output image I' satisfies: Ki Z1Z2 Represents Ki(:,:,z1,z2), I Z1 Represents I(:,:,z1),I' Z2 Represents I(:,:,z2), * represents passing through convolution kernel Ki Z1Z2 For Figure I Z1 convolution, I' Z2 The corresponding elements in the middle satisfy: Where x∈1:m,y∈1:n,tr() represents the trace function on the two-dimensional matrix, I T Z1 (x,y) ±d Indicates from I T Z1 The extracted 3×3 matrix centered at (x, y) is as follows:

[0031]

[0032]

[0033] The pooling part includes a MaxPool layer, a 3×3 convolutional layer, a BN layer, a ReLU layer, and an upper layer;

[0034] Pooling output image T out5 =Us(r(b( 1 c 3×3 (Mp(T in ),K5))));

[0035] in 1 c 3×3 This represents a 3×3 regular convolutional layer with a stride of 1. K5 is the kernel parameter matrix for this part, and Mp represents a max pooling layer; for I'∈R m×n×c ,I∈R 2m×2n×c If I' = Mp(I), the elements (x, y, z) located in I' (x∈1:m, y∈1:n, z∈1:c) satisfy: I′(x, y, z) = max(I(2x-1, 2y-1, z), I(2x-1, 2y, z), I(2x, 2y-1, z), I(2x, 2y, z)); Us represents a bilinear upsampling layer; for I'∈R 2m×2n×c ,I∈R m×n×c If I' = Us(I), the elements (x, y, z) located at I and corresponding to I' (x∈1:m, y∈1:n, z∈1:c) satisfy:

[0036] I′(2x,2y,z)=I(x,y,z)

[0037]

[0038]

[0039]

[0040] To accommodate the size of the tongue image, the convolution dilation rate for gas expansion was adjusted to 3, 6, and 9.

[0041] Preferably, multi-scale semantic segmentation is achieved through a 1×1 convolution part, three 3×3 convolution parts and a pooling part. The outputs of the above five parts are fully fused by the merging module and then processed by a 3×3 convolution layer, a BN layer and a ReLU layer in sequence.

[0042] Final output image T out =r(b( 1 c 3×3 (Cat(T out1 ,T out2 ,T out3 ,T out4 ,T out5 ),K out )));

[0043] Where Cat represents the merging module, for T out1 ,T out2 ,T out3 ,T out4 ,T out5 ∈R m×n×c The images are the outputs of a 1×1 convolution, three 3×3 convolutions, and a pooling part, respectively. The merged image belongs to R. m×n×5c And the kernel parameter matrix K out ∈R 3×3×5c×c This means that the image output from the convolutional layer contains c channels;

[0044] The image processed by the spatial pyramid is finally sent to decoder 1 and decoder 2;

[0045] The decoding process is implemented by bilinear upsampling layers of decoder 1 and decoder 2. Before each decoder's upsampling layer, there are regular convolutional layers for information fusion. Each decoder T... d =Us 2 (Sr(T e ));

[0046] Where T e T represents the input image. d Sr represents the decoded image, Sr represents the regular convolutional layer stage, and Us represents the decoded image. 2 Us(Us(.)) represents a simple 4-order bilinear upsampling process. Us represents the bilinear upsampling layer; for I'∈R 2m×2n×c ,I∈R m×n×c If I' = Us(I), the elements (x, y, z) located in I and corresponding to I' (x∈1:m, y∈1:n, z∈1:c) satisfy:

[0047] I′(2x,2y,z)=I(x,y,z)

[0048]

[0049]

[0050]

[0051] Compared with the prior art, the beneficial effects of the present invention are as follows: The present invention improves accuracy: By constructing a deep convolutional network based on dual spatial pyramids (Deep-DSP), the scheme effectively improves the accuracy of tongue image segmentation; the design of the network structure allows for better learning of features under multiple encoding states, thereby more accurately segmenting the tongue region and helping to improve the accuracy of TCM tongue diagnosis.

[0052] Enhanced generalization: By combining image acquisition with a camera and mobile phone, and employing image processing steps such as background removal and manual cropping, this approach effectively enhances the generalization of the dataset. This makes the model more adaptable to different lighting conditions, tongue anomalies, and shooting devices, improving the model's versatility across various tongue images.

[0053] Efficient Training and Inference: Training strategies such as Adaptive Optimizer (ADAM) and L1 loss function are employed to make the model training process more efficient. Furthermore, comparative experiments with other models (U-Net, PSPNet, DeepEar) demonstrate that Deep-DSP performs excellently in segmentation tasks, exhibiting lower time and space complexity, making the model more feasible for practical applications.

[0054] Smooth segmentation boundaries: Compared to other models, Deep-DSP achieves the smoothest segmentation boundaries and the most complete tongue image. This helps reduce noise and errors in the segmentation results, improving the overall quality of the segmented image.

[0055] Superior Objective Performance: Through quantitative analysis using objective metrics (accuracy, precision, recall, F1 score, and specificity), Deep-DSP achieved the best performance in experiments, most closely resembling the reference image. This demonstrates the outstanding performance of this scheme in image segmentation tasks.

[0056] In summary, the technical solution of the present invention has achieved significant beneficial effects in terms of improving accuracy, generalization, training efficiency, smoothness, and objectivity, providing a more reliable and advanced image segmentation tool for TCM tongue diagnosis. Attached Figure Description

[0057] Figure 1 This is a global network architecture diagram of the encoder-decoder architecture and spatial pyramid block combination of the present invention. Detailed Implementation

[0058] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0059] Please see Figure 1 This invention provides a technical solution: a tongue image segmentation method based on a dual-space pyramid depthwise convolutional network, step one: image acquisition:

[0060] To improve the generalization of the dataset: under the same lighting conditions, tongue images were captured by a camera with the lens about 8cm directly above the mouth, and the images were scaled to the same pixel size to collect standard tongue images; at the same time, tongue images were randomly captured by a mobile phone with the lens parallel to the face to ensure that the oral cavity area occupies one-quarter of the image.

[0061] The acquired images cover a variety of tongue abnormalities, including tongues with coatings, cracks, etc.

[0062] In this embodiment, 500 images of the front of the tongue and 56 images of the back of the tongue were collected (training set).

[0063] Step 2: Image Processing: Remove the background from the image acquired in Step 1 to obtain a corresponding background-free reference image, which is then used in the training of the neural network along with the original image.

[0064] In this embodiment, the tongue region in the image was obtained by manually cropping the image using Photoshop. Due to differences between different tongue images, such as background lighting, tongue smears, papules, and fissures, the tolerance value of the selected tongue region needs to be adjusted to ensure the entire tongue region is covered. If necessary, the approximate area needs to be manually outlined in the tongue image using the magic wand tool, and this process is repeated multiple times to add or crop the selected tongue region. The guide channel values ​​of the pixels in the selected tongue region and redundant background region are kept to [0,0,0], and then the obtained image is used as the corresponding reference image.

[0065] Step 3: Construct a deep convolutional network based on a dual spatial pyramid (Deep-DSP):

[0066] The Deep-DSP, a deep convolutional network with dual spatial pyramids, consists of two encoders, two spatial pyramid blocks, and two decoders. After the image is input into encoder 1 and reduced to 1 / 4 of its original size, it is processed by an intermediate stream and then simultaneously input into spatial pyramid block 1 and encoder 2.

[0067] The image input to encoder 2 is reduced to 1 / 4 of its input size, and then processed by spatial pyramid block 2 before being input to decoder 1. Decoder 1 enlarges the input image by 4 times and merges it with the output image of spatial pyramid block 1 through a merging module before inputting it into decoder 2.

[0068] The output image of decoder 2 is fused with the input image that has not been processed by encoder 1 through the second merging module and then input to the final output layer to obtain the binary image output by the network. Finally, the output image is obtained by the dot product of the original image and the output image.

[0069] Encoder 1 includes 3×3 regular convolutional layers, batch norm layers (BN), and rectified linear unit (ReLU) layers.

[0070] T j+1 =Sr j (T j )=r(b( 1 c 3×3 (T j ,Ke j )));where Sr j This represents the j-th stage encompassing a regular convolutional layer, r represents a modified linear unit, and b represents a batch norm layer. 1 c 3×3 Represents a 3×3 conventional convolutional layer, K ej Represents the convolution kernel, T j T represents the input image at stage j. j+1 This represents the input image of stage j+1, which is also the output image of stage j.

[0071] Encoder 2 includes 3×3 convolutional layers, which have similar functions to pooling layers and are trainable. It includes BN layers and ReLU layers. Compared with encoder 1, encoder 2 needs to process images with a length and width that are 1 / 4 the size of the images processed by encoder 1.

[0072] Sp j This represents a convolutional layer with a stride of 2 (represented as...). 1 c 2 3×3 The j-th stage of ) for I'∈R m×n×c ,I∈R 2m×2n×c If I'= 1 c 2 3×3 (I,Ke j ),(Ke j ∈R 3×3×c×c The corresponding elements (x, y, z) located at I' (x∈1:m, y∈1:n, z∩1:c) satisfy:

[0073] Each encoder consists of one backbone layer and two auxiliary layers. The backbone layer comprises two parts, each consisting of three regular convolutional modules. The two auxiliary layers are connected to the first and second halves of the backbone layer, respectively. Because the auxiliary layers have shallower convolutions, they focus more on the original information than the backbone layer. By increasing the connections in the auxiliary layers, the influence of the original information on the encoding is enhanced. Encoder T t =Cat(Sp2·Sr1·Sr0·T) 0 ,Sp0·T 0 ); where Cat represents the merging module, the first input of Cat represents the main layer, the second input represents the auxiliary layer, and T 0 T represents the input image. tThis represents the output image.

[0074] The intermediate flow consists of 3×3 convolutional layers, batch norm layers, and ReLU layers;

[0075] T t′ =Sm2·Sm1·Sm0·T 0′ ;

[0076] Where Sm j ' represents the intermediate flow stage, calculated as: T j′+1 =Sm j′ (T j′ )=r(b( 3 c 3×3 (T j′ ,Ke j′ )));

[0077] The spatial pyramid block consists of a 1×1 convolutional part, three 3×3 non-convolutional parts, and a pooling part.

[0078] The 1×1 convolutional part includes 1×1 convolutional layers, batch norm (BN) layers, and integer positive linear activation function (ReLU) layers.

[0079] Convolution output image T out1 =r(b(c 1×1 (T in ,M1)));where T in ∈R m×n×c (m, n, and c represent the height, width, and number of channels of the image, respectively) represents the input encoded image, T out1 ∈R m×n×c This represents the output image from the first part. 1×1 Let M1 ∈ R be a 1×1 convolutional layer. c×c The matrix representing the kernel parameters; for I, I'∈R m×n×c If I' = c 1×1 (I), the corresponding channel transpose vectors I(x,y), I'(x,y)∈R 1×c (x∈1:m,y∈1:n) satisfy: I′(x,y)=I(x,y)M1; r represents the ReLU layer. If I'=r(I), the corresponding elements (x,y,z) (x∈1:m,y∈1:n,z∈1:c) from I' and I satisfy: b represents the BN layer. If I' = b(I), the corresponding elements (x, y, z) from I' and I (x∈1:m, y∈1:n, z∈1:c) satisfy: Where mean(I) z D(I) represents the average value of the z-channel pixels in image I. z) represents the variance of the z-channel pixels in image I, and ε is an additional value in the denominator to maintain numerical stability, set to ε = 1 × 10⁻⁶. -5 ;

[0080] Similar to the 1×1 convolutional part, the 3×3 regular convolutional part consists of a 3×3 regular convolutional layer, a BN layer, and a ReLU layer;

[0081] Regular convolution output image T outi =r(b( d c 3×3 (T in ,K i )));where T outi ∈R m×n×c This represents the output image of the i-th part, where i = 3, 4, or 5. d c 3×3 This represents a 3×3 dilated convolutional layer with a dilation rate of d. To ensure that the processed image is not distorted, the padding amount is equal to the dilation rate. K i ∈R 3×3×c×c Let I' represent the kernel parameter matrix from the i-th part. d c 3×3 (I,K i The channel map I'(:,:,z2)(z2∈1:c) of the corresponding output image I' satisfies: Ki Z1Z2 Represents Ki(:,:,z1,z2), I Z1 Represents I(:,:,z1),I' Z2 Represents I(:,:,z2), * represents passing through convolution kernel Ki Z1Z2 For Figure I Z1 convolution, I' Z2 The corresponding elements in the middle satisfy: Where x∈1:m,y∈1:n,tr() represents the trace function on the two-dimensional matrix, I T Z1 (x,y) ±d Indicates from I T Z1 The extracted 3×3 matrix centered at (x, y) is as follows:

[0082]

[0083]

[0084] The pooling component includes a MaxPool layer, a 3×3 convolutional layer, a BN layer, a ReLU layer, and an upper layer.

[0085] Pooling output image T out5 =Us(r(b(1 c 3×3 (Mp(T in ),K5))));

[0086] in 1 c 3×3 This represents a 3×3 regular convolutional layer with a stride of 1. K5 is the kernel parameter matrix for this part, and Mp represents a max-pooling layer. For I'∈R m×n×c ,I∈R 2m×2n×c If I' = Mp(I), the elements (x, y, z) located in I' (x∈1:m, y∈1:n, z∈1:c) satisfy: I′(x, y, z) = max(I(2x-1, 2y-1, z), I(2x-1, 2y, z), I(2x, 2y-1, z), I(2x, 2y, z)); Us represents a bilinear upsampling layer. For I'∈R 2m×2n×c ,I∈R m×n×c If I' = Us(I), the elements (x, y, z) located in I and corresponding to I' (x∈1:m, y∈1:n, z∈1:c) satisfy:

[0087] I′(2x,2y,z)=I(x,y,z)

[0088]

[0089]

[0090]

[0091] To accommodate the size of the tongue image, the convolution dilation rate for gas expansion was adjusted to 3, 6, and 9.

[0092] Multi-scale semantic segmentation is achieved through a 1×1 convolution part, three 3×3 convolution parts and a pooling part. The outputs of the above five parts are fully fused by the merging module and then processed by a 3×3 convolution layer, a BN layer and a ReLU layer in sequence.

[0093] Final output image T out =r(b( 1 c 3×3 (Cat(T out1 ,T out2 ,T out3 ,T out4 ,T out5 ),K out )));

[0094] Where Cat represents the merging module, for T out1 ,T out2 ,T out3 ,T out4 ,Tout5 ∈R m×n×c The images are the outputs of a 1×1 convolution, three 3×3 convolutions, and a pooling part, respectively. The merged image belongs to R. m×n×5c And the kernel parameter matrix K out ∈R 3×3×5c×c This means that the image output from the convolutional layer contains c channels.

[0095] The image processed by the spatial pyramid is finally sent to decoder 1 and decoder 2.

[0096] The decoding process is implemented by bilinear upsampling layers of decoder 1 and decoder 2. Before each decoder's upsampling layer, there are regular convolutional layers for information fusion. Each decoder T... d =Us 2 (Sr(T e ));

[0097] Where T e T represents the input image. d Sr represents the decoded image, Sr represents the regular convolutional layer stage, and Us represents the decoded image. 2 Let Us(Us(.)) represent a simple 4-order bilinear upsampling process. Us represents the bilinear upsampling layer. For I'∈R 2m×2n×c ,I∈R m×n×c If I' = Us(I), the elements (x, y, z) located in I and corresponding to I' (x∈1:m, y∈1:n, z∈1:c) satisfy:

[0098] I′(2x,2y,z)=I(x,y,z)

[0099]

[0100]

[0101]

[0102] Step 4: Model Training

[0103] The 430 anterior tongue images and 50 dorsum tongue images with different features obtained in step one were used as the training set. The remaining 70 anterior tongue images and 6 dorsum tongue images were used as the test set. The Deep-DSP constructed in step three was then trained.

[0104] The image size was resized to 320×320, and the proposed model was trained using an Adaptive Moment Estimation (ADAM) algorithm. The loss function between the output image and the reference image was set to L1 loss. The learning rate was initialized to 1e-3 and decreased by 1e-6 every 100 iterations. The results were extracted after 100 iterations.

[0105] Step 5: Input any tongue image into the trained Deep-DSP model, and output a tongue image with the background removed.

[0106] PSPNet offers low time and space complexity but fails to produce high-quality semantic segmentation images. DeepLab v3+ extends the receptive field of pixels in the input image, but its time and space complexity is too high. Overall, Deep-DSP achieves good performance across various aspects and is suitable for practical semantic segmentation tasks.

[0107] To verify the performance of the Deep-DSP model, U-Net, PSPNet, DeepEar, and Deep-DSP were trained to segment images. The segmentation results show that the proposed Deep-DSP obtained the smoothest segmentation boundaries and the most complete segmented tongue image, without any unnecessary area. Although the Deep-DSP segmentation results exhibit a small number of pixel errors in some tongue images containing coatings or cracks, five metrics—accuracy, precision, recall, F1 score, and specificity—were used for quantitative analysis. The test results are shown in Table 1. The results indicate that Deep-DSP obtains the best objective results, i.e., it is closest to the reference image.

[0108] Table 1

[0109]

[0110] Then, the average running time of different networks was calculated, as shown in Table 2. It only takes 2.279×10-3 seconds to process an image using the Deep-DSP model.

[0111] Table 2

[0112]

[0113]

[0114] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A tongue image segmentation method based on a dual-space pyramid depthwise convolutional network, characterized in that: Step 1: Image Acquisition: To improve the generalization of the dataset, images of the tongue were captured using a camera and a mobile phone. Standard tongue images were captured by the camera, and random tongue images were captured by the mobile phone. The images were scaled to the same pixel size. A total of 500 images of the front of the tongue and 56 images of the back of the tongue were collected as the training set. Step 2: Image Processing: Remove the background from the acquired image to obtain a background-free reference image; this reference image is then used in the training of the neural network, along with the original image. Step 3: Constructing a Deep-DSP based on dual spatial pyramids: The Deep-DSP deep convolutional network based on dual spatial pyramids includes two encoders, two spatial pyramid blocks, and two decoders. After the image is input into encoder 1 and reduced to 1 / 4 of its original size, it is processed by an intermediate stream and then simultaneously input into spatial pyramid block 1 and encoder 2. The image input to encoder 2 is reduced to 1 / 4 of its input size, and then the image processed by spatial pyramid block 2 is input to decoder 1. Decoder 1 enlarges the input image by 4 times and merges it with the output image of spatial pyramid block 1 through a merging module before feeding it into decoder 2; the output image of decoder 2 is merged with the input image that has not been processed by encoder 1 through a second merging module and fed into the final output layer to obtain the binary image output by the network. Finally, the output image is obtained by the dot product with the original image. Step 4: Model Training The 430 anterior tongue images and 50 dorsum tongue images with different features from step one were used as the training set; the remaining 70 anterior tongue images and 6 dorsum tongue images were used as the test set; the Deep-DSP constructed in step three was trained; the image size was adjusted to 320×320, and the proposed model was trained using an adaptive optimizer, with the loss function between the output image and the reference image set to L1 loss; the learning rate was initialized to 1e-3, decreasing by 1e-6 every 100 iterations, and the results after 100 cycles were extracted; Step 5: Input any tongue image into the trained Deep-DSP model, and output a tongue image with the background removed.

2. The tongue image segmentation method based on a dual-space pyramid depthwise convolutional network according to claim 1, characterized in that: The collected images cover a variety of tongue abnormalities, including coatings and cracks.

3. The tongue image segmentation method based on a dual-space pyramid depthwise convolutional network according to claim 1, characterized in that: In step two, the image is manually cropped using Photoshop to ensure the accuracy of the tongue region; multiple operations are performed on the tongue image to add or crop the selected tongue region; the pixel values ​​of the selected tongue region and redundant background region are kept to [0,0,0], and the obtained image is regarded as the corresponding reference image.

4. The tongue image segmentation method based on a dual-space pyramid depthwise convolutional network according to claim 3, characterized in that: Encoder 1 includes 3×3 regular convolutional layers, batch norm layers, and modified linear unit layers. T j+1 =Sr j (T j )=r(b( 1 c 3×3 (T j ,Ke j )));where Sr j This represents the j-th stage encompassing a regular convolutional layer, r represents a modified linear unit, and b represents a batch norm layer. 1 c 3×3 Represents a 3×3 conventional convolutional layer, K ej Represents the convolution kernel, T j T represents the input image in stage j. j+1 This represents the input image of stage j+1, which is also the output image of stage j.

5. The tongue image segmentation method based on a dual-space pyramid depthwise convolutional network according to claim 4, characterized in that: Encoder 2 includes 3×3 convolutional layers, which have similar functions to pooling layers and are trainable. It includes BN layers and ReLU layers. Compared with encoder 1, encoder 2 needs to process images with a length and width that are 1 / 4 of the images processed by encoder 1. Sp j This represents a convolutional layer with a stride of 2 (represented as...). 1 c 2 3×3 The j-th stage; For I'∈R m×n×c ,I∈R 2m×2n×c If I'= 1 c 2 3×3 (I,Ke j ),(Ke j ∈R 3×3×c×c The corresponding elements (x, y, z) located at I' (x∈1:m, y∈1:n, z∈1:c) satisfy: Each encoder consists of one backbone layer and two auxiliary layers. The backbone layer comprises two parts, each consisting of three regular convolutional modules. The two auxiliary layers are connected to the first and second halves of the backbone layer, respectively. Because the auxiliary layers have shallower convolutions, they focus more on the original information than the backbone layer. By increasing the connections in the auxiliary layers, the influence of the original information on the encoding is enhanced. Encoder T t =Cat(Sp2·Sr1·Sr0·T) 0 ,Sp0·T 0 ); where Cat represents the merging module, the first input of Cat represents the main layer, the second input represents the auxiliary layer, and T 0 T represents the input image. t This represents the output image.

6. The tongue image segmentation method based on a dual-space pyramid depthwise convolutional network according to claim 5, characterized in that: The intermediate flow consists of 3×3 convolutional layers, batch norm layers, and ReLU layers; T t′ =Sm2·Sm1·Sm0·T 0′ ; Where Sm j ' represents the intermediate flow stage, calculated as: T j′+1 =Sm j′ (T j′ )=r(b( 3 c 3×3 (T j′ ,Ke j′ ))).

7. The tongue image segmentation method based on a dual-space pyramid depthwise convolutional network according to claim 6, characterized in that: The spatial pyramid block consists of a 1×1 convolutional part, three 3×3 non-convolutional parts, and a pooling part; The 1×1 convolutional part includes 1×1 convolutional layers, batch norm (BN) layers, and integer positive linear activation function (ReLU) layers; Convolution output image T out1 =r(b(c 1×1 (T in ,M1)));where T in ∈R m×n×c This represents the input encoded image, where m, n, and c are the image height, width, and number of channels, respectively, and T. out1 ∈R m×n×c This represents the output image from the first part; c 1×1 Let M1 ∈ R be a 1×1 convolutional layer. c×c A matrix representing kernel parameters; For I,I'∈R m×n×c If I' = c 1×1 (I), the corresponding channel transpose vectors I(x,y), I'(x,y)∈R 1×c (x∈1:m,y∈1:n) satisfies: I′(x,y)=I(x,y)M1; r represents the ReLU layer; if I'=r(I), the corresponding elements (x,y,z) (x∈1:m,y∈1:n,z∈1:c) from I' and I satisfy: b represents the BN layer; If I'=b(I), The corresponding elements (x, y, z) (x∈1:m, y∈1:n, z∈1:c) from I' and I satisfy: Where mean(I) z D(I) represents the average value of the z-channel pixels in image I. z ) represents the variance of the z-channel pixels in image I, and ε is an additional value in the denominator to maintain numerical stability, set to ε = 1 × 10⁻⁶. -5 ; The 3×3 regular convolutional part consists of a 3×3 regular convolutional layer, a BN layer, and a ReLU layer; Regular convolution output image T outi =r(b( d c 3×3 (T in ,K i )));where T outi ∈R m×n×c Let i represent the output image of the i-th part, where i = 3, 4, or 5. d c 3×3 This represents a 3×3 dilated convolutional layer with a dilation rate of d. To ensure that the processed image is not distorted, the padding amount is equal to the dilation rate; K i ∈R 3×3×c×c This represents the kernel parameter matrix from the i-th part; if I' = d c 3×3 (I,K i The channel map I'(:,:,z2)(z2∈1:c) of the corresponding output image I' satisfies: Ki Z1Z2 Represents Ki(:,:,z1,z2), I Z1 Represents I(:,:,z1),I' Z2 Represents I(:,:,z2), * represents passing through convolution kernel Ki Z1Z2 For Figure I Z1 convolution, I' Z2 The corresponding elements in the middle satisfy: Where x∈1:m,y∈1:n,tr() represents the trace function on the two-dimensional matrix, I T Z1 (x,y) ±d Indicates from I T Z1 The extracted 3×3 matrix centered at (x, y) is as follows: The pooling part includes a MaxPool layer, a 3×3 convolutional layer, a BN layer, a ReLU layer, and an upper layer; Pooling output image T out5 =Us(r(b( 1 c 3×3 (Mp(T in ),K5)))); in 1 c 3×3 This represents a 3×3 regular convolutional layer with a stride of 1. K5 is the kernel parameter matrix for this part, and Mp represents a max pooling layer. For I'∈R m×n×c ,I∈R 2m×2n×c If I' = Mp(I), the elements (x, y, z) located in I' (x∈1:m, y∈1:n, z∈1:c) satisfy: I′(x, y, z) = max(I(2x-1, 2y-1, z), I(2x-1, 2y, z), I(2x, 2y-1, z), I(2x, 2y, z)); Us represents a bilinear upsampling layer; for I'∈R 2m×2n×c ,I∈R m×n×c If I' = Us(I), the elements (x, y, z) located at I and corresponding to I' (x∈1:m, y∈1:n, z∈1:c) satisfy: I′(2x,2y,z)=I(x,y,z) To accommodate the size of the tongue image, the convolution dilation rate for gas expansion was adjusted to 3, 6, and 9.

8. The tongue image segmentation method based on a dual-space pyramid depthwise convolutional network according to claim 7, characterized in that: Multi-scale semantic segmentation is achieved through a 1×1 convolution part, three 3×3 convolution parts and a pooling part. The outputs of the above five parts are fully fused by the merging module and then processed by a 3×3 convolution layer, a BN layer and a ReLU layer in sequence. Final output image T out =r(b( 1 c 3×3 (Cat(T out1 ,T out2 ,T out3 ,T out4 ,T out5 ),K out ))); Where Cat represents the merging module, for T out1 ,T out2 ,T out3 ,T out4 ,T out5 ∈R m×n×c The images are the outputs of a 1×1 convolution, three 3×3 convolutions, and a pooling part, respectively. The merged image belongs to R. m×n×5c And the kernel parameter matrix K out ∈R 3×3×5c×c This means that the image output from the convolutional layer contains c channels; The image processed by the spatial pyramid is finally sent to decoder 1 and decoder 2; The decoding process is implemented by bilinear upsampling layers of decoder 1 and decoder 2. Before each decoder's upsampling layer, there are regular convolutional layers for information fusion. Each decoder T... d =Us 2 (Sr(T e )); Where T e T represents the input image. d Sr represents the decoded image, Sr represents the regular convolutional layer stage, and Us represents the decoded image. 2 Us(Us(.)) represents a simple 4-order bilinear upsampling process; Us represents the bilinear upsampling layer; for I'∈R 2m×2n×c ,I∈R m×n×c If I' = Us(I), the elements (x, y, z) located in I and corresponding to I' (x∈1:m, y∈1:n, z∈1:c) satisfy: I′(2x,2y,z)=I(x,y,z)

Images