Tongue image segmentation method and device, storage medium and computer equipment
By using downsampling encoder, upsampling encoder and fully connected layer packet convolution processing in tongue image segmentation technology, the problem of low accuracy of tongue image segmentation in the prior art under unstable light sources and environments is solved, and the problem of low accuracy of tongue image segmentation in the prior art is realized. Higher accuracy of tongue image segmentation.
Patent Information
- Application Number
- CN202411189804.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-28
- Publication Date
- 2025-02-07
AI Technical Summary
The existing tongue image segmentation technology cannot accurately identify the edge area of the tongue when the light source is unstable, the light intensity is unstable and the shooting environment is not fixed, resulting in low accuracy of tongue image segmentation.
The initial tongue image segmentation model is adopted, including a downsampling encoder, an upsampling encoder and a fully connected layer. Through feature extraction and grouping convolution processing, the target tongue image segmentation model is trained to improve the accuracy of tongue image segmentation.
Through grouping convolution and depth separation convolution processing, we can pay more comprehensive attention to the global features of the image, improve the accuracy of tongue image segmentation, and is suitable for tongue image segmentation in unstable light sources and unfixed environments.
Smart Images
Figure CN119399211A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of tongue image segmentation, and in particular to a tongue image segmentation method, apparatus, storage medium and computer equipment. Background Art
[0002] Tongue image segmentation can extract tongue images for users. However, existing tongue image segmentation can only perform feature extraction and recognition for a relatively small receptive field, and the consideration of global information is seriously insufficient. Therefore, it is only suitable for standard acquisition environments. This is because the light source is stable, the light intensity is appropriate, and the shooting environment is fixed in the standard acquisition environment. Therefore, when performing tongue image segmentation, feature extraction and recognition for a relatively small receptive field can also identify the tongue image. However, for images taken in natural environments with unstable light sources, unstable light intensity, and unstable shooting environments, the edge area of the tongue is easily unstable in shape due to punctures and tooth marks, resulting in the inability of the existing tongue image segmentation to accurately identify the edge area of the tongue. Therefore, the existing technology has a technical defect of low accuracy in tongue image segmentation. Summary of the invention
[0003] The purpose of this application is to overcome the shortcomings and deficiencies in the prior art and to provide a tongue image segmentation method, apparatus, storage medium and computer equipment that can improve the accuracy of tongue image segmentation.
[0004] A first aspect of an embodiment of the present application provides a tongue image segmentation method, comprising:
[0005] Acquire multiple image samples to be segmented, multiple corresponding image segmentation result samples, and an initial tongue image segmentation model; the initial tongue image segmentation model includes a downsampling encoder, an upsampling encoder, and a fully connected layer;
[0006] Inputting the image sample to be segmented into the downsampling encoder for feature extraction and group convolution processing to obtain first image data;
[0007] Inputting the image sample to be segmented and the first image data into an upsampling decoder for upsampling and grouped convolution processing to obtain second image data; inputting the second image data into the fully connected layer to obtain an initial image segmentation result;
[0008] According to the plurality of image segmentation result samples and the plurality of initial image segmentation results, training the model parameters of the initial tongue image segmentation model to obtain a trained target tongue image segmentation model;
[0009] Acquire a target image to be segmented; input the target image to be segmented into the target tongue image segmentation model to obtain a tongue image segmentation result in the image to be segmented.
[0010] A second aspect of an embodiment of the present application provides a tongue image segmentation device, comprising:
[0011] A sample model acquisition module, used to acquire a plurality of image samples to be segmented, a plurality of corresponding image segmentation result samples and an initial tongue image segmentation model; the initial tongue image segmentation model includes a downsampling encoder, an upsampling encoder and a fully connected layer;
[0012] A first image data acquisition module, used for inputting the image sample to be segmented into the downsampling encoder for feature extraction and group convolution processing to obtain first image data;
[0013] An initial image segmentation result acquisition module is used to input the image sample to be segmented and the first image data into an upsampling decoder for upsampling and grouped convolution processing to obtain second image data; and input the second image data into the fully connected layer to obtain an initial image segmentation result;
[0014] A training module, used for training the model parameters of the initial tongue image segmentation model according to the plurality of image segmentation result samples and the plurality of initial image segmentation results, to obtain a trained target tongue image segmentation model;
[0015] The tongue image segmentation result acquisition module is used to acquire the target image to be segmented; input the target image to be segmented into the target tongue image segmentation model to obtain the tongue image segmentation result in the image to be segmented.
[0016] A third aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the tongue image segmentation method described above are implemented.
[0017] A fourth aspect of an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable by the processor, wherein the processor implements the steps of the tongue image segmentation method as described above when executing the computer program.
[0018] Compared with the related art, the present application uses the downsampling encoder of the initial tongue image segmentation model to perform feature extraction and group convolution processing on the image samples to be segmented to obtain the first image data, and then uses the upsampling decoder of the initial tongue image segmentation model to upsample and group convolution processing on the first image data to obtain the second image data, and then obtains the initial image segmentation result corresponding to the second image data according to the fully connected layer of the initial tongue image segmentation model, and then trains a target tongue image segmentation model including a downsampling encoder, an upsampling encoder and a fully connected layer according to multiple image segmentation result samples and multiple initial image segmentation results, so as to obtain the tongue image segmentation result of the target image to be segmented. Among them, since both the downsampling encoder and the upsampling decoder use group convolution for convolution processing, compared with ordinary convolution processing, they can pay more attention to the global features of the image, thereby improving the accuracy of tongue image segmentation.
[0019] In order to provide a clearer understanding of the present application, the specific implementation of the present application will be described below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] Figure 1 This is a flowchart of a tongue image segmentation method according to an embodiment of the present application.
[0021] Figure 2 A schematic diagram of an initial tongue image segmentation model of a tongue image segmentation method according to an embodiment of the present application.
[0022] Figure 3 A schematic diagram of a group convolutional layer of a tongue image segmentation device according to an embodiment of the present application.
[0023] Figure 4 Schematic diagram of a module of a tongue image segmentation device according to an embodiment of the present application.
[0024] 100. Tongue image segmentation device; 101. Sample model acquisition module; 102. First image data acquisition module; 103. Initial image segmentation result acquisition module; 104. Training module; 105. Tongue image segmentation result acquisition module. DETAILED DESCRIPTION
[0025] In order to make the objectives, technical solutions and advantages of the present application more clear, the embodiments of the present application will be further described in detail below with reference to the accompanying drawings.
[0026] It should be clear that the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the embodiments of the present application.
[0027] When the following description relates to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. In the description of the present application, it should be understood that the terms "first", "second", "third", etc. are only used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence, nor can they be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances. The singular forms of "a", "said" and "the" used in the present application and the appended claims are also intended to include the majority form, unless the context clearly indicates other meanings. The words "if" / "if" used herein can be interpreted as "at the time of" or "when" or "in response to determination".
[0028] In addition, in the description of this application, unless otherwise specified, "plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.
[0029] See also Figure 1 , which is a flow chart of the tongue image segmentation method of the first embodiment of the present application, comprising:
[0030] Step 1: Obtain multiple image samples to be segmented, multiple corresponding image segmentation result samples, and an initial tongue image segmentation model. Figure 2 , the initial tongue image segmentation model includes a downsampling encoder, an upsampling encoder and a fully connected layer, Figure 2 In the figure, s1 is the input layer of the initial tongue image segmentation model, s2-s9 are downsampling encoders, s10-s16 are upsampling encoders, s17 is the fully connected layer, and s18 is the output layer of the initial tongue image segmentation model.
[0031] Step 2: Input the image sample to be segmented into the downsampling encoder for feature extraction and group convolution processing to obtain the first image data.
[0032] Wherein, the downsampling encoder includes multiple pairs of feature extraction layers and group convolution layers connected in sequence, so step 2 may include:
[0033] Step 21: Input the image sample to be segmented into the feature extraction layer for image feature extraction processing to obtain image feature data.
[0034] Step 22: Input the image feature data into the corresponding group convolution layer, perform group convolution processing through the group convolution layer, and obtain first convolution data.
[0035] Step 23: Input the first convolution data to the next feature extraction layer, and input the image feature data output by the next feature extraction layer to the corresponding group convolution layer, until all feature extraction layers and group convolution layers of the downsampling encoder are traversed, and the first convolution data output by the last group convolution layer of the downsampling encoder is determined as the first image data.
[0036] Step 3: Input the image sample to be segmented and the first image data into the upsampling decoder for upsampling and grouped convolution processing to obtain second image data; input the second image data into the fully connected layer to obtain an initial image segmentation result.
[0037] The upsampling encoder includes a plurality of pairs of first upsampling layers and group convolution layers connected in sequence, and a second upsampling layer connected to the last group convolution layer. Therefore, step 3 may include:
[0038] Step 31: input the image sample to be segmented and the first image data into the first upsampling layer for upsampling processing to obtain image upsampling data.
[0039] Step 32: Input the image upsampled data into the corresponding group convolution layer, perform group convolution processing through the group convolution layer, and obtain second convolution data.
[0040] Step 33: Input the second convolution data and the first image data into the next first upsampling layer, and input the image upsampling data output by the next first upsampling layer into the corresponding group convolution layer, until all the first upsampling layers and group convolution layers of the upsampling decoder are traversed to obtain the target second convolution data output by the last group convolution layer of the upsampling decoder.
[0041] Step 34: Input the target second convolution data into the second upsampling layer for upsampling processing to obtain the second image data.
[0042] It should be noted that each node of the fully connected layer is connected to all nodes of the previous layer, and is used to integrate the features of the second image data extracted by the downsampling encoder and the upsampling encoder to play a classification role, thereby obtaining the initial image segmentation result.
[0043] Step 4: According to the plurality of image segmentation result samples and the plurality of initial image segmentation results, the model parameters of the initial tongue image segmentation model are trained to obtain a trained target tongue image segmentation model.
[0044] Among them, the cross entropy loss function can be used to calculate the loss result, so as to train the model parameters of the initial tongue image segmentation model according to the loss result. The cross entropy loss function can be expressed as the following formula:
[0045]
[0046] Among them, L CE is the loss result; N is the batch size of the input, and the batch size refers to the number of training samples used in one iteration of model update; C is the number of categories; H and W are the sizes of the images, t n,h,w,c is the value of the true label c-th category at position h and w in the n-th image of the initial image segmentation result, and yn,h,w,c is the value of the true label c-th category at position h and w in the n-th image of the image segmentation result sample.
[0047] Step 5: Obtain a target image to be segmented; input the target image to be segmented into the target tongue image segmentation model to obtain a tongue image segmentation result in the image to be segmented.
[0048] The target image to be segmented refers to an image including the tongue image and background elements, and the tongue image segmentation result refers to the image result of segmenting the tongue image from the target image to be segmented.
[0049] Compared with the related art, the present application uses the downsampling encoder of the initial tongue image segmentation model to perform feature extraction and group convolution processing on the image samples to be segmented to obtain the first image data, and then uses the upsampling decoder of the initial tongue image segmentation model to perform upsampling and group convolution processing on the first image data to obtain the second image data, and then obtains the initial image segmentation result corresponding to the second image data according to the fully connected layer of the initial tongue image segmentation model, and then trains a target tongue image segmentation model including a downsampling encoder, an upsampling encoder and a fully connected layer according to multiple image segmentation result samples and multiple initial image segmentation results, so as to obtain the tongue image segmentation result of the target image to be segmented. Among them, since both the downsampling encoder and the upsampling decoder use group convolution and depth separation convolution and depth separation dilated convolution to perform convolution processing, the depth separation convolution and depth separation dilated convolution cascade can pay more attention to the spatial features of the image compared to ordinary convolution processing; the two cascaded group convolutions pay attention to the full channel information of the image with fewer parameters compared to ordinary convolution processing, thereby improving the accuracy of tongue image segmentation.
[0050] See also Figure 3 In a feasible embodiment, the group convolution layer includes at least one group convolution module, and the group convolution module includes a group convolution processing submodule and a feedforward neural network module. Figure 3 The leftmost flowchart is the group convolution module, and the group convolution processing submodule is Figure 3 The black dashed box in the leftmost flowchart is the feedforward neural network module. Figure 3 The red dashed box in the leftmost flowchart, and Figure 3 The arrows filled with white in the figure represent the partial structure of the corresponding modules or units in the group convolution module.
[0051] The step of performing group convolution processing through the group convolution layer comprises:
[0052] Sub-step 1: Standardize the data input to the group of convolutional layers to obtain standardized data.
[0053] Among them, the standardization processing can be achieved through a standardization layer, such as using a BN layer (BatchNorm). The BN layer can force the network output mean and variance back to a normal distribution of 0 and 1 through standardization operations, and then retain the original feature characteristics and prevent the data from being concentrated near 0 through scaling and offset operations.
[0054] Sub-step 2: Input the standardized data into the group convolution processing submodule, perform group convolution and attention extraction processing through the group convolution processing submodule, and obtain a first group of convolution attention data.
[0055] like Figure 3 As shown, the group convolution processing submodule (LSKA-SF Attention) includes a group convolution unit (LSKA-SF); sub-step 2 may include:
[0056] Sub-step 21: Perform convolution and activation processing on the standardized data to obtain second convolution activation data.
[0057] Among them, the convolution and activation of sub-step 21 can be obtained by Figure 3 1×1Conv and GELU implementation in the middle black dashed box.
[0058] Sub-step 22: Input the second convolution activation data into the group convolution unit, perform group convolution and attention extraction processing through the group convolution unit, and obtain a second group of convolution attention data.
[0059] The group convolution unit (LSKA-SF) includes sequentially connecting the first group convolution (1×1GConv1) and the second group convolution (1×1GConv2); sub-step 22 also includes:
[0060] Sub-step 221: input the second convolution activation data into the first group of convolutions, determine the number of groups according to the number of input channels, perform group convolution using the number of groups, and obtain a first group of convolution data.
[0061] Among them, the first set of convolutions is a type of group convolution, which is a technique used in deep learning. It divides the input feature map into several groups by channel, and then performs regular convolution on each group. This technique helps reduce the amount of calculation and model parameters, thereby improving the efficiency and performance of the model.
[0062] Determine the number of groups based on the number of input channels, which can be achieved through the following formula:
[0063] The receptive range of the group convolution is obtained through the following formula:
[0064] When not an integer
[0065] Among them, k is the receptive range of group convolution, C is the number of channels, R is the channel number scaling ratio, and γ is the preset operation parameter;
[0066] The number of groups is obtained using the following formula:
[0067]
[0068] Among them, G is the number of groups.
[0069] Sub-step 222: Perform depth separation convolution processing and depth separation dilation convolution processing on the first group of convolution data to obtain depth convolution data.
[0070] See also Figure 3 , sub-step 222 may Figure 3 The is implemented through 1×5DW-Conv, 5×1DW-Conv, 1×7DW-Conv and 7×1DW-Conv. Specifically, 1×5DW-Conv and 1×7DW-Conv are used to implement deep separation convolution processing, and 5×1DW-Conv and 7×1DW-Conv are used to implement deep separation dilated convolution processing.
[0071] Sub-step 223: input the depth convolution data into the second group of convolutions, determine the number of groups according to the number of input channels, perform group convolution using the number of groups, and obtain a second group of convolution data.
[0072] Among them, the second group of convolutions is a kind of group convolution, but the connection relationship of the second group of convolutions is different from the connection relationship of the first group of convolutions.
[0073] Sub-step 224: Perform attention extraction processing on the second convolution activation data and the second group of convolution data to obtain the second group of convolution attention data.
[0074] Among them, the attention extraction processing of sub-step 224 can be implemented by using matrix multiplication processing of cross multiplication.
[0075] Sub-step 23: Perform convolution processing on the second set of convolution attention data to obtain the first set of convolution attention data.
[0076] Among them, the convolution processing of sub-step 23 can be implemented by using the convolution (1×1Conv) below the group convolution unit (LSKA-SF).
[0077] Sub-step 3: Fuse and normalize the data input to the group of convolutional layers and the first group of convolutional attention data to obtain fused and normalized data to obtain first fused and normalized data.
[0078] Among them, the fusion processing of sub-step 3 can be implemented by element-by-element addition, and the normalization processing can be implemented by BN.
[0079] Sub-step 4: Input the first fused standardized data into a feedforward neural network module for convolution and activation processing to obtain first convolution activation data.
[0080] Among them, the feedforward neural network module is Figure 3 In the FFN, the feedforward neural network adopts a unidirectional multi-layer structure. The feedforward neural network has a simple structure and is widely used. It can approximate any continuous function and square integrable function with arbitrary precision. And it can accurately realize any finite training sample set. From the system point of view, the feedforward network is a static nonlinear mapping. Through the composite mapping of simple nonlinear processing units, complex nonlinear processing capabilities can be obtained. From the computational point of view, it lacks rich dynamic behavior. Most feedforward networks are learning networks, and their classification and pattern recognition capabilities are generally stronger than those of feedback networks.
[0081] Sub-step 5: Fusing the first fused normalized data and the convolution activation data to obtain the output of the group convolution module.
[0082] The fusion processing in sub-step 5 can be implemented by element-by-element addition.
[0083] In this embodiment, through sub-steps 1-5, the input data can be smoothly grouped and convolved through the group convolution layer to implement the process of step 22 and / or step 23.
[0084] In order to facilitate understanding of the embodiments of the present application, Figure 1 The following example is used for illustration:
[0085] S1: Input is the image input, and its shape is [1, 3, 512, 512], where 1 represents the input batch size b, 3 represents the number of channels c of the input image, and the two 512s represent the image sizes h and w respectively.
[0086] S2: The Patch Embedding layer consists of a convolution kernel size of 3, a stride of 2, a padding of 1, 3 input channels, 32 output channels, and a batch normalization. After a Patch Embedding layer, the output shape is a tensor of [1, 32, 256, 256].
[0087] The LSKA-SF Block x2 of S3, S5, S7, S9, S11, S13, and S15 is composed of two LSKA-SF Blocks in cascade. The LSKA-SF Block structure is as follows: Figure 3 .
[0088] S3: The convolution kernel size of the two 1×1Convs of LSKA-SF Attention is 1, the stride is 1, the padding is 0, the number of input channels is 32, and the number of output channels is 32. GELU is the activation function. In LSKA-SF, the convolution kernel size of 1×1GConv1 is 1, the stride is 1, the padding is 0, the number of groups is 4, the number of input channels is 32, and the number of output channels is 16; the convolution kernel size of 1×5DW-Conv is (1, 5), the stride is (1, 1), the padding is (0, 2), the number of groups is 16, the number of input channels is 16, and the number of output channels is 16; the convolution kernel size of 5×1DW-Conv is (5, 1), the stride is (1, 1), the padding is (2, 0), the number of groups is 16, the number of input channels is 16, and the number of output channels is 16; the convolution kernel size of 1×7DW-D-Conv is (1, 7), the stride is (1, 1), the padding is (0, 9), the expansion rate is (3, 3), the number of groups is 16, the number of input channels is 16, and the number of output channels is 16; The convolution kernel size is (7, 1), the stride is (1, 1), the padding is (9, 0), the expansion rate is (3, 3), the number of groups is 16, the number of input channels is 16, and the number of output channels is 16; the convolution kernel size of 1×1GConv2 is 1, the stride is 1, the padding is 0, the number of groups is 4, the number of input channels is 16, and the number of output channels is 32; the convolution kernel size of the first 1×1Conv of FFN is 1, the stride is 1, the padding is 0, the number of input channels is 32, and the number of output channels is 256; the convolution kernel size of 3×3DW-Conv is (3, 3), the stride is (1, 1), the padding is (1, 1), the number of groups is 256, the number of input channels is 256, and the number of output channels is 256; the convolution kernel size of the second 1×1Conv is 1, the stride is 1, the padding is 0, the number of input channels is 256, and the number of output channels is 32. S3 outputs a tensor of shape [1, 32, 256, 256].
[0089] S4: The Patch Embedding layer consists of a convolution kernel size of 3, a stride of 2, a padding of 1, an input channel number of 32, an output channel number of 64, and a batch normalization. After a Patch Embedding layer, the output shape is a tensor of [1, 64, 128, 128].
[0090] S5: The convolution kernel size of the two 1×1Convs of LSKA-SF Attention is 1, the stride is 1, the padding is 0, the number of input channels is 64, and the number of output channels is 64. GELU is the activation function. In LSKA-SF, the convolution kernel size of 1×1GConv1 is 1, the stride is 1, the padding is 0, the number of groups is 8, the number of input channels is 64, and the number of output channels is 32; the convolution kernel size of 1×5DW-Conv is (1, 5), the stride is (1, 1), the padding is (0, 2), the number of groups is 32, the number of input channels is 32, and the number of output channels is 32; the convolution kernel size of 5×1DW-Conv is (5, 1), the stride is (1, 1), the padding is (2, 0), the number of groups is 32, the number of input channels is 32, and the number of output channels is 32; the convolution kernel size of 1×7DW-D-Conv is (1, 7), the stride is (1, 1), the padding is (0, 9), the expansion rate is (3, 3), the number of groups is 32, the number of input channels is 32, and the number of output channels is 32; The convolution kernel size is (7, 1), the stride is (1, 1), the padding is (9, 0), the expansion rate is (3, 3), the number of groups is 32, the number of input channels is 32, and the number of output channels is 32; the convolution kernel size of 1×1GConv2 is 1, the stride is 1, the padding is 0, the number of groups is 4, the number of input channels is 32, and the number of output channels is 64; the convolution kernel size of the first 1×1Conv of FFN is 1, the stride is 1, the padding is 0, the number of input channels is 64, and the number of output channels is 512; the convolution kernel size of 3×3DW-Conv is (3, 3), the stride is (1, 1), the padding is (1, 1), the number of groups is 512, the number of input channels is 512, and the number of output channels is 512; the convolution kernel size of the second 1×1Conv is 1, the stride is 1, the padding is 0, the number of input channels is 512, and the number of output channels is 64. S5 outputs a tensor of shape [1, 64, 128, 128].
[0091] S6: The Patch Embedding layer consists of a convolution kernel size of 3, a stride of 2, a padding of 1, an input channel number of 64, an output channel number of 160, and a batch normalization. After a Patch Embedding layer, the output shape is a tensor of [1, 160, 64, 64].
[0092] S7: The convolution kernel size of the two 1×1Convs of LSKA-SF Attention is 1, the stride is 1, the padding is 0, the number of input channels is 160, and the number of output channels is 160. GELU is the activation function. In LSKA-SF, the convolution kernel size of 1×1GConv1 is 1, the stride is 1, the padding is 0, the number of groups is 10, the number of input channels is 160, and the number of output channels is 80; the convolution kernel size of 1×5DW-Conv is (1, 5), the stride is (1, 1), the padding is (0, 2), the number of groups is 80, the number of input channels is 80, and the number of output channels is 80; the convolution kernel size of 5×1DW-Conv is (5, 1), the stride is (1, 1), the padding is (2, 0), the number of groups is 80, the number of input channels is 80, and the number of output channels is 80; the convolution kernel size of 1×7DW-D-Conv is (1, 7), the stride is (1, 1), the padding is (0, 9), the expansion rate is (3, 3), the number of groups is 80, the number of input channels is 80, and the number of output channels is 80; the convolution kernel size of 7×1DW-D-Conv is (1, 7), the stride is (1, 1), the padding is (0, 9), the expansion rate is (3, 3), the number of groups is 80, the number of input channels is 80, and the number of output channels is 80; The kernel size is (7, 1), the stride is (1, 1), the padding is (9, 0), the expansion rate is (3, 3), the number of groups is 80, the number of input channels is 80, and the number of output channels is 80; the convolution kernel size of 1×1GConv2 is 1, the stride is 1, the padding is 0, the number of groups is 8, the number of input channels is 80, and the number of output channels is 160; the convolution kernel size of the first 1×1Conv of FFN is 1, the stride is 1, the padding is 0, the number of input channels is 160, and the number of output channels is 640; the convolution kernel size of 3×3DW-Conv is (3, 3), the stride is (1, 1), the padding is (1, 1), the number of groups is 640, the number of input channels is 640, and the number of output channels is 640; the convolution kernel size of the second 1×1Conv is 1, the stride is 1, the padding is 0, the number of input channels is 640, and the number of output channels is 160. S7 outputs a tensor of shape [1, 160, 128, 128].
[0093] S8: The Patch Embedding layer consists of a convolution kernel size of 3, a stride of 2, a padding of 1, an input channel number of 160, an output channel number of 256, and a batch normalization. After a Patch Embedding layer, the output shape is a tensor of [1, 256, 32, 32].
[0094] The execution bodies of S9, S11, S13, and S15 are set the same. Take S9 as an example. The convolution kernel size of the two 1×1Convs of LSKA-SF Attention in S9 is 1, the step size is 1, the padding is 0, the number of input channels is 256, and the number of output channels is 256. GELU is the activation function. In LSKA-SF, the convolution kernel size of 1×1GConv1 is 1, the stride is 1, the padding is 0, the number of groups is 16, the number of input channels is 256, and the number of output channels is 128; the convolution kernel size of 1×5DW-Conv is (1, 5), the stride is (1, 1), the padding is (0, 2), the number of groups is 128, the number of input channels is 128, and the number of output channels is 128; the convolution kernel size of 5×1DW-Conv is (5, 1), the stride is (1, 1), the padding is (2, 0), the number of groups is 128, the number of input channels is 128, and the number of output channels is 128; 1×7DW-D -Conv has a convolution kernel size of (1, 7), a stride of (1, 1), a padding of (0, 9), a dilation rate of (3, 3), a grouping number of 128, a number of input channels of 128, and a number of output channels of 128; the convolution kernel size of 7×1DW-D-Conv is (7, 1), a stride of (1, 1), a padding of (9, 0), a dilation rate of (3, 3), a grouping number of 128, a number of input channels of 128, and a number of output channels of 128; the convolution kernel size of 1×1GConv2 is 1, a stride of 1, a padding of 0, a grouping number of 8, a number of input channels of 128, and a number of output channels of 256. The convolution kernel size of the first 1×1Conv of FFN is 1, the stride is 1, the padding is 0, the number of input channels is 256, and the number of output channels is 1024; the convolution kernel size of the 3×3DW-Conv is (3, 3), the stride is (1, 1), the padding is (1, 1), the number of groups is 1024, the number of input channels is 1024, and the number of output channels is 1024; the convolution kernel size of the second 1×1Conv is 1, the stride is 1, the padding is 0, the number of input channels is 1024, and the number of output channels is 256. S9 outputs a tensor of shape [1, 256, 32, 32]. S11 outputs a tensor of shape [1, 256, 64, 64]. S13 outputs a tensor of shape [1, 256, 128, 128]. S15 outputs a tensor of shape [1, 256, 256, 256].
[0095] The execution bodies of S10, S12, S14, and S16 are all upsampling modules. The outputs of S1 and S9 are input into S10 to obtain a tensor with a shape of [1, 256, 64, 64]. The outputs of S1 and S11 are input into S12 to obtain a tensor with a shape of [1, 256, 128, 128]. The outputs of S1 and S13 are input into S14 to obtain a tensor with a shape of [1, 256, 256, 256]. The outputs of S1 and S15 are input into S16 to obtain a tensor with a shape of [1, 256, 512, 512].
[0096] S17: The execution body consists of a convolution kernel with a size of 1, a stride of 1, a padding of 0, an input channel number of 256, and an output channel number of 2. After a Linear layer, the output shape is a tensor of [1, 2, 512, 512], which is the Output.
[0097] See also Figure 4 The second embodiment of the present application provides a tongue image segmentation device 100, comprising:
[0098] The sample model acquisition module 101 is used to acquire a plurality of image samples to be segmented, a plurality of corresponding image segmentation result samples and an initial tongue image segmentation model; the initial tongue image segmentation model includes a downsampling encoder, an upsampling encoder and a fully connected layer;
[0099] A first image data acquisition module 102 is used to input the image sample to be segmented into the downsampling encoder for feature extraction and group convolution processing to obtain first image data;
[0100] The initial image segmentation result acquisition module 103 is used to input the image sample to be segmented and the first image data into an upsampling decoder for upsampling and grouped convolution processing to obtain second image data; and input the second image data into the fully connected layer to obtain an initial image segmentation result;
[0101] A training module 104 is used to train the model parameters of the initial tongue image segmentation model according to the plurality of image segmentation result samples and the plurality of initial image segmentation results to obtain a trained target tongue image segmentation model;
[0102] The tongue image segmentation result acquisition module 105 is used to acquire a target image to be segmented; input the target image to be segmented into the target tongue image segmentation model to obtain a tongue image segmentation result in the image to be segmented.
[0103] It should be noted that the tongue image segmentation device 100 provided in the second embodiment of the present application only uses the division of the above-mentioned functional modules as an example when executing the tongue image segmentation method. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the tongue image segmentation device 100 provided in the second embodiment of the present application and the tongue image segmentation method of the first embodiment of the present application belong to the same concept. The implementation process thereof is detailed in the method embodiment and will not be repeated here.
[0104] A third aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the tongue image segmentation method described above are implemented.
[0105] A fourth aspect of an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable by the processor, wherein the processor implements the steps of the tongue image segmentation method as described above when executing the computer program.
[0106] The device embodiments described above are merely illustrative, wherein the components described as separate parts may or may not be physically separated, and the parts displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application. Ordinary technicians in this field can understand and implement it without creative work.
[0107] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0108] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the function selected in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 function selected in a box or multiple boxes.
[0109] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 steps for the function selected in a box or multiple boxes.
[0110] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0111] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0112] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0113] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0114] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any training, equivalent substitution, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims
1. A tongue image segmentation method, characterized in that: include: Acquire multiple image samples to be segmented, multiple corresponding image segmentation result samples, and an initial tongue image segmentation model; the initial tongue image segmentation model includes a downsampling encoder, an upsampling encoder, and a fully connected layer; Inputting the image sample to be segmented into the downsampling encoder for feature extraction and group convolution processing to obtain first image data; Inputting the image sample to be segmented and the first image data into an upsampling decoder for upsampling and grouped convolution processing to obtain second image data; Inputting the second image data into the fully connected layer to obtain an initial image segmentation result; According to the plurality of image segmentation result samples and the plurality of initial image segmentation results, training the model parameters of the initial tongue image segmentation model to obtain a trained target tongue image segmentation model; Acquire a target image to be segmented; input the target image to be segmented into the target tongue image segmentation model to obtain a tongue image segmentation result in the image to be segmented.
2. The tongue image segmentation method according to claim 1, characterized in that: The downsampling encoder includes a plurality of pairs of feature extraction layers and group convolution layers connected in sequence; The step of inputting the image sample to be segmented into the downsampling encoder for feature extraction and group convolution processing to obtain the first image data comprises: Inputting the image sample to be segmented into the feature extraction layer to perform image feature extraction processing to obtain image feature data; Inputting the image feature data into a corresponding group convolution layer, performing group convolution processing through the group convolution layer, and obtaining first convolution data; The first convolution data is input to the next feature extraction layer, and the image feature data output by the next feature extraction layer is input to the corresponding group convolution layer, until all feature extraction layers and group convolution layers of the downsampling encoder are traversed, and the first convolution data output by the last group convolution layer of the downsampling encoder is determined as the first image data.
3. The tongue image segmentation method according to claim 1, characterized in that: The upsampling encoder comprises a plurality of pairs of first upsampling layers and group convolutional layers connected in sequence, and a second upsampling layer connected to the last of the group convolutional layers; The step of inputting the image sample to be segmented and the first image data into an upsampling decoder for upsampling and grouped convolution processing to obtain the second image data comprises: Inputting the image sample to be segmented and the first image data into the first upsampling layer for upsampling processing to obtain image upsampling data; Inputting the image upsampled data into a corresponding group convolution layer, performing group convolution processing through the group convolution layer, and obtaining second convolution data; Inputting the second convolution data and the first image data to the next first upsampling layer, and inputting the image upsampling data output by the next first upsampling layer to the corresponding group convolution layer, until all the first upsampling layers and group convolution layers of the upsampling decoder are traversed to obtain the target second convolution data output by the last group convolution layer of the upsampling decoder; The target second convolution data is input into the second upsampling layer for upsampling processing to obtain the second image data.
4. The tongue image segmentation method according to claim 2 or 3, characterized in that: The group convolution layer includes at least one group convolution module, and the group convolution module includes a group convolution processing submodule and a feedforward neural network module; the step of performing group convolution processing through the group convolution layer includes: Standardizing the data input to the group of convolutional layers to obtain standardized data; Inputting the standardized data into a group convolution processing submodule, performing group convolution and attention extraction processing through the group convolution processing submodule to obtain a first group of convolution attention data; fusing and normalizing the data input to the group of convolutional layers and the first group of convolutional attention data to obtain fused normalized data and first fused normalized data; Inputting the first fused normalized data into a feedforward neural network module for convolution and activation processing to obtain first convolution activation data; The first fused normalized data and the convolution activation data are fused to obtain the output of the group convolution module.
5. The tongue image segmentation method according to claim 4, characterized in that: The group convolution processing submodule includes a group convolution unit; the step of inputting the standardized data into the group convolution processing submodule, performing group convolution and attention extraction processing by the group convolution processing submodule, and obtaining a first group of convolution attention data includes: Performing convolution and activation processing on the standardized data to obtain second convolution activation data; Inputting the second convolution activation data into the group convolution unit, performing group convolution and attention extraction processing through the group convolution unit, and obtaining a second group of convolution attention data; The second set of convolutional attention data is convolved to obtain the first set of convolutional attention data.
6. The tongue image segmentation method according to claim 5, characterized in that: The group convolution unit includes sequentially connecting a first group convolution and a second group convolution; inputting the second convolution activation data into the group convolution unit, performing group convolution and attention extraction processing through the group convolution unit, and obtaining a second group of convolution attention data, including: Inputting the second convolution activation data into the first group of convolutions, determining the number of groups according to the number of input channels, performing group convolution using the number of groups, and obtaining a first group of convolution data; Performing depth separation convolution processing and depth separation dilated convolution processing on the first group of convolution data to obtain depth convolution data; Inputting the depth convolution data into the second group of convolutions, determining the number of groups according to the number of input channels, performing group convolution using the number of groups, and obtaining a second group of convolution data; Perform attention extraction processing on the second convolution activation data and the second group of convolution data to obtain the second group of convolution attention data.
7. The tongue image segmentation method according to claim 6, characterized in that: The step of determining the number of groups according to the number of input channels comprises: The receptive range of the group convolution is obtained through the following formula: When not an integer Among them, k is the receptive range of group convolution, C is the number of channels, R is the channel number scaling ratio, and γ is the preset operation parameter; The number of groups is obtained using the following formula: Among them, G is the number of groups.
8. A tongue image segmentation device, characterized in that: include: A sample model acquisition module, used to acquire a plurality of image samples to be segmented, a plurality of corresponding image segmentation result samples and an initial tongue image segmentation model; the initial tongue image segmentation model includes a downsampling encoder, an upsampling encoder and a fully connected layer; A first image data acquisition module, used for inputting the image sample to be segmented into the downsampling encoder for feature extraction and group convolution processing to obtain first image data; An initial image segmentation result acquisition module, used for inputting the image sample to be segmented and the first image data into an upsampling decoder for upsampling and group convolution processing to obtain second image data; Inputting the second image data into the fully connected layer to obtain an initial image segmentation result; A training module, used for training the model parameters of the initial tongue image segmentation model according to the plurality of image segmentation result samples and the plurality of initial image segmentation results, to obtain a trained target tongue image segmentation model; The tongue image segmentation result acquisition module is used to acquire the target image to be segmented; input the target image to be segmented into the target tongue image segmentation model to obtain the tongue image segmentation result in the image to be segmented.
9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the tongue image segmentation method according to any one of claims 1 to 7 are implemented.
10. A computer device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable by the processor, wherein the processor implements the steps of the tongue image segmentation method according to any one of claims 1 to 7 when executing the computer program.