Tongue image segmentation method, device, medium and equipment

Abstract

The present invention discloses a tongue image segmentation method, apparatus, medium, and equipment, relating to the technical field of tongue image segmentation. The dual U-net network is improved by adding a local spatiotemporal convolution module to the encoder of one of the U-net branches, and adding a global temporal attention mechanism module to the decoder of the U-net branch. This improvement is intended to address the technical problem that the existing technology cannot correctly express the changes in the biological structure of the tongue image in a time series, resulting in insufficient segmentation accuracy. By introducing local spatiotemporal convolution and global temporal attention mechanisms into the encoder and decoder structures of one of the branches of the dual Unet network, respectively, the dynamic changes of the tongue image in consecutive frames can be captured more accurately.

Description

Technical Field

[0001] The present invention relates to the technical field of tongue image segmentation, and in particular to a tongue image segmentation method, device, medium and equipment. Background Art

[0002] Tongue diagnosis is a core method in Traditional Chinese Medicine (TCM). Practice has shown that the tongue's condition can directly reflect a person's health. High-precision segmentation of the tongue from tongue images is crucial for automatically identifying potential illnesses using computer vision.

[0003] Current medical image segmentation methods often rely on single-frame tongue images or basic sequence processing techniques. However, dynamically changing biological structural features are difficult to accurately capture in a single tongue frame. Conventional methods are also inefficient and have limited accuracy when integrating information from multiple frames. While some systems attempt to integrate multi-frame information through simple frame stacking or averaging, these methods often fail to accurately represent the biological structural changes in the tongue image over time, resulting in insufficient segmentation accuracy. Summary of the Invention

[0004] Based on this, in order to solve the technical problems in the prior art, the present invention provides a tongue image segmentation method, device, medium and equipment.

[0005] The present invention provides a tongue image segmentation method, comprising:

[0006] An improved U-net branch is obtained by adding a local spatiotemporal convolution module to the encoder of one of the U-net branches in the dual U-net network and adding a global temporal attention mechanism module to the decoder of the U-net branch; the unimproved U-net branch in the dual U-net network is used as the first branch, and the improved U-net branch in the dual U-net network is used as the second branch to construct an improved dual U-net network including the first branch, the second branch and the segmentation layer.

[0007] Collect tongue image datasets to train the improved dual U-net network and obtain a tongue segmentation model; including:

[0008] Collect a tongue image dataset, which includes consecutive tongue image frames and the true segmentation label corresponding to each tongue image frame; stack the tongue image frames belonging to the same sample to form a multidimensional tensor;

[0009] Initialize the weight coefficient of the improved dual U-net network;

[0010] The multidimensional tensor is input into the first branch of the improved dual U-net network to output the initial tongue image spatial features. The multidimensional tensor and the initial tongue image spatial features are input into the second branch of the improved dual U-net network to output the tongue image temporal global decoding features. The tongue image temporal global decoding features are input into the segmentation layer to output the predicted segmentation label corresponding to each frame of the tongue image.

[0011] The loss function is used to calculate the loss value of the true segmentation label and the predicted segmentation label; the weight coefficient in the dual U-net network is iteratively improved according to the loss value, and the new sample is re-segmented according to the iterative weight coefficient; the loss function is:

[0012]

[0013]

[0014]

[0015] in, is the total loss function; is the spatial loss function corresponding to the first branch, n is the number of tongue image data frames to be segmented, is the predicted segmentation label of the tongue image in frame t, is the true segmentation label of the tongue image in frame t; is the adjustment parameter; is the time loss function corresponding to the second branch, is the predicted segmentation label of the tongue image of the n+1th frame predicted by the second branch, is the true segmentation label of the tongue image in the n+1th frame;

[0016] Repeat the iterative steps until the loss function converges or reaches the preset number of iterations to obtain the trained improved dual U-net network.

[0017] The tongue image data to be segmented is input into the tongue segmentation model. The first branch extracts spatial features of each tongue image frame in the tongue image data to be segmented, and obtains the initial tongue image spatial features of each tongue image frame containing the tongue's physiological structure, morphology, and size information. The initial tongue image spatial features are input into the second branch. The local spatiotemporal convolution module of the second branch encoder performs a local spatiotemporal convolution operation on the initial tongue image spatial features:

[0018]

[0019]

[0020] in, is the n+1 frame of tongue image data to be segmented;STCM is the spatiotemporal convolution operation, is the output of the spatiotemporal convolution operation; Conv3D It is a 3D local convolution operation; It is the output of the local spatiotemporal convolution operation; this operation locally encodes the initial tongue image spatial features along the time dimension to obtain the tongue image temporal local encoding features.

[0021] The tongue image temporal local encoding features are decoded through the global temporal attention mechanism module of the second branch decoder:

[0022]

[0023] in, SelfAttention Is a convolutional long short-term neural network or Transformer The attention operation in is the output of the attention operation, It is the input of the attention operation; this operation captures the global temporal correlation of the tongue image spatial features in the time dimension and obtains the tongue image temporal global decoding features; the tongue image temporal global decoding features are segmented through the segmentation layer to obtain the segmentation results of the tongue image data to be segmented.

[0024] The present invention provides a tongue image segmentation device, comprising:

[0025] A model construction module is configured to obtain an improved U-net branch by adding a local spatiotemporal convolution module to the encoder of one of the U-net branches in the dual U-net network and adding a global temporal attention mechanism module to the decoder of the U-net branch; the improved dual U-net network is constructed by using the unimproved U-net branch in the dual U-net network as the first branch and the improved U-net branch in the dual U-net network as the second branch, comprising the first branch, the second branch, and a segmentation layer;

[0026] The model training module is used to collect tongue image datasets to train the improved dual U-net network and obtain the tongue image segmentation model;

[0027] The tongue image segmentation module is used to input the tongue image data to be segmented into the tongue image segmentation model, extract the spatial features of each frame of the tongue image data to be segmented through the first branch, obtain the initial tongue image spatial features of each frame of the tongue image containing the physiological structure, morphology and size information of the tongue image, and input the initial tongue image spatial features into the second branch; perform local spatiotemporal convolution operation on the initial tongue image spatial features through the local spatiotemporal convolution module of the second branch encoder to locally encode the initial tongue image spatial features along the time dimension to obtain the tongue image temporal local encoding features; decode the tongue image temporal local encoding features through the global temporal attention mechanism module of the second branch decoder to capture the global temporal correlation of the tongue image spatial features in the time dimension to obtain the tongue image temporal global decoding features; segment the tongue image temporal global decoding features through the segmentation layer to obtain the segmentation result of the tongue image image data to be segmented.

[0028] The present invention provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the tongue image segmentation method is implemented.

[0029] The present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned tongue image segmentation method when executing the program.

[0030] At least one of the above technical solutions adopted by the present invention can achieve the following beneficial effects:

[0031] In the tongue image segmentation method provided by the present invention, local spatiotemporal convolution and global temporal attention mechanisms are respectively introduced into the encoder and decoder structures of one branch of the dual Unet network to capture more accurate dynamic changes of tongue images in consecutive frames. Specifically, the role of the local spatiotemporal convolution module is to extract the spatiotemporal features of local areas at a lower level of the network, thereby capturing the dynamic changes of tongue images in consecutive frames. The global temporal attention mechanism plays a role in the high-level network. It can pay attention to the temporal dependencies between frames in the entire sequence and strengthen such dependencies. Through this spatiotemporal convolution design, the Unet network can directly process continuous multi-frame image inputs and extract joint spatiotemporal features; this not only enables the model to effectively complete the segmentation task of the current frame, but also uses the segmentation information of historical frames to optimize the segmentation results of the current frame, thereby achieving a more coherent and smooth segmentation effect in the time series. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0033] Figure 1 A schematic flow chart of a tongue image segmentation method provided by the present invention;

[0034] Figure 2 A schematic diagram of a dual Unet network structure provided by the present invention;

[0035] Figure 3 A schematic diagram of a computer device for implementing a tongue image segmentation method provided by the present invention. DETAILED DESCRIPTION

[0036] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with specific embodiments of the present invention and corresponding drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of the present invention.

[0037] Existing technologies face multiple challenges when processing dynamic medical tongue images. First, single-frame tongue image processing cannot effectively capture changes in continuous physiological activities, such as the respiratory cycle or gastrointestinal motility. Second, although some systems attempt to integrate multi-frame information through simple frame stacking or averaging, this approach often fails to accurately represent biological structural changes in the time series, resulting in insufficient segmentation accuracy. To address this issue, the present invention introduces an attention mechanism operator model. Its superior ability to process temporal information enables the system to more accurately represent and segment the complex changes in dynamic tongue images. Furthermore, traditional multi-frame fusion techniques are often complex to implement in existing systems, require high computing resources, struggle to respond in real time, and lack interpretability of the processing results for dynamic tongue images. The present invention provides a segmentation system specifically designed for dynamic medical tongue images. Utilizing a dual Unet architecture and multi-frame fusion technology, it can effectively process and segment continuous tongue image data. This system is particularly suitable for medical scenarios requiring dynamic monitoring and real-time segmentation, such as tongue lesion detection and dynamic change analysis. It can provide doctors with more accurate diagnostic evidence and enhance the speed and quality of clinical decision-making. The present invention solves the problem of insufficient accuracy of traditional medical tongue image segmentation systems when processing dynamic tongue images by combining dual Unet structure, multi-frame fusion technology and attention mechanism operator network.

[0038] Example 1

[0039] Figure 1 The process of the tongue image segmentation method of this embodiment is shown in FIG. Figure 1 The method is described in detail and specifically comprises the following steps:

[0040] S1: By adding a local spatiotemporal convolution module to the encoder of one of the U-net branches in the dual U-net network and adding a global temporal attention mechanism module to the decoder of the U-net branch, an improved U-net branch is obtained; the unimproved U-net branch in the dual U-net network is used as the first branch, and the improved U-net branch in the dual U-net network is used as the second branch, an improved dual U-net network including the first branch, the second branch and the segmentation layer is constructed.

[0041] like Figure 2 As shown, a two-stream architecture is designed: the spatial stream, or the first branch, handles the traditional Unet spatial segmentation task. The temporal stream, or the second branch, captures temporal dependencies through spatiotemporal convolution and a temporal attention mechanism. Initially, a dual Unet network combined with an attention operator model processes tongue images from various medical imaging devices. These devices may include, but are not limited to, ultrasound, MRI, or CT scanners, which provide a range of dynamic or static medical tongue images. Before being fed into the system for deep learning processing, the tongue images undergo a series of preprocessing steps. These steps, including color correction and noise removal, aim to improve the overall quality of the tongue images and ensure the accuracy of subsequent analysis. By introducing an attention operator model into the first Unet module of the dual Unet network, the system achieves more efficient and accurate processing of time series data. Next, the processed frames are fed into the second Unet module, which specializes in multi-frame data and uses temporal analysis techniques to learn and extract dynamic relationships and change patterns between consecutive frames. During this process, the design of the attention mechanism operator allows the system to not only identify static features but also capture subtle characteristics such as the dynamic changes in tongue images under different lighting conditions. Ultimately, the data, which combines spatial and temporal features, is passed through a segmenter for final disease segmentation, providing accurate diagnostic results.

[0042] S101: First branch.

[0043] The first Unet module specifically processes a single-frame tongue image, extracting detailed spatial features from it. This step is crucial because it accurately identifies and locates key medical objects in the tongue image, such as tumors or other pathological structures. Through this efficient feature extraction technique, the system can quickly identify key medical objects in the tongue image, providing the necessary foundation for subsequent multi-frame fusion analysis. This Unet module is specifically designed to process the spatial dimension of the tongue image, leveraging its deep convolutional network architecture to extract key spatial features, such as the morphology and size of various physiological structures in the medical tongue image. The output of this stage is a highly refined feature map that provides the foundation for capturing detailed physiological information within each frame.

[0044] The first Unet module is tasked with processing these preprocessed single-frame tongue images. It utilizes a multi-layer convolutional neural network structure within a deep learning network to analyze each tongue frame and identify key medical tongue objects, such as tumors or other lesions. This process involves using complex tongue recognition algorithms to precisely locate these objects and quickly generate information about their location and basic properties. The output of this stage is crucial, providing the necessary input data for the next step, tongue segmentation. These initially identified tongue regions are then passed to the system's second Unet module.

[0045] S102: The second branch.

[0046] The second Unet module processes multi-frame tongue image data, leveraging its multi-frame fusion technology and attention mechanism operator structure to analyze dynamic changes within the tongue image sequence. This module's design enables the system to effectively capture and analyze changes in time series, such as tissue growth or lesion progression, which is crucial for early diagnosis and treatment planning. The attention mechanism operator not only enhances the system's ability to capture temporal information but also improves segmentation accuracy, enabling the system to produce more reliable diagnostic results.

[0047] The second Unet module further processes these identified tongue regions. It focuses on tongue segmentation, using advanced tongue processing techniques to meticulously delineate the precise edges and internal details of each identified object. This involves applying a multi-layered tongue segmentation algorithm to ensure the precise segmentation and mapping of the boundaries of each lesion or tumor, providing a precise tongue image foundation for in-depth medical analysis and diagnosis. This segmentation result is crucial because it provides detailed tongue features and morphological information, which serve as the foundation for subsequent data analysis modules to determine pathological status. These processed single-frame tongue features are fed into the second Unet module, which is similar in architecture to the first but specifically optimized for processing multiple frames in a sequence. Using advanced temporal analysis techniques, such as recurrent neural networks or attention-based operator networks, it learns the dynamic changes from one frame to the next, effectively extracting the variations in tongue features across multiple frames. This fusion approach allows the system to not only identify static tongue features but also capture dynamic changes, such as subtle variations in tongue appearance under varying conditions. Features of the time dimension are extracted and synthesized from continuous frames, and then a comprehensive feature set containing temporal dynamic information is generated, which provides more comprehensive data support for the final tongue segmentation. These comprehensive features that combine spatial and temporal dimensions are fed into the segmenter, which performs disease segmentation according to the pre-trained model and outputs accurate diagnosis results. Local spatiotemporal convolution and global temporal attention mechanism are introduced into the encoder and decoder structures of Unet respectively. Local spatiotemporal convolution is used to capture local spatiotemporal features at a low level, while global temporal attention mechanism is used to capture global temporal correlations at a high level. In order to enable Unet to process multiple time frames at the same time, a spatiotemporal convolution module (SpatiotemporalConvolution Module, STCM) is introduced to perform spatiotemporal convolution operations in the encoder part of Unet. This module not only performs convolution in space, but also extracts features in the temporal dimension. Specifically, in each encoder layer of Unet, a local spatiotemporal convolution module is inserted, and 3D convolution operations are used to extract local spatiotemporal features. For the input feature map , the local spatiotemporal convolution can be expressed as:

[0048] The input image sequence is , these frames can be stacked into a tensor with shape (batch_size, n_frames, height, width, channels). 3D convolution can be performed through the spatiotemporal convolution module (STCM) to capture temporal and spatial features:

[0049] ;

[0050] ;

[0051] ;

[0052] in, It is l The output of the local spatiotemporal convolution operation at the layer, is a 3D local convolution operation, It is l The input features of the layer encoder, is the output of the spatiotemporal convolution operation, is the n+1 frame of tongue image data to be segmented; It is a spatiotemporal convolution operation.

[0053] Global Temporal Attention Mechanism: In the decoder part of Unet, the Transformer or ConvLSTM self-attention mechanism is used to capture the global temporal dependency, further perform temporal modeling on the spatiotemporal features, and predict the segmentation result of the next frame. , the formula is:

[0054] ;

[0055] ;

[0056] in, is a convolutional long short-term memory network, Is the segmentation head. For the decoder input feature map , the global temporal attention mechanism can be expressed as:

[0057] ;

[0058] in, It is the global feature map after the temporal attention mechanism. SelfAttention Is a convolutional long short-term neural network or Transformer Attention operation in .

[0059] S2: Collect tongue image datasets to train the improved dual U-net network and obtain the tongue segmentation model.

[0060] A tongue image dataset is collected, which includes continuous tongue image frames and the true segmentation label corresponding to each tongue image frame; each tongue image frame belonging to the same sample is stacked to form a multidimensional tensor with a shape of (batch_size, n_frames, height, width, channels).

[0061] Initialize the weight coefficients of the improved dual U-net network. Input the multidimensional tensor into the first branch of the improved dual U-net network, and perform 3D convolution through the spatiotemporal convolution module (STCM) to capture temporal and spatial features. The formula is expressed as:

[0062] ;

[0063] in, It is a spatiotemporal feature map. The spatiotemporal feature is input to the subsequent part of Unet for segmentation to obtain the segmentation result of each frame. .

[0064] The other time series branch further models the spatiotemporal features through the Transformer or ConvLSTM module to predict the segmentation result of the next frame. , the formula is:

[0065] ;

[0066] .

[0067] Spatial loss: segmentation result of the current frame and the true label Calculate the loss:

[0068] ;

[0069] in, is the spatial loss function corresponding to the first branch, n is the number of tongue image data frames to be segmented, is the predicted segmentation label of the tongue image in frame t, is the true segmentation label of the tongue image in frame t.

[0070] Temporal loss: predicting the next frame segmentation result and the true label Calculate the loss:

[0071] ;

[0072] in, is the time loss function corresponding to the second branch, is the predicted segmentation label of the tongue image of the n+1th frame predicted by the second branch, is the true segmentation label of the tongue image in frame n+1.

[0073] The final loss is:

[0074] ;

[0075] in, is a weight parameter used to adjust the weights of spatial loss and temporal loss.

[0076] Through spatiotemporal convolution, UNet can directly process inputs from multiple frames and extract joint spatiotemporal features. Through a multi-task learning framework, the model can not only handle the segmentation task of the current frame, but also predict the segmentation results of the next frame, thereby improving temporal consistency.

[0077] S3: Input the tongue image data to be segmented into the tongue image segmentation model, extract the spatial features of each frame of the tongue image in the tongue image data to be segmented through the first branch, obtain the initial tongue image spatial features of each frame of the tongue image containing the physiological structure, morphology and size information of the tongue image, and input the initial tongue image spatial features into the second branch; perform local spatiotemporal convolution operation on the initial tongue image spatial features through the local spatiotemporal convolution module of the second branch encoder to locally encode the initial tongue image spatial features along the time dimension to obtain the tongue image temporal local encoding features; decode the tongue image temporal local encoding features through the global temporal attention mechanism module of the second branch decoder to capture the global temporal correlation of the tongue image spatial features in the time dimension to obtain the tongue image temporal global decoding features; segment the tongue image temporal global decoding features through the segmentation layer to obtain the segmentation result of the tongue image image data to be segmented.

[0078] Furthermore, to enhance the system's operability and the transparency of the diagnostic process, the present invention provides a user-friendly interface and detailed diagnostic process visualization tools. This allows medical professionals to easily operate the system and gain a deep understanding of the AI's decision-making logic. These tools allow medical professionals to intuitively visualize each step from tongue image acquisition to final diagnosis, enabling them to better evaluate the system's output and make informed clinical decisions. This system design, based on dual Unet and multi-frame fusion, leverages the latest advances in deep learning and tongue image processing technology, providing an innovative and effective method for dynamic medical tongue image segmentation and analysis. This system significantly enhances the automation and intelligence of medical tongue image analysis, significantly improving diagnostic speed and accuracy, thereby optimizing clinical workflows and enhancing patient outcomes. The segmented tongue image data is then transmitted to a data analysis module. This module combines the patient's historical medical records with tongue image data processed in real time by the Unet module. Here, a deep learning algorithm is used to comprehensively analyze each identified and segmented object, taking into account the detailed characteristics of the tongue image and the patient's clinical history, to infer its potential pathological characteristics and conditions. This analysis process utilizes the latest medical tongue analysis technologies, including but not limited to AI-based tongue image recognition and pathology prediction models, which are capable of extracting key biomarkers and disease indicators from complex medical tongue images.

[0079] Ultimately, all of this analysis data and results are systematically integrated into a comprehensive diagnostic report. This report not only details every step and result of the tongue analysis, but also includes the specific diagnostic recommendations generated based on this data and their scientific basis. The data and recommendations in this report are derived through a comprehensive evaluation process using deep learning algorithms and attention-based operator models, aiming to provide doctors with more accurate diagnostic information and treatment recommendations. Through this comprehensive, highly automated process, the system not only significantly improves the efficiency and accuracy of medical diagnoses, but also enhances the quality and responsiveness of medical services through its advanced tongue image processing and analysis technologies. Furthermore, the high degree of automation in this process reduces the workload of doctors, allowing them to focus more on patient care rather than tedious data processing. This transformation from raw medical images to detailed diagnostic reports not only demonstrates the power of modern medical technology but also serves as a cutting-edge example of the application of deep learning and artificial intelligence in medicine.

[0080] This invention solves the problem of insufficient accuracy of traditional medical tongue image segmentation systems when processing dynamic tongue images by combining dual Unet structures, multi-frame fusion technology and attention mechanism operator networks. The advantage of this technical solution is that it can comprehensively utilize the spatial and temporal information of tongue images, significantly improving the sensitivity and accuracy of monitoring and segmentation of dynamic physiological processes. In addition, by combining the temporal information processing capabilities of the attention mechanism operator, this system can significantly improve the segmentation accuracy when processing dynamic tongue image data, making it suitable for real-time analysis and processing of complex medical scenarios.

[0081] Beyond the current implementation, future developments of this technical solution could include integrating additional relevant health data (such as patient symptom records) with tongue analysis results to provide more comprehensive diagnostic support. Furthermore, with the advancement of machine learning technology, the introduction of more advanced deep learning models could be considered to further enhance segmentation performance. Furthermore, the system's algorithm could be adapted to accommodate tongue data generated by different types of medical tongue imaging devices, enhancing its applicability and flexibility.

[0082] The above is a tongue image segmentation method provided by one or more embodiments of the present invention. Based on the same idea, the present invention also provides a corresponding method as follows: Figure 3 The tongue image segmentation device shown includes:

[0083] The model construction module is used to obtain an improved U-net branch by adding a local spatiotemporal convolution module to the encoder of one of the U-net branches in the dual U-net network and adding a global temporal attention mechanism module to the decoder of the U-net branch; the unimproved U-net branch in the dual U-net network is used as the first branch, and the improved U-net branch in the dual U-net network is used as the second branch to construct an improved dual U-net network including the first branch, the second branch and the segmentation layer.

[0084] The model training module is used to collect tongue image datasets to train the improved dual U-net network and obtain the tongue image segmentation model.

[0085] The tongue image segmentation module is used to input the tongue image data to be segmented into the tongue image segmentation model, extract the spatial features of each frame of the tongue image data to be segmented through the first branch, obtain the initial tongue image spatial features of each frame of the tongue image containing the physiological structure, morphology and size information of the tongue image, and input the initial tongue image spatial features into the second branch; perform local spatiotemporal convolution operation on the initial tongue image spatial features through the local spatiotemporal convolution module of the second branch encoder to locally encode the initial tongue image spatial features along the time dimension to obtain the tongue image temporal local encoding features; decode the tongue image temporal local encoding features through the global temporal attention mechanism module of the second branch decoder to capture the global temporal correlation of the tongue image spatial features in the time dimension to obtain the tongue image temporal global decoding features; segment the tongue image temporal global decoding features through the segmentation layer to obtain the segmentation result of the tongue image image data to be segmented.

[0086] The specific limitations of the tongue image segmentation device can be found in the limitations of the tongue image segmentation method described above and will not be further elaborated here. Each module in the tongue image segmentation device can be implemented in whole or in part via software, hardware, or a combination thereof. Each of these modules can be embedded in or independent of a processor in a computer device in hardware form, or stored in a computer device memory in software form, allowing the processor to call and execute the corresponding operations of each module.

[0087] The present invention also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 The tongue segmentation method provided.

[0088] The present invention also provides a computer device structure. At the hardware level, the computer device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory. Of course, it may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 The tongue segmentation method provided.

[0089] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware using a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes in the above-described method embodiments. Any reference to memory, storage, database, or other media used in the various embodiments provided herein may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0090] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of the present invention.

Claims

1. A tongue image segmentation method, characterized in that: include: By adding a local spatiotemporal convolution module to the encoder of one of the U-net branches in the dual U-net network and a global temporal attention mechanism module to the decoder of the U-net branch, an improved U-net branch is obtained; The unimproved U-net branch in the dual U-net network is used as the first branch, and the improved U-net branch in the dual U-net network is used as the second branch, and an improved dual U-net network including the first branch, the second branch and the segmentation layer is constructed; The tongue image dataset was collected to train the improved dual U-net network and obtain the tongue image segmentation model; The tongue image data to be segmented is input into the tongue image segmentation model. The spatial features of each frame of the tongue image data to be segmented are extracted through the first branch to obtain the initial tongue image spatial features of each frame of the tongue image, which contain information about the physiological structure, morphology and size of the tongue image. The initial tongue image spatial features are input into the second branch. The local spatiotemporal convolution module of the second branch encoder performs a local spatiotemporal convolution operation on the initial tongue image spatial features to locally encode the initial tongue image spatial features along the time dimension to obtain the tongue image temporal local encoding features. The global temporal attention mechanism module of the second branch decoder decodes the tongue image temporal local encoding features to capture the global temporal correlation of the tongue image spatial features in the time dimension to obtain the tongue image temporal global decoding features. The tongue image temporal global decoding features are segmented through the segmentation layer to obtain the segmentation results of the tongue image data to be segmented.

2. The tongue image segmentation method according to claim 1, wherein: The local spatiotemporal convolution module of the second branch encoder performs a local spatiotemporal convolution operation on the initial tongue image spatial features, specifically including: in, is the n+1 frame of tongue image data to be segmented; STCM is the spatiotemporal convolution operation, is the output of the spatiotemporal convolution operation; Conv3D It is a 3D local convolution operation; is the output of the local spatiotemporal convolution operation.

3. The tongue image segmentation method according to claim 2, wherein: The global temporal attention mechanism module of the second branch decoder is used to decode the tongue image temporal local encoding features, specifically including: in, SelfAttention Is a convolutional long short-term neural network or Transformer The attention operation in is the output of the attention operation, is the input of the attention operation.

4. The tongue image segmentation method according to claim 1, wherein: The tongue image dataset is collected to train the improved dual U-net network, specifically including: Collect a tongue image dataset, which includes consecutive tongue image frames and the true segmentation label corresponding to each tongue image frame; stack the tongue image frames belonging to the same sample to form a multidimensional tensor; Initialize the weight coefficient of the improved dual U-net network; The multidimensional tensor is input into the first branch of the improved dual U-net network to output the initial tongue image spatial features. The multidimensional tensor and the initial tongue image spatial features are input into the second branch of the improved dual U-net network to output the tongue image temporal global decoding features. The tongue image temporal global decoding features are input into the segmentation layer to output the predicted segmentation label corresponding to each frame of the tongue image. The loss function is used to calculate the loss value of the true segmentation label and the predicted segmentation label; the weight coefficient in the dual U-net network is iteratively improved according to the loss value, and the new sample is re-segmented according to the iterative weight coefficient; Repeat the iterative steps until the loss function converges or reaches the preset number of iterations to obtain the trained improved dual U-net network.

5. The tongue image segmentation method according to claim 4, wherein: The loss function specifically includes: in, is the total loss function; is the spatial loss function corresponding to the first branch, n is the number of tongue image data frames to be segmented, is the predicted segmentation label of the tongue image in frame t, is the true segmentation label of the tongue image in frame t; is the adjustment parameter; is the time loss function corresponding to the second branch, is the predicted segmentation label of the tongue image of the n+1th frame predicted by the second branch, is the true segmentation label of the tongue image in frame n+1.

6. A tongue image segmentation device, characterized in that: include: A model building module for obtaining an improved U-net branch by adding a local spatiotemporal convolution module to the encoder of one of the U-net branches in the dual U-net network and a global temporal attention mechanism module to the decoder of the U-net branch; The unimproved U-net branch in the dual U-net network is used as the first branch, and the improved U-net branch in the dual U-net network is used as the second branch, and an improved dual U-net network including the first branch, the second branch and the segmentation layer is constructed; The model training module is used to collect tongue image datasets to train the improved dual U-net network and obtain the tongue image segmentation model; The tongue image segmentation module is used to input the tongue image data to be segmented into the tongue image segmentation model, extract spatial features from each frame of the tongue image data to be segmented through the first branch, obtain initial tongue image spatial features for each frame of the tongue image containing information about the physiological structure, morphology, and size of the tongue image, and input the initial tongue image spatial features into the second branch; the local spatiotemporal convolution module of the second branch encoder performs a local spatiotemporal convolution operation on the initial tongue image spatial features to locally encode the initial tongue image spatial features along the time dimension to obtain tongue image temporal local encoding features; the global temporal attention mechanism module of the second branch decoder decodes the tongue image temporal local encoding features to capture the global temporal correlation of the tongue image spatial features in the time dimension to obtain tongue image temporal global decoding features; The tongue image temporal global decoding features are segmented through the segmentation layer to obtain the segmentation results of the tongue image data to be segmented.

7. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

8. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method according to any one of claims 1 to 5 is implemented.

Images