A semi-supervised medical image segmentation method with eye-movement guided hybrid data enhancement

Abstract

The present invention discloses a semi-supervised medical image segmentation method guided by eye movement and enhanced by hybrid data. The method comprises the following steps: collecting the gaze position information of a doctor when reading a medical image by an eye tracking device; training the model by using a teacher-student network model and adopting a two-stage training method; adopting GazeMix as a data enhancement strategy in training, extracting the gaze area in a labeled or unlabeled image, pasting it onto the background of other unlabeled images, and generating a hybrid image data pair; pre-training a teacher network in combination with the GazeMix data enhancement strategy in the first stage; training a student network in the second stage, using GazeMix to generate diversified data, and pre-training the teacher network for pseudo-label generation; adopting multi-scale gaze perception, extracting model perception information at different scales and using it for perception loss calculation; an optimizer iteratively optimizes the student network parameters, and simultaneously uses an exponential moving average method to update the teacher network.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image processing, and in particular relates to a semi-supervised medical image segmentation method guided by eye movement and enhanced by hybrid data. Background Art

[0002] In the field of medical image segmentation, automatic segmentation of multiple organs or lesion areas has important clinical value and is widely used in disease diagnosis, surgical planning, and treatment effect evaluation. However, due to the professionalism and high cost required to annotate medical images, training high-precision segmentation models is often limited by insufficient annotated data. Therefore, how to use limited annotated data to train network models has become a key point in this type of research.

[0003] In recent years, the application of semi-supervised learning in medical image segmentation has gradually increased, mainly through pseudo-label generation, data enhancement and consistency constraints to improve the model's efficiency in utilizing unlabeled data. However, the existing pseudo-label generation methods are difficult to guarantee accuracy when generating labels on unlabeled data; consistency constraint methods often rely on model structure or data transformation, and are difficult to fully adapt to complex medical image backgrounds. In addition, although data enhancement methods (such as rotation, cropping, flipping, etc.) can increase the diversity of data sets, they often have limited effects in medical images because such enhancements lack accurate focus on the target area.

[0004] At the same time, radiologists naturally focus on certain key areas when viewing medical images. This gaze pattern can help them quickly identify lesions or organs in complex backgrounds. Through eye tracking technology, we can capture the doctor's gaze position and generate a doctor's perception heat map to reflect the doctor's focus when reading the image. Studies have shown that the human visual system has strong adaptability under different backgrounds and can quickly focus on the target area in complex images, which is exactly the ability that current segmentation models lack. If the doctor's gaze information can be introduced into the model, the model will be able to better learn to focus on key areas, thereby improving segmentation accuracy. Existing studies have attempted to introduce human visual attention into deep learning models, such as using doctors' gaze information to improve model performance in knee arthritis diagnosis, brain MR image segmentation, and ultrasound image recognition. However, existing methods usually only input gaze information into the network as an additional feature, or simply as reference data, and fail to fully integrate human visual information in the network architecture. Therefore, current medical image segmentation models still lack deep guidance from human visual perception and fail to fully utilize human eye gaze information to improve the model's focusing ability and segmentation accuracy.

[0005] Humans have a relatively stable perception of key areas, which is reflected in the fact that humans can usually quickly lock and focus on the target area under different backgrounds for the same foreground target. In contrast, existing automatic segmentation models lack a similar human attention mechanism, which makes it difficult for the model to focus on the correct target area when processing medical images with complex backgrounds and variable organ shapes. Especially in semi-supervised learning scenarios, the scarcity of labeled data further exacerbates the learning difficulty of the model, resulting in poor generalization ability. Summary of the invention

[0006] Purpose of the invention: The purpose of the present invention is to provide a semi-supervised medical image segmentation method with eye-movement guided hybrid data enhancement, which uses an eye tracker to collect eye gaze position information during the reading process of radiologists and generate doctor perception. The heat map reflects the area that the doctor focuses on in the image, providing key visual guidance information for subsequent model training. Semi-supervised learning uses a teacher-student network architecture for model training and designs a data enhancement strategy called GazeMix based on human eye gaze information. GazeMix extracts gaze areas from labeled and unlabeled images, copies and pastes them into the background of other unlabeled images, and generates new mixed image pairs. This method reduces the dependence on a large amount of labeled data, enabling the model to better understand the context of the target. At the same time, a multi-scale gaze perception module (MGP) is introduced to extract and fuse the perception ability of the model at different scales. Pseudo-labels are generated through a pre-trained teacher network as a supervisory signal for unlabeled data. Combined with the real labels of a small amount of labeled data, a weighted loss function combination is adopted to comprehensively consider the perception loss and segmentation loss to ensure the balance between the model's focus on key areas and the overall segmentation effect.

[0007] To achieve the above object, the present invention provides an eye movement guided hybrid data enhanced semi-supervised medical image segmentation method, comprising the following steps:

[0008] S1: Collecting the gaze information of radiologists when reading medical images through eye tracking equipment;

[0009] S2: The gaze information collected in step S1 is converted into doctor perception, and the eye movement position matrix is ​​constructed based on the original eye movement gaze information. , use Gaussian blur to Processing to get doctor's perception ;

[0010] S3: Based on the doctor's perception in step S2 , designing the GazeMix data augmentation method, which involves extracting the image parts containing the doctor’s gaze area from the annotated and unannotated images and pasting them into the background of other unannotated images to generate new image pairs to expand the diversity and semantic consistency of the dataset;

[0011] S4: The GazeMix data augmentation method in step S3 is used for the teacher-student network model. GazeMix extracts the foreground from the labeled image and pastes it to the background of other unlabeled images to generate a mixed image. The mixed image and the original labeled image data are used to construct a dataset. Pre-training on the constructed dataset;

[0012] S5: Generate a mixed image based on GazeMix in steps S2 and S3 to perform the second stage student network model GazeMix generates mixed image data and inputs them into the pre-trained teacher model. and student models , teacher model The output of is used as pseudo label;

[0013] S6: During the student network model training process in step S5, a multi-scale gaze perception module MGP is designed to extract and fuse the network perception at different scales. And perform perceptual loss calculation to enhance the model's feature extraction ability in complex medical images;

[0014] S7: According to step S2, S6 , Constructing the perceptual loss function , constrain the alignment of the network’s perception and the doctor’s gaze area, and improve the model’s ability to focus on key areas in complex backgrounds by gradually optimizing the network’s perception, thereby improving the accuracy of medical image segmentation;

[0015] S8: The perceptual loss function in step S7 With segmentation loss Perform weighted calculations and use the optimizer to optimize the student model Update parameters;

[0016] S9: Update the teacher model using the exponential moving average EMA method .

[0017] Furthermore, the doctor perception conversion of the gaze information in step S2 specifically includes the construction of a position matrix and the generation of doctor perception by Gaussian blur processing; first, an initial position matrix is ​​constructed according to the size of the reading image. , where the position and value of each element represent the specific pixel distribution of the doctor's gaze. Next, Gaussian blur processing is used to smooth the data distribution in the matrix and reduce noise and interference. After Gaussian blur processing, the position matrix is ​​converted into the doctor's perception , The grayscale value represents the focus of attention.

[0018] Furthermore, the GazeMix data enhancement strategy in step S3 firstly performs a search in the labeled image or the unlabeled image according to the doctor's perception obtained in step S2. The doctor's gaze area is cropped out. The grayscale value in the extracted area must be greater than a certain threshold, which is set to 0.1 here. The area contains the organ or lesion information that the doctor is concerned about in the image. Then these extracted gaze areas are pasted to the corresponding positions of other unlabeled images to generate new mixed image pairs.

[0019] Furthermore, the semi-supervised training strategy in step S4 adopts a teacher-student model for two-stage training. In the first stage, the teacher model is pre-trained using labeled data. Specifically, GazeMix is ​​used to extract the foreground from the labeled image and paste it into the background of other unlabeled images to generate a mixed image. The labels of the original labeled image are mixed in the same way to generate a mixed label as supervision. The generated mixed image and the original image constitute a new data set. The teacher model is trained on the new data set. Perform pre-training.

[0020] Furthermore, the retraining of the student network model in step S5 includes using GazeMix to perform data augmentation operations on the labeled and unlabeled data to generate two types of mixed image pairs, wherein the labeled data is used as the foreground and the unlabeled data is used as the background, a cropping mask is obtained according to the corresponding doctor perception of the foreground image, and the gaze area in the labeled data is extracted and pasted onto the background of the unlabeled data. Specifically, the labeled data is used As foreground, unlabeled data Generate labeled mixed data as background , and the same GazeMix operation is performed on the labels of the labeled data. The background value of the label is unified to 0, generating a labeled sample pair with labeled foreground and unlabeled background , for unlabeled data as foreground, use unlabeled data As foreground, unlabeled data Generate unlabeled mixed data images as background ; Subsequently, the GazeMix-enhanced data is input into the student network and the teacher network, where the teacher network uses a separate Input, output the network As The pseudo labels are used for supervision of the student network. The teacher network does not use gradient descent to update parameters. In the training process of the two-stage student network, the sample pairs are divided into labeled sample pairs. and pseudo-labeled sample pairs Used to supervise the training of student networks on labeled and unlabeled data.

[0021] Furthermore, the multi-scale gaze perception module (MGP) in step 6 includes a channel attention module and a spatial attention module. The MGP module is embedded in front of the segmentation head layer of the model segmentation output, and uses image upsampling features. As input, The shape and size of , extracting network perception capabilities through the channel attention module and the spatial attention module in turn; the channel attention module directly adopts As input, the feature map is first compressed using a global average pooling operation of Dimension, generate Global eigenvector of shape size , and then use the MLP layer to transform the feature vector Do the calculation and generate the attention weight on the channel dimension through the sigmoid activation function , focus on the channel Multiply the original features to get the channel-aware enhanced feature map ; Use convolution kernels of different sizes (1x1, 3x3, and 7x7) to convolution the feature map Perform spatial attention perception feature extraction to capture the fine-grained and overall spatial features at different scales. Splice the features of three different scales according to channels, adjust the number of channels through a 1x1 convolution layer, and achieve feature fusion at different scales. Finally, constrain the weight of the image pixel level to the range of 0-1 through the sigmoid activation function, and obtain the spatial perception attention weight. As network perception for doctor perception Do the loss calculation and Perform multiplication operations to obtain feature maps after multiple enhanced perception in space and channels Used as input for the last layer of convolution and outputs the final segmentation result.

[0022] Furthermore, the perceptual loss function described in step S7 is Network perception using mean square error contrast model Perception with doctors , so as to gradually align the model's attention to the key areas that doctors pay attention to. The perceptual loss expression is:

[0023]

[0024] in, Represents the value of each pixel in the doctor's perception, Represents the value of each pixel in network perception, Represents the number of pixels perceived by all doctors.

[0025] Furthermore, when the model is trained in step S8, the pseudo-labels, real labels and network perception are used in combination to construct a loss function, including segmentation loss, which is composed of Dice loss and cross entropy loss weighted, and the segmentation result output by the model is compared with the real label and pseudo-label of the labeled data, and the perception loss is compared with the perception area of ​​the model and the human gaze area through the mean square error MSE. The comprehensive loss weighted result expression is:

[0026]

[0027] in, is the cross entropy loss, is the Dice loss, Represents the segmentation result predicted by the network, represents the true label of the image or the pseudo label generated by the teacher network, Represents the segmentation results of different categories. Represents the true labels of different categories or the pseudo labels generated by the teacher network, Indicates the number of pixels in the real label. Indicates the number of categories of true labels, , , , is the weighting coefficient, indicating the weight ratio, and the sizes are 0.5, 0.5, and 0.1 respectively.

[0028] Furthermore, in step S9, the teacher model is updated using the exponential moving average (EMA) method. , in each training step, the teacher network Exponential moving average (EMA) updates are performed to gradually optimize the teacher network and maintain a stable pseudo-label generation capability. The teacher model parameter update expression is:

[0029]

[0030] in, and Represent the model parameters of the teacher network and the student network respectively, It is the smoothing coefficient, which is used to control the decay speed of the old parameters and its value is 0.99.

[0031] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0032] 1. The present invention makes full use of the characteristics of easy acquisition and accurate information of eye movement information and the current situation of less labeled data in medical image segmentation, and proposes a semi-supervised medical image segmentation method guided by eye movement and enhanced by hybrid data;

[0033] 2. The present invention uses the teacher-student network architecture as the basic architecture for semi-supervised learning training, and introduces human eye gaze information to guide the model to focus on key areas. By combining pseudo labels and real labels, dual sample pairs are constructed for collaborative learning of teacher and student networks, further enhancing the learning ability and segmentation accuracy of the model on unlabeled data;

[0034] 3. This paper uses a new data enhancement strategy, GazeMix, to combine the gaze areas in the labeled and unlabeled images to generate new image pairs. The GazeMix strategy not only expands the diversity of the dataset, but also ensures the semantic consistency of the foreground and background, which helps the model learn the changes of the target area under different backgrounds;

[0035] 4. The present invention uses a multi-scale gaze perception module to extract model perception and enable the model to fuse features at different spatial scales and channel dimensions, thereby improving the ability to recognize details and overall structures in complex medical images;

[0036] 5. The present invention uses the gaze loss function to align the model's attention with the human eye's gaze area, thereby enhancing the model's ability to focus on key areas and improving the segmentation effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A schematic diagram of the topological structure of the semi-supervised medical image segmentation framework provided by the present invention.

[0038] Figure 2 A schematic diagram of the topological structure of the eye movement guided hybrid data enhancement strategy GazeMix provided by the present invention,

[0039] Figure 3 A schematic diagram of the topological structure of the multi-scale gaze perception module provided by the present invention,

[0040] Figure 4 A schematic diagram of the process of the eye movement guided hybrid data enhanced semi-supervised medical image segmentation method provided by the present invention,

[0041] Figure 5 This is a comparison chart of the segmentation results of the test image on the ACDC dataset.

[0042] Figure 6 The figure shows the comparison of the segmentation results of the test image on the CAMUS dataset. DETAILED DESCRIPTION

[0043] The present invention is further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0044] Embodiment: Eye gaze information is easy to obtain and accurate. In the field of medical image segmentation where labeled data is scarce, using the eye gaze information of radiologists when reading medical images combined with semi-supervised learning and using a small amount of labeled data to train models has great value and potential. However, existing segmentation methods that use eye gaze information focus on classification tasks and do not fully utilize the rich information contained in gaze heat maps in segmentation tasks. Therefore, this method uses a teacher-student network model as the basic architecture of semi-supervised learning, introduces human eye gaze information to guide the model to focus on key areas, and constructs dual sample pairs through the combination of pseudo labels and real labels for collaborative learning of teacher and student networks; A relatively large difficulty in semi-supervised medical image segmentation is how to effectively use a small amount of labeled data and a large amount of unlabeled data to train a reliable network model. This method uses a new data enhancement strategy GazeMix to combine the gaze areas in labeled and unlabeled images to generate new image pairs. The GazeMix strategy not only expands the diversity of the data set, but also ensures the semantic consistency of the foreground and background, which helps the model learn the changes of the target area under different backgrounds; the present invention uses weighted composite loss to optimize network parameters. The gaze loss function aligns the model's attention with the human eye gaze area, thereby enhancing the model's focusing ability on key areas and improving the segmentation effect. The segmentation loss combines the Dice loss with the cross entropy loss to optimize details and pixel classification results; the training process adopts a two-stage training mode. In the first stage, the labels on the labeled data are used to construct a data set to pre-train the teacher network. In the second stage, the results of the pre-trained teacher network predicting the unlabeled data are used as pseudo labels, which are combined with the true labels to construct the upstroke sample pairs for re-training the student network. Both stages of training use the Adam optimizer to adjust the parameters of the model. In the second stage, the teacher network no longer automatically updates the parameters, but is updated through the exponential moving average (EMA).

[0045] like Figure 4 As shown, it is a flow chart of the present invention, a semi-supervised medical image segmentation method guided by eye movement and enhanced by hybrid data, comprising the following steps:

[0046] S1: Collecting the gaze information of radiologists when reading medical images through eye tracking equipment;

[0047] The collected data sets are ACDC and CAMUS data sets. An eye tracker is used to collect and save the eye movement data of radiologists when they read single images in the two data sets.

[0048] S2: The gaze information collected in step S1 is converted into doctor perception, and the eye movement position matrix is ​​constructed based on the original eye movement gaze information. , use Gaussian blur to Processing to get doctor's perception ;

[0049] The doctor-perceived conversion of gaze information specifically includes the construction of a position matrix and Gaussian blur processing, so as to convert the original gaze data into a heat map that can clearly represent the doctor's focus area. First, the initial position matrix is ​​constructed according to the size of the original corresponding image. The position and value of each element of the position matrix directly reflects the specific pixel distribution of the doctor's gaze on the image. Next, Gaussian blur processing is applied to the position matrix to smooth the data distribution in the matrix and reduce noise and unnecessary interference. After Gaussian blur processing, the position matrix is ​​transformed into the doctor's perception , The concentration of gaze is represented by a grayscale image. The grayscale image uses grayscale values ​​to represent the concentration of the doctor's gaze, where the area with higher grayscale values ​​represents a higher intensity of the doctor's attention. The grayscale image can concisely and intuitively reflect the doctor's gaze distribution in the image.

[0050] S3: Based on the doctor's perception in step S2 , designing the GazeMix data augmentation method, which involves extracting the image parts containing the doctor’s gaze area (foreground) from the annotated and unannotated images and pasting them into the background of other unannotated images to generate new image pairs to expand the diversity and semantic consistency of the dataset;

[0051] The GazeMix data augmentation strategy generates new mixed image pairs by combining the gaze information in the labeled or unlabeled images to enrich the diversity of the data and strengthen the model's learning of key areas. The steps of the GazeMix data augmentation strategy are as follows: First, in the labeled or unlabeled images, the doctor's perception is The doctor's gaze area is cropped out, and the heat map provides information about the doctor's gaze concentration in the image, reflecting the doctor's focus. In order to accurately extract these areas, a threshold is set for the gaze area in the heat map, and areas with grayscale values ​​greater than 0.1 are screened out to ensure that the extracted areas do contain the organs or lesions that the doctor is concerned about in the image. These extracted gaze areas are then pasted to the corresponding positions of other unlabeled images to generate new mixed image pairs. This combination retains the key information that the doctor is concerned about, while increasing the background diversity, enabling the model to learn to recognize the same target area in different backgrounds. By integrating the focus area into the background of unlabeled data, the model can obtain more semantically consistent training samples without relying on a large amount of labeled data.

[0052] S4: The GazeMix data augmentation method in step S3 is used for the teacher-student network model. GazeMix extracts the foreground from the labeled image and pastes it to the background of other unlabeled images to generate a mixed image. The mixed image and the original labeled image data are used to construct a dataset. Pre-training on the constructed dataset;

[0053] The semi-supervised training strategy uses a teacher-student model for two-stage training. In the first stage, the teacher model is pre-trained using labeled data. Specifically, GazeMix is ​​used for data augmentation to expand the diversity of the dataset, and the foreground areas of concern to the doctor are extracted from the annotated image based on the doctor's perception. These areas contain the key organs or lesions that the doctor is looking at. The same GazeMix operation is performed on the labels of the original annotated image to generate mixed labels that match the mixed image. The mixed labels ensure that the accurate semantic information of the foreground area is retained on the mixed image, thereby providing a complete supervision signal for the teacher model during the training process. The generated mixed images and their corresponding mixed labels are input into the teacher model for training, so that the teacher model can learn the target area features of the image under different background and foreground combinations. At this stage, the teacher model learns the feature representation of the key area through diversified samples and has strong generalization ability.

[0054] S5: Generate a mixed image based on GazeMix in steps S2 and S3 to perform the second stage student network model GazeMix generates mixed image data and inputs them into the pre-trained teacher model. and student models , teacher model The output of is used as pseudo label;

[0055] In the retraining phase of the student network model, the GazeMix data enhancement strategy is used to expand the labeled and unlabeled data and generate two types of mixed image pairs to improve the segmentation performance of the student network on labeled and unlabeled data. The specific steps are as follows: GazeMix data enhancement generates two types of mixed image pairs: type one labeled sample pairs :Generate a cropping mask based on the gaze heat map of the labeled data, and extract the key area (i.e., the foreground part) where the doctor is looking from the labeled data. Then, paste the extracted foreground area onto the background of the unlabeled data to generate a mixed image pair with labeled foreground and unlabeled background. For the corresponding labeled labels, perform the same GazeMix operation to generate labeled images with the same foreground, and uniformly set the background value of the label to 0 to ensure consistency with the mixed image, generating a labeled sample pair with labeled foreground and unlabeled background. ; Type 2 pseudo-label sample pair : For unlabeled data as foreground, other unlabeled data as background use the same GazeMix operation to generate a mixed image , and then input the GazeMix enhanced data into the student network and the teacher network, where the teacher network uses a separate Input, will output As The pseudo labels are used for supervision of the student network. Gradient descent is not used to update parameters in the second-stage training of the teacher network. During the training of the second-stage student network, the sample pairs are divided into and Used for student network training.

[0056] S6: During the student network model training process in step S5, a multi-scale gaze perception module (MGP) is designed to extract and fuse the network perception at different scales. And perform perceptual loss calculation to enhance the model's feature extraction ability in complex medical images;

[0057] The Multi-scale Gaze Perception (MGP) module is designed to enhance the model's multi-scale perception of key areas. It includes a channel attention module and a spatial attention module, and is embedded before the segmentation head layer of the model segmentation output to improve segmentation accuracy. The MGP module is embedded before the segmentation head layer of the model segmentation output, using image upsampling features As input, The shape and size of , extracting network perception capabilities through the channel attention module and the spatial attention module in turn; the channel attention module directly adopts As input, the feature map is first compressed using a global average pooling operation of Dimension, generate Global eigenvector of shape size , and then use the MLP layer to transform the feature vector Do the calculation and generate the attention weight on the channel dimension through the sigmoid activation function , focus on the channel With the original features Perform multiplication operation to obtain the channel-aware enhanced feature map ; Use convolution kernels of different sizes (1x1, 3x3, and 7x7) to convolution the feature map Perform spatial attention perception feature extraction to capture the fine-grained and overall spatial features at different scales. Splice the features of three different scales according to channels, adjust the number of channels through a 1x1 convolution layer, and achieve feature fusion at different scales. Finally, constrain the weight of the image pixel level to the range of 0-1 through the sigmoid activation function, and obtain the spatial perception attention weight. As network perception for doctor perception Do the loss calculation and Perform multiplication operations to obtain feature maps after multiple enhanced perception in space and channels Used as input for the last layer of convolution and outputs the final segmentation result.

[0058] Through the MGP module, the model obtains a more precise focus on the target area in the spatial and channel dimensions, thereby effectively improving the segmentation performance.

[0059] S7: According to step S2, S6 , Constructing the perceptual loss function , constrain the alignment of the network’s perception and the doctor’s gaze area, and improve the model’s ability to focus on key areas in complex backgrounds by gradually optimizing the network’s perception, thereby improving the accuracy of medical image segmentation;

[0060] Perceived loss Network perception using mean square error (MSE) contrast model Perception with doctors , so that the model gradually aligns to the key areas that doctors are concerned about, and the perceptual loss expression is:

[0061]

[0062] in, Represents the value of each pixel in the doctor's perception, Represents the value of each pixel in network perception, Represents the number of pixels perceived by all doctors.

[0063] S8: The perceptual loss function in step S7 With segmentation loss Perform weighted calculations and use the Adam optimizer to optimize the student model Update parameters;

[0064] The loss function is a weighted combination of the segmentation loss function and the perception loss function. The segmentation loss is composed of the weighted Dice loss and the cross entropy loss. The perception loss compares the perception area of ​​the model with the human gaze area through the mean square error (MSE). The weighted result of the comprehensive loss is expressed as:

[0065]

[0066] in, is the cross entropy loss, is the Dice loss, Represents the segmentation result predicted by the network, represents the true label of the image or the pseudo label generated by the teacher network, Represents the segmentation results of different categories. Represents the true labels of different categories or the pseudo labels generated by the teacher network, Indicates the number of pixels in the real label. Indicates the number of categories of true labels, , , , is the weighting coefficient, indicating the weight ratio, and the sizes are 0.5, 0.5, and 0.1 respectively.

[0067] S9: Update the teacher model using the exponential moving average (EMA) method ;

[0068] In each training step, the teacher model The exponential moving average (EMA) update is performed, and the teacher model parameters are slightly adjusted according to the changes in the student network parameters, thereby gradually accumulating the latest learning results of the student network while maintaining a certain stability, so that the teacher network can provide a consistent and reliable supervision signal to the student network. The parameter update expression of the teacher model is:

[0069]

[0070] in, and Represent the model parameters of the teacher network and the student network respectively, It is the smoothing coefficient of EMA to ensure that the parameter update of the teacher network is relatively smooth and maintain a stable pseudo-label generation effect. The value is 0.99.

[0071] In the comparative experiment, the experimental results of this method are compared with those of MC-Net, SS-Net and BCP methods on the ACDC dataset. The comparison results are as follows: Figure 5 a-f. Among them,

[0072] (a) is the original image;

[0073] (b) is the true segmentation label of the original image;

[0074] (c) is the segmentation predicted by the MC-Net algorithm;

[0075] (d) is the segmentation result predicted by the SS-Net algorithm;

[0076] (e) is the segmentation result predicted by the BCP algorithm;

[0077] (f) is the prediction result of the present invention;

[0078] In the comparative experiment, the experimental results of this method are compared with those of MC-Net, SS-Net and BCP methods on the CAMUS dataset. Figure 6 a-f. Among them,

[0079] (a) is the original image;

[0080] (b) is the true segmentation label of the original image;

[0081] (c) is the segmentation predicted by the MC-Net algorithm;

[0082] (d) is the segmentation result predicted by the SS-Net algorithm;

[0083] (e) is the segmentation result predicted by the BCP algorithm;

[0084] (f) is the prediction result of the present invention; it can be seen that compared with other semi-supervised learning algorithms, the method proposed in the present invention can segment the target more accurately and has high-quality visualization results.

[0085] It should be noted that the above embodiments are not intended to limit the protection scope of the present invention, and equivalent changes or substitutions made on the basis of the above technical solutions all fall within the protection scope of the claims of the present invention.

Claims

1. A semi-supervised medical image segmentation method guided by eye movement and hybrid data enhancement, characterized in that: The method comprises the following steps: S1: Collecting the gaze information of radiologists when reading medical images through eye tracking equipment; S2: Perform doctor perception conversion on the gaze information collected in step S1, and construct the eye movement position matrix G according to the original eye movement gaze information pos , use Gaussian blur to pos Processing to obtain doctor perception G heat ; S3: Based on the doctor's perception G in step S2 heat , designing the GazeMix data augmentation method, which involves extracting the image parts containing the doctor’s gaze area from the annotated and unannotated images and pasting them into the background of other unannotated images to generate new image pairs to expand the diversity and semantic consistency of the dataset; S4: The GazeMix data augmentation method in step S3 is used for the teacher-student network model. GazeMix extracts the foreground from the labeled image and pastes it to the background of other unlabeled images to generate a mixed image. The mixed image and the original labeled image data are used to construct a dataset. The teacher model T is pre-trained on the constructed dataset. S5: GazeMix generates mixed images according to steps S2 and S3 to train the student network model S in the second stage. The mixed image data generated by GazeMix is ​​input into the pre-trained teacher model T and the student model S respectively, and the output of the teacher model T is used as a pseudo label. S6: During the student network model training process in step S5, a multi-scale gaze perception module MGP is designed to extract and fuse the network perception G at different scales. net And perform perceptual loss calculation to enhance the model's feature extraction ability in complex medical images; S7: According to step S2, G in S6 heat , G net Constructing the perceptual loss function L gaze , constrain the alignment of the network’s perception and the doctor’s gaze area, and improve the model’s ability to focus on key areas in complex backgrounds by gradually optimizing the network’s perception, thereby improving the accuracy of medical image segmentation; S8: The perceptual loss function L in step s7 is gaze and segmentation loss L seg Perform weighted calculations and use the optimizer to update the parameters of the student model S; S9: Update the teacher model T using the exponential moving average EMA method; The GazeMix data enhancement method in step S3 further includes: (1) Using doctors’ perception of G heat Generate a cropping mask to extract the portion of the image containing the area where the doctor is looking; (2) By cropping the crop mask part of the labeled image or the unlabeled image, the foreground area where the doctor is looking is extracted, and these foreground areas are pasted to the background position of other unlabeled images to generate new mixed image samples; The multi-scale gaze perception module in step S6 includes: (1) Channel attention module, which compresses the channel dimension of the input feature map through global average pooling operation and generates channel attention weights to enhance the model's attention to key channels; (2) The spatial attention module uses convolution kernels of different sizes (1x1, 3x3, and 7x7) to perform multi-scale spatial feature extraction to capture the fine-grained and overall spatial features of the image and fuse information at different scales.

2. The eye-movement guided hybrid data enhanced semi-supervised medical image segmentation method according to claim 1, characterized in that: The process of generating the doctor's perception of the gaze information in step S2 includes constructing a position matrix, mapping image pixels, and generating grayscale values ​​through Gaussian blur.

3. The eye-movement guided hybrid data enhanced semi-supervised medical image segmentation method according to claim 1, characterized in that: In step S4, the teacher-student network model and the two-stage training strategy are used for semi-supervised training. In the first stage of training, GazeMix is ​​used to transfer the labeled data X l As foreground, unlabeled data x u Generates labeled mixed data as background Do the same thing for the labels of the labeled data to generate get A dataset is constructed together with the original labeled data, and the teacher model T is pre-trained on the constructed dataset.

4. The eye-movement guided hybrid data enhanced semi-supervised medical image segmentation method according to claim 1, characterized in that: The second stage training of the student model S in step S5 includes: (1) Use GazeMix to perform data augmentation on labeled and unlabeled data, using labeled data x l As foreground, unlabeled data X u Generate labeled mixed data as background Using unlabeled data x u As the foreground, the unlabeled data x u Generate unlabeled mixed data as background (2) and Input student model S, Input teacher model T, where the teacher model is The output is used as a pseudo label and Construct pseudo-label sample pairs The real labels generated after GazeMix data augmentation are used as labeled sample pairs (3) Calculate the segmentation loss based on the two groups of sample pairs.

5. The eye-movement guided hybrid data enhanced semi-supervised medical image segmentation method according to claim 1, characterized in that: The perceptual loss function L described in step S7 gaze Network perception G using mean square error contrast model net Sense G with Doctor heat , so as to gradually align the model's attention to the key areas that doctors pay attention to. The perceptual loss expression is: in, Represents the value of each pixel in the doctor's perception, Represents the value of each pixel in network perception, and n represents the number of pixels perceived by all doctors.

6. The eye-movement guided hybrid data enhanced semi-supervised medical image segmentation method according to claim 1, characterized in that: When the model is trained in step S8, the pseudo-labels, real labels and network perception are used together to construct a loss function, including segmentation loss, which is composed of Dice loss and cross entropy loss weighted, and the segmentation results output by the model are compared with the real labels and pseudo-labels of the labeled data. The perception loss compares the perception area of ​​the model with the human gaze area through the mean square error MSE. The comprehensive loss weighted result expression is: THE total =L seg +λL gaze L seg =αL ce +βL dicc Among them, L ce is the cross entropy loss, L dice is the Dice loss, p represents the segmentation result predicted by the network, y represents the true label of the image or the pseudo label generated by the teacher network, and p ic Represents the segmentation results of different categories, y ic Represents the true labels of different categories or the pseudo labels generated by the teacher network, N represents the number of pixels in the true label, M represents the number of categories of the true label, α, β, λ are weighting coefficients, representing the weight proportion, and the sizes are 0.5, 0.5, and 0.1 respectively.

7. The eye-movement guided hybrid data enhanced semi-supervised medical image segmentation method according to claim 6, characterized in that: In step S9, the model is trained through the teacher-student architecture, and the teacher model T is updated with an exponential moving average EMA in each training step, so that the teacher network is gradually optimized and the stable pseudo-label generation capability is maintained. The teacher model parameter update expression is: f(θ)=σT(θ)+(1-σ)S(θ) Among them, T(θ) and S(θ) represent the model parameters of the teacher network and the student network respectively, and σ is the smoothing coefficient, which is used to control the decay rate of the old parameters and its value is 0.99.

Images