Medical image segmentation method and system under eye movement assisted training

Abstract

The application discloses a medical image segmentation method and system under eye movement auxiliary training; the method is as follows: one, a data set containing eye movement data is established. Step two, the eye movement data in the data set is preprocessed to form a saliency view. Three, an eye movement information learning module is constructed. Four, a medical image segmentation module is constructed; the medical image segmentation module can fuse the attention features output by the eye movement information learning module and segment the target region in the medical image through a deep learning mode. Five, the trained eye movement information learning module and medical image segmentation module are used to identify the measured image. The application has low eye movement data acquisition cost and high acquisition efficiency, can automatically collect eye movement data for auxiliary segmentation when a doctor reads a medical image, can integrate the eye movement data into a segmentation model, can make the model learn the attention feature information of a person, and can achieve better segmentation effect in the case that medical image data is insufficient.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image segmentation (not for direct diagnosis purposes), in particular to a medical image segmentation method based on eye movement information assistance. BACKGROUND

[0002] Medical image segmentation has a very wide range of applications, including tumor segmentation, organ segmentation, blood vessel segmentation, etc. Although medical image segmentation cannot achieve disease diagnosis, by accurately segmenting medical images, doctors can more conveniently focus on the key areas in the medical images, thereby improving the analysis speed of doctors.

[0003] Medical image segmentation is an important task in medical diagnosis, surgical planning, and other healthcare applications. In recent years, deep learning methods have shown great potential in this field, providing excellent performance. However, these data-driven methods require a large amount of annotated image data to train deep learning models. Unfortunately, obtaining and labeling medical images can be very time-consuming and demanding, requiring a lot of effort from experts in the field. This bottleneck has to some extent restricted the development of medical image segmentation, hindering progress in this critical medical field.

[0004] To address this problem, some studies have proposed using interactive information such as mouse clicks, inputting additional text descriptions, etc. to assist in guiding the model's segmentation in order to achieve satisfactory segmentation results. However, the drawback of these technical methods is that they still require additional effort from professionals, affecting the normal workflow of professionals such as doctors. SUMMARY

[0005] The present application proposes a medical image segmentation method and system based on eye movement assistance to address the problems in the current medical image segmentation field.

[0006] In a first aspect, the present application provides a medical image segmentation method under eye movement assistance training, which includes the following steps:

[0007] Step one, establish a data set; the data set contains multiple medical images and eye movement data of the observer when observing the medical images.

[0008] Step two, pre-process the eye movement data in the data set to form saliency views.

[0009] Step three, build an eye movement information learning module; the eye movement information learning module can learn attention features in the eye movement data through eye movement saliency views.

[0010] Step four, build a medical image segmentation module; the medical image segmentation module can fuse the attention features output by the eye movement information learning module and segment the target area in the medical image through deep learning.

[0011] The medical image segmentation module comprises an encoding module, an eye movement information fusion enhancement module and a decoding module.

[0012] The eye movement information fusion enhancement module is configured to receive the feature map output by the encoding module and the attention feature generated by the eye movement information learning module, and to fuse and enhance the two.

[0013] The decoding module is configured to decode the image feature information output by the eye movement information fusion enhancement module, and to obtain the target region in the medical image.

[0014] Step five, using the trained eye movement information learning module and medical image segmentation module to identify the measured image.

[0015] As a preferred, step 1-1. Collect the time-series eye movement data of the observer when observing the medical image through the eye movement information collection platform; the time-series eye movement data includes the position information of the observer's gaze on the medical image.

[0016] As a preferred, the specific process of step two is as follows:

[0017] Step 2-1. Divide each medical image corresponding eye movement data points into multiple data groups according to time.

[0018] Step 2-2. Calculate the attention level of each eye movement data point respectively.

[0019] Step 2-3. For each medical image, calculate the average value of the attention level of all eye movement data points in each data group respectively; delete the data group whose average value is less than or equal to the attention level threshold. The attention level threshold is the average value of the attention level of all eye movement data points in all data groups. The eye movement data points in each data group that is not deleted are subjected to Gaussian blur together to generate an eye movement saliency view.

[0020] As a preferred, the eye movement information learning module adopts a network structure based on Transformer, takes the medical image as the input of the eye movement information learning module, and takes the corresponding eye movement saliency view as the supervision source of supervised learning. After passing through multiple Transformer layers, the output result is obtained; the eye movement information learning module is trained through back propagation.

[0021] As preferred, the eye movement information fusion enhancement module comprises a CBAM attention mechanism module, two channel reduction modules, two Embeded Paches modules and a multi-head attention module. In the working process of the eye movement information fusion enhancement module, the feature map input by the encoding module is first sent to the CBAM attention mechanism module for the first step of attention enhancement. Then, the feature map filtered by the CBAM attention mechanism module and the attention feature information are respectively sent to the two channel reduction modules for compression. Then, the information output by the two channel reduction modules is respectively sent to the two Embeded Paches modules; the two Embeded Paches modules encode the position information of the input feature data and transform the data dimension. Finally, the information output by the two Embeded Paches modules is jointly sent to the multi-head attention module; the multi-head attention module takes the processed attention feature information as a query vector; takes the processed feature map as a key vector and a value vector, and obtains the image feature information under the eye movement information fusion through multi-head attention calculation.

[0022] As preferred, the eye movement information learning module and the medical image segmentation module are trained as a whole network structure. In the training process, the medical image segmentation module is supervised by the real target region label, learns to segment the medical image through the back propagation algorithm, and the eye movement information learning module learns the attention features in the eye movement information and provides the medical image segmentation module for assisting the segmentation of the medical image.

[0023] As preferred, the specific process of step five is as follows: input the measured medical image into the trained eye movement information learning module; the eye movement information learning module labels the predicted eye movement attention region on the measured medical image; the measured medical image and the predicted eye movement attention region are jointly input into the medical image segmentation module for encoding, fusion enhancement and decoding, the target region is segmented on the measured medical image, and the target region is framed.

[0024] In the second aspect, the application provides a medical image segmentation system under eye movement auxiliary training, which comprises an eye movement information acquisition module, an eye movement information preprocessing module, an eye movement information learning module, a medical image segmentation module and a visualization processing module. The eye movement information acquisition module is used to collect the eye movement data of the observer when watching the medical image through the eye tracker, and establish a data set. The eye movement information preprocessing module is used to preprocess the eye movement data, exclude noise point information, and form an eye movement saliency view.

[0025] The eye movement information learning module uses the eye movement saliency view to learn the attention features in the eye movement data.

[0026] The medical image segmentation module includes an encoding module, an eye-tracking information fusion and enhancement module, and a decoding module. The encoding module encodes the medical image data using convolutional layers. The eye-tracking information fusion and enhancement module receives the feature map output from the encoding module and the attention features generated by the eye-tracking information learning module, and fuses and enhances them. The decoding module decodes the image feature information output from the eye-tracking information fusion and enhancement module to identify the target region in the medical image.

[0027] The visualization processing module is used to generate visualization results of target regions in medical images.

[0028] Preferably, the eye-tracking information learning module adopts a Transformer-based network structure.

[0029] Preferably, the medical image segmentation module adopts an Encoder-Decoder model structure.

[0030] The beneficial effects of this invention are as follows:

[0031] 1. This invention utilizes the low cost and high efficiency of eye-tracking data acquisition. It automatically collects eye-tracking data for auxiliary segmentation while doctors are reading medical images, without interfering with normal work processes. Combined with deep learning technology, eye-tracking data is integrated into the segmentation model, allowing the model to learn human attention characteristics and achieve better segmentation results when medical image data is insufficient.

[0032] 2. The model structure proposed in this invention adopts a multi-task joint training mode (i.e., segmentation task and eye-tracking information learning task). At the same time, the proposed eye-tracking information learning module and eye-tracking information fusion enhancement module are universal and can be used in any deep learning task that combines eye-tracking information. Attached Figure Description

[0033] Figure 1 This is a flowchart of Embodiment 1 of the present invention;

[0034] Figure 2 This is an example diagram of eye movement information collected by an eye tracker when an observer observes medical images in Embodiment 1 of the present invention;

[0035] Figure 3 This is the eye movement saliency view obtained after preprocessing the eye movement data in Embodiment 1 of the present invention;

[0036] Figure 4 This is a network structure diagram of the eye-tracking information learning module in Embodiment 1 of the present invention;

[0037] Figure 5 This is a network structure diagram of the medical image segmentation module in Embodiment 1 of the present invention;

[0038] Figure 6 is a test result graph of embodiment 1 of the present application. DETAILED DESCRIPTION

[0039] The present application is described below by way of specific examples. Other advantages and effects of the present application can be easily understood by those skilled in the art from the description. The present application can also be implemented or applied by different specific embodiments, and the details in the description can be modified or changed based on different views and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0040] Embodiment 1

[0041] As shown in Figure 1 , a medical image segmentation method under eye movement auxiliary training includes the following steps:

[0042] S1. An observer with medical image resolution capability observes a plurality of medical images with target regions in sequence; during the observation process, eye movement data of the observer is collected by an eye tracker; the specific process of this step is as follows:

[0043] S11. Two medical colonoscopy polyp data sets, Kvasir-SEG and CVC-ClinicDB, are selected as the objects for collecting eye movement data.

[0044] S12. An eye movement information collection platform is built to collect eye movement data of the observer when observing medical images, mainly recording the position information of the observer's gaze on the picture examples. During the collection of eye movement data, the observer sits in front of the computer, the Tobii eye tracking 5 eye tracker is located below the computer screen, after completing the calibration, the eye movement data recording software is started, and the eye movement data of the observer on the current medical image is recorded, as shown in Figure 2 . In this embodiment, eye movement data of 1300 image examples is collected.

[0045] S13. For a single medical image example, the collection time is 8 seconds, and the eye tracker records the eye movement position information within the collection time.

[0046] S2. The eye movement data obtained in step S1 is sent to an eye movement information preprocessing module for preprocessing; the eye movement information preprocessing module excludes noise point information to form an eye movement saliency view, which indicates the important region information in the image data; the specific process of this step is as follows:

[0047] S21. The eye movement data points corresponding to each medical image are divided into 8 data groups according to time.

[0048] S22. Calculate the attention level of each eye movement data point respectively; when calculating the attention level, a curve fitting method is adopted, and the pixel distance between eye movement fixation points and the frequency of the distance are used as points on the curve for fitting, and the fitting parameters are used as a quantitative evaluation method of the high and low of the attention level. Specifically, for each set of eye movement data, the pixel distance s between eye movement fixation points and its corresponding frequency p are counted, forming multiple pairs of data (s, p); the power function p = γs -2 is used to fit these data, and the value of the fitting parameter γ represents the attention level of this set of eye movement data.

[0049] S23. For each medical image, calculate the average value of the attention level of all eye movement data points in each data group respectively; delete the data group whose average value is less than or equal to the attention level threshold. The attention level threshold is the average value of the attention level of all eye movement data points in the 8 data groups. The eye movement data points in each data group that is not deleted are subjected to Gaussian blur together, and the eye movement saliency view can be obtained, as shown in Figure 3 In this embodiment, the radius of Gaussian blur is set to 251; after Gaussian blur processing, the view representing the attention characteristics of the observer can be obtained.

[0050] S3. The eye movement information learning module learns the attention features in the eye movement data using the eye movement saliency view, supervises the learning of the eye movement information learning module, and improves the accuracy of the model.

[0051] As shown in Figure 4 , the eye movement information learning module adopts a network structure based on Transformer, which learns the human attention features in the eye movement data with the excellent long-distance exploration capability. The original medical image sample is used as the input of the eye movement information learning module, and the corresponding eye movement saliency view is used as the supervision source of supervised learning. After passing through 4 layers of Transformer layer, the output result is obtained, and the eye movement information learning module is trained through back propagation, so that the module learns the attention feature information in the eye movement information.

[0052] The eye movement information learning module uses the VisionTransformer structure which performs well in the field of vision to learn the human attention features in the eye movement information. The original picture is used as the input, and the eye movement saliency view is used as the supervision source. The learning rate is set to 10 -4 during training, and the binary cross-entropy loss is used as the loss function to train the eye movement information learning module.

[0053] S4. The medical image segmentation module fuses the information of the eye movement information learning module and uses deep learning technology to segment the target region in the medical image; the specific method of this step S4 is as follows:

[0054] The medical image segmentation module adopts an Encoder-Decoder model structure, and the information learned by the eye movement information learning module enhances the coding feature expression of the encoder in the medical image segmentation module. The enhanced coding features are input into the decoder to obtain the segmentation result.

[0055] As shown in Figure 5 The medical image segmentation module includes an encoding module, an eye movement information fusion enhancement module, and a decoding module.

[0056] The encoding module is configured to receive the medical image data to be segmented, encode the medical image data by using a convolutional layer, and input the obtained feature map into the eye movement information fusion module.

[0057] The eye movement information fusion enhancement module is configured to receive the feature map of the encoding module and the attention feature information generated by the eye movement information learning module in step S3, and fuse and enhance the two. The eye movement information fusion enhancement module includes one CBAM attention mechanism module (i.e., a convolution block-based attention mechanism module), two channel reduction modules, two Embeded Paches modules (i.e., position encoding models), and one multi-head attention module.

[0058] In the working process of the eye movement information fusion enhancement module, the feature map input by the encoding module is first input into the CBAM attention mechanism module for the first attention enhancement, and the redundant information therein is filtered.

[0059] Then, the feature map filtered by the CBAM attention mechanism module and the attention feature information are respectively input into the corresponding channel reduction modules to compress the information, so as to reduce the information quantity without losing accuracy and improve the inference speed of the model.

[0060] Then, the information output by the two channel reduction modules is respectively input into the two Embeded Paches modules; the two Embeded Paches modules encode the position information of the input feature data and transform the data dimension.

[0061] After that, the information output by the two Embeded Paches modules is jointly input into the multi-head attention module; the multi-head attention module takes the processed (i.e., sequentially processed by the CBAM attention mechanism module, the channel reduction module, and the Embeded Paches module) attention feature information as a query vector; takes the processed feature map as a key vector and a value vector, and obtains the image feature information under the eye movement information fusion through multi-head attention calculation.

[0062] Finally, the image feature information under the eye movement information fusion is input into the decoding module to extract the image of the target region (a region containing the target region).

[0063] The extracted image is compared with a preset label, and the parameters in the medical image segmentation module are iteratively updated to complete the training process.

[0064] The eye movement information learning module and the medical image segmentation module are trained as a whole network structure. During the training process, the medical image segmentation module is supervised by the real segmentation label, and learns to segment the medical image through the back propagation algorithm. The eye movement information learning module learns the attention features in the eye movement information and provides the information to the medical image segmentation module to assist the segmentation of the segmentation module. The overall network structure is shown in Figure 5 .

[0065] The collected 1300 eye movement data and corresponding medical images are divided into training set and test set according to the ratio of 8:2, and are sent into the network shown in Figure 5 , and trained for 200 epochs to reach convergence. The training results are shown in Figure 6 .

[0066] The medical image segmentation module uses four specific models for replacement; the four segmentation models are UNet, MSNet, PraNet, and FCBFormer. The results labeled as "No Gaze" are the model training results without fusing eye movement information, and the results labeled as "Gaze" are the training results with fusing eye movement information. The segmentation accuracies of the four segmentation models without fusing eye movement information are 0.7524, 0.8477, 0.7910, and 0.8470, respectively. The segmentation accuracies of the models with fusing eye movement information are 0.7962, 0.8557, 0.8542, and 0.8632, respectively. The segmentation accuracies of the four segmentation models are improved to different degrees after fusing eye movement information, which shows that the assistance of eye movement information helps to improve the accuracy of medical image segmentation.

[0067] Therefore, it is shown that the embodiment uses the characteristics of eye movement information and combines deep learning technology to propose a medical image segmentation method that fuses eye movement information, which can effectively improve the accuracy of the segmentation model and improve the segmentation effect of medical images in the case of insufficient medical image data. At the same time, the medical image segmentation method based on eye movement information assistance can record eye movement data while professional personnel read medical images during the training set construction process, which does not affect the normal workflow and does not require additional efforts from professional personnel.

[0068] S5. Identify the target region of the measured image.

[0069] The measured medical image is input into the trained eye movement information learning module; the eye movement information learning module labels a predicted eye movement attention area on the measured medical image; the measured medical image and the predicted eye movement attention area are jointly input into a medical image segmentation module for encoding, fusion enhancement and decoding, a target area is segmented on the measured medical image, and the target area is framed; the measured medical image framed by the target area is displayed to the doctor through a visualization processing module, so as to facilitate the observation and judgment of the doctor on the medical image.

[0070] The target area generated in the embodiment is only used for assisting the observation of the doctor or scientific research, and the information obtained as an intermediate result is not used for directly diagnosing diseases.

[0071] Embodiment 2

[0072] A medical image segmentation system under eye movement assisted training comprises:

[0073] An eye movement information acquisition module acquires eye movement data of an observer when watching a medical image by using an eye tracker, and the eye movement data is preprocessed by an eye movement information preprocessing module.

[0074] The eye movement information preprocessing module excludes noise point information, and forms an eye movement saliency view, wherein the saliency view indicates the information of important areas in image data.

[0075] The eye movement information learning module learns the attention features in the eye movement data by using the eye movement saliency view, supervises the learning of a medical image segmentation module, and improves the precision of the model.

[0076] The medical image segmentation module fuses the eye movement information, and segments a target area image by using a deep learning technology.

[0077] The visualization processing module generates a visual segmentation result.

[0078] The specific embodiments described in the present application are only examples for illustrating the spirit of the present application. Those skilled in the art of the present application can make various modifications or supplements to the described specific embodiments or use similar ways to replace, but will not deviate from the spirit of the present application or exceed the scope defined by the appended claims.

Claims

1. A medical image segmentation method under eye movement assisted training, characterized in that: The method comprises the following steps: Step 1: establishing a data set; The data set contains multiple medical images and eye movement data of observers when observing the medical images; Step 2: preprocessing the eye movement data in the data set to form a saliency view; Step 3: constructing an eye movement information learning module; the eye movement information learning module can learn the attention features in the eye movement data through the eye movement saliency view; The eye movement information learning module adopts a network structure based on a Transformer, takes the medical image as the input of the eye movement information learning module, takes the corresponding eye movement saliency view as the supervision source of supervised learning, obtains an output result after passing through multiple Transformer layers, and is trained through back propagation; Step 4: constructing a medical image segmentation module; The medical image segmentation module can fuse the attention features output by the eye movement information learning module and segment the target region in the medical image through deep learning; The medical image segmentation module comprises an encoding module, an eye movement information fusion enhancement module and a decoding module; the encoding module is used for encoding the medical image data through a convolutional layer; The eye movement information fusion enhancement module is used for receiving the feature map output by the encoding module and the attention features generated by the eye movement information learning module and fusing and enhancing the two; the eye movement information fusion enhancement module comprises one CBAM attention mechanism module, two channel reduction modules, two Embeded Paches modules and one multi-head attention module; in the working process of the eye movement information fusion enhancement module, the feature map input by the encoding module is first sent into the CBAM attention mechanism module for the first attention enhancement; then, the feature map filtered through the CBAM attention mechanism module and the attention feature information are respectively sent into the two channel reduction modules for compression; then, the information output by the two channel reduction modules is respectively sent into the two Embeded Paches modules; the two Embeded Paches modules encode the position information of the input feature data and transform the data dimension; finally, the information output by the two Embeded Paches modules is jointly sent into the multi-head attention module; the multi-head attention module takes the processed attention feature information as a query vector, takes the processed feature map as a key vector and a value vector, performs multi-head attention calculation, and obtains the image feature information under the eye movement information fusion; The decoding module is used for decoding the image feature information output by the eye movement information fusion enhancement module and the target region in the medical image; The eye movement information learning module and the medical image segmentation module are trained as a whole network structure; in the training process, the medical image segmentation module takes the real target region label as the supervision source, learns to segment the medical image through the back propagation algorithm, the eye movement information learning module simultaneously learns the attention features in the human eye movement information and provides the attention features to the medical image segmentation module for assisting the segmentation of the medical image. Step five, using the trained eye movement information learning module and medical image segmentation module to identify the measured image; input the measured medical image into the trained eye movement information learning module; The eye movement information learning module labels the predicted eye movement attention area on the measured medical image; The measured medical image and the predicted eye movement attention area are jointly input into the medical image segmentation module for encoding, fusion enhancement and decoding, and the target area is segmented on the measured medical image, and the target area is framed.

2. The medical image segmentation method under eye movement assisted training according to claim 1, characterized in that: Step 1-1. Collect the time sequence eye movement data of the observer when observing the medical image through the eye movement information collection platform; the time sequence eye movement data includes the position information of the observer's gaze on the medical image.

3. The medical image segmentation method under eye movement assisted training according to claim 1, characterized in that: The specific process of step two is as follows: Step 2-1. Divide each medical image corresponding eye movement data points into multiple data groups according to time; Step 2-2. Calculate the attention level of each eye movement data point respectively; Step 2-3. For each medical image, calculate the average value of the attention level of all eye movement data points in each data group; delete the data group whose average value is less than or equal to the attention level threshold; the attention level threshold is the average value of the attention level of all eye movement data points in all data groups; the eye movement data points in each data group that is not deleted are jointly subjected to Gaussian blur to generate an eye movement saliency view.

4. A medical image segmentation system under eye movement assisted training, characterized in that: The medical image segmentation system includes an eye movement information collection module, an eye movement information preprocessing module, an eye movement information learning module, a medical image segmentation module, and a visualization processing module; the medical image segmentation system is used to execute the medical image segmentation method under the eye movement assisted training as claimed in claim 1; the eye movement information collection module is used to collect the eye movement data of the observer when watching the medical image through the eye tracker, and establish a data set; The eye movement information preprocessing module is used to preprocess the eye movement data, exclude noise point information, and form an eye movement saliency view; The eye movement information learning module uses the eye movement saliency view to learn the attention features in the eye movement data; The medical image segmentation module includes an encoding module, an eye movement information fusion enhancement module, and a decoding module; the encoding module is used to encode the medical image data through a convolution layer; the eye movement information fusion enhancement module is used to receive the feature map output by the encoding module and the attention features generated by the eye movement information learning module, and fuse and enhance the two; the decoding module is used to decode the image feature information output by the eye movement information fusion enhancement module, and the target area in the medical image; The visualization processing module is used to generate the visualization result of the target area in the medical image.

5. The medical image segmentation system under eye movement assisted training according to claim 4, characterized in that: The eye movement information learning module adopts a network structure based on Transformer.

6. The medical image segmentation system under eye movement assisted training according to claim 4, wherein: The medical image segmentation module adopts an Encoder-Decoder model structure.

Images