A method for improving the accuracy of an ultrasonic image recognition model through doctors' eye movement data

Abstract

The present invention discloses a method for improving the accuracy of an ultrasonic image recognition model through doctors' eye movement data, comprising the following steps: collecting doctors' eye movement data, creating an eye movement data matrix, generating a mask matrix, generating a fixation heat map, performing data augmentation, starting model training, and stopping training according to the fixation heat map. The method of the present invention reduces overfitting of the model caused by overtraining, thereby improving the recognition accuracy of the existing ultrasonic diagnosis and treatment system.

Description

Technical Field

[0001] The present invention relates to the technical fields of artificial intelligence, computer vision, and medical image analysis, especially the field of ultrasonic image recognition and analysis. In particular, it relates to a method for improving the accuracy of an ultrasonic image recognition model through doctors' eye movement data. Background Art

[0002] Compared with X-ray imaging and CT imaging, since ultrasonic imaging uses mechanical waves, it will not cause radioactive damage to patients. However, also because it uses mechanical waves, the organ structures and lesions shown in the generated images are not as clear as those in CT imaging. It is very difficult for ultrasonic doctors without years of experience to quickly analyze and diagnose. A major reason why ultrasonic equipment cannot be widely used in current clinical medicine is that ultrasonic doctors need to receive high-level training to ensure high-quality diagnostic images and high accuracy. Therefore, the research on computer-aided ultrasonic diagnosis and treatment systems is very meaningful.

[0003] In recent years, with the rapid development of deep learning models, the accuracy of computer vision and image recognition has been greatly improved, accelerating the application of deep learning in the field of medical image analysis and providing technical support for ultrasonic assisted diagnosis and treatment systems. The training of deep models requires a large number of data samples with good labels. For natural images, the collection and classification of samples are relatively easy. However, in the field of medical images, due to patient privacy and the need for professional radiologists to participate in image classification, it is very difficult to obtain training samples with good labels. For deep learning models with a relatively deep depth, too few training sets will inevitably lead to overfitting of the trained model, insufficient generalization ability, a decrease in model accuracy, and misdiagnosis can be fatal in the medical field. At the same time, another problem with deep learning models is that it is very difficult to clearly find the basis for the model's judgment from its recognition results, that is, it is impossible to deeply understand which part of the image the deep learning model makes such a judgment, which should also be avoided in the field of medical image analysis.

[0004] Currently, existing methods for reducing model overfitting include model regularization, increasing the data set, and early stopping of training. Regularization methods in the field of natural image analysis can be appropriately applied to the field of medical images, but data augmentation methods such as rotation and symmetry may not be suitable for ultrasonic images; at the same time, setting a validation set to adjust the training duration will inevitably reduce the data in the training set, which has both advantages and disadvantages for the field of medical images where it is already difficult to collect training data. Summary of the Invention

[0005] The present invention provides a method for improving the accuracy of an ultrasonic image recognition model through doctors' eye movement data. The problem to be solved is to limit the image features learned by the existing deep network, reduce the overfitting of the model, so as to achieve the purpose of improving the recognition accuracy of the auxiliary diagnosis and treatment system.

[0006] To solve the above problems, the present invention adopts the following technical solutions:

[0007] A method for improving the accuracy of an ultrasonic image recognition model through doctors' eye movement data, comprising the following steps: collecting doctors' eye movement data, creating an eye movement data matrix, generating a mask matrix, generating a fixation heat map, performing data augmentation, starting model training, and stopping training according to the fixation heat map.

[0008] Further, the collection and processing of eye movement data: using an eye tracker to collect the eye movement data of professional ultrasonic doctors when observing the ultrasonic images in the data set, and distinguishing the data points into fixation data and saccade data according to the displacement length of each collected data point. The data points within the preset step threshold range are fixation data points, and the positions where the fixation data points are located are recorded and saved as (x, y, t), where x and y are the position information of the fixation data points, and t is the time stamp information of the current image frame;

[0009] Creating an eye movement data matrix: creating an eye movement data matrix E', with a size of X*Y*T, where: X*Y is the corresponding resolution of the ultrasonic image frame, and T is the size of the ultrasonic image frames in this segment;

[0010] Generating a mask matrix: the matrix value E'(x, y, t) at the position (x, y, t) where the fixation data point is located is 1, and the values at other positions are 0. According to the distance D from the doctor's eyes to the screen and the DPI of the monitor, the visual attention radius is calculated And in the eye movement data matrix E', taking each fixation data point as the center, the points within the corresponding range of the visual attention radius r are all assigned a value of 1, so as to generate a mask matrix E;

[0011] Generating a fixation heat map: in a matrix with the same size as the observed image, setting the value M'(x, y, t) at the position (x, y, t) where the fixation data point is located to 255, and the values at other positions to 0, generating a fixation data matrix M', and then performing convolution calculation on the fixation data matrix M' with a Gaussian convolution kernel G with a standard deviation of 0.5 and a mean of 1 corresponding to the size of the visual attention radius r, so as to generate a fixation heat map Where: is the symbol of the convolution operation;

[0012] Data augmentation: Using the mask matrix E generated from the fixation data, perform binary masking on the original image O in the training set, that is, multiply the mask matrix E by the original image O, and add the generated new image E·O to the training set N’ to obtain a new training set N = {N’, E·O}, and send them together into the model for training, so as to realize data augmentation for the images in the dataset;

[0013] Start training: For each training sample Z in N, the size of Z is a three-dimensional matrix of X*Y*T, where: X*Y is the corresponding resolution of the ultrasound image frame, and T is the size of the ultrasound image frames in this segment. Divide it into Z = {Z 1 , Z 2 , …, Z t , … Z T} in the order of frames, where t ∈ [1, T], and: Z t is the t-th two-dimensional matrix of size X*Y obtained by splitting the three-dimensional matrix Z along the third dimension. Then, send it into the selected feature extraction convolutional neural network C (such as Vgg, Resnet, etc.) in turn. The role of the neural network is abstracted as a function C(z). According to the formula Ф t = C(Z t ), obtain a three-dimensional matrix Ф = {Ф 1 , Ф 2 , …, Ф t , …, Ф T}, that is, send the two-dimensional matrices Z 1 , Z 2 , …, Z t , … Z T into the neural network C, and output to obtain two-dimensional matrices Ф 1 , Ф 2 , …, Ф t , …, Ф T . The sizes of these two-dimensional matrices are X c *Y c , and X c and Y c are determined by the selected convolutional neural network and combined into a three-dimensional matrix Ф in the order of the timestamp information t. The size of the three-dimensional matrix is X c *Y c *T. Then, send the three-dimensional matrix Ф in the order of the forward timestamp matrix Ф + = {Ф 1 , Ф 2 , …, Ф t , …, Ф T} into the recurrent neural network R(Ф). According to the formula H + = R(Ф + ), obtain the forward feature three-dimensional matrix H + . The size of the forward feature three-dimensional matrix is X r *Yr *T r , X r , Y r , T r Determined by the selected recurrent neural network and press T r Dimension decomposition into a positive feature two-dimensional matrix H 1+ ,H 2+ ,…,H t+ ,…,H T+ ; Transform the three-dimensional matrix Ф into a matrix Ф in reverse timestamp order - ={Ф T ,…,Ф t ,…,Ф 1} is sent to the recurrent neural network R(Ф), according to the formula H - =R(Ф - ) to obtain the reverse characteristic three-dimensional matrix H - , the reverse characteristic three-dimensional matrix size is X r *Y r *T r , X r , Y r , T r Determined by the selected recurrent neural network and press T r Dimension decomposition into the reverse feature two-dimensional matrix {H 1- ,H 2- ,…,H t- ,…,H T-}, then H + With H - Press T r Dimension combination, that is, the characteristic three-dimensional matrix H={H + , H -}={H 1+ ,H 2+ ,…,H t+ ,…,H T+ ,H 1- ,H 2- ,…,H t- ,…,H T-}Get the characteristic three-dimensional matrix H={H 1 ,H 2 ,…,H t ,…H 2*Tr}, the value range of t is (1,2*T r ), the feature three-dimensional matrix H has a size of X r *Y r *(2*T r ), and then the feature two-dimensional matrix H corresponding to the t-th frame t Input into 3 convolutional layers and two adjustment layers to get the output matrix R t , Rt has dimensions of X * Y * M R , where X * Y is the same as the size of the input video image frame, and M R is the number of channels of the set convolution kernel; after the adjustment layer, there are global average pooling and fully connected layers to classify the image frame at this moment; extract the weight parameters of the fully connected layer, and after dot-multiplying with R t , sum in the M R dimension direction to obtain the corresponding CAM (Class Attention Map);

[0014] End training in advance: After each round of training, calculate the loss by comparing the obtained CAM with the fixation heat map M corresponding to the video sample. When the average loss reaches the threshold set in advance, the training can be stopped.

[0015] Furthermore, when collecting eye movement data, the ultrasound doctor sits in front of the monitor and the eye tracker. The eye tracker is placed D cm below the monitor screen. After calibration, the ultrasound images in the training set are presented on the monitor. The ultrasound doctor observes the image segments and makes image classification judgments.

[0016] Furthermore, during the image classification judgment, during the analysis of each image, use the eye tracker to obtain and record the eye movement data of the ultrasound doctor within the monitor image range, that is, record the fixation coordinates x, y of each sampling of the eye tracker, the time stamp t corresponding to the ultrasound image frame, and the displacement size s; save the eye movement data with s lower than the preset displacement threshold, and according to its coordinates (x, y, t), assign values to the corresponding coordinate positions in the matrix E' with size X * Y * T, and create an eye movement data matrix E' with the same size as the observed image.

[0017] Compared with the prior art, the present invention has the following beneficial effects:

[0018] The present invention collects the eye movement data of the ultrasound doctor when observing ultrasound images, uses this data to perform data augmentation on the training set, and at the same time, according to the attention heat map generated by the doctor's eye movement data, selects an appropriate number of training times to stop training, reducing overfitting caused by overtraining of the model, thereby improving the recognition accuracy of the existing ultrasound diagnosis and treatment system. Description of the Drawings

[0019] Figure 1 is a schematic flow chart of the method of the present invention;

[0020] Figure 2 is a schematic diagram of eye movement data processing of the present invention;

[0021] Figure 3 is a schematic diagram of the feature extraction module of the present invention;

[0022] Figure 4 Schematic diagram of the image classification module and CAM generation of the present invention. Detailed implementation manners

[0023] The specific content of the present invention will be described below with reference to the accompanying drawings and embodiments.

[0024] Embodiment 1

[0025] As Figure 1 shown, a method for improving the accuracy of an ultrasonic image recognition model through doctor's eye movement data includes the following steps: collecting doctor's eye movement data, creating an eye movement data matrix, generating a mask matrix, generating a fixation heat map, performing data augmentation, starting model training, and stopping training according to the fixation heat map;

[0026] Eye movement data collection and processing: Using an eye tracker to collect the eye movement data of professional ultrasonic doctors when observing the ultrasonic images in the data set, and distinguishing the data points into fixation data and saccade data according to the displacement length of each collected data point. The data points within the preset step threshold range are fixation data points, and the positions where the fixation data points are located are recorded and saved as (x, y, t), where x and y are the position information of the fixation data points, and t is the time stamp information of the current image frame;

[0027] When collecting the eye movement data, the ultrasonic doctor sits in front of the monitor and the eye tracker. The eye tracker is placed D cm below the monitor screen. After the eye tracker is calibrated, the ultrasonic images in the training set are presented on the monitor. After the ultrasonic doctor observes the image segment, an image classification judgment is made;

[0028] Creating an eye movement data matrix: Creating an eye movement data matrix E', with a size of X * Y * T, where: X * Y is the resolution corresponding to the ultrasonic image frame, and T is the size of the ultrasonic image frame of this segment of ultrasonic images;

[0029] During the image classification judgment, during the analysis of each image, the eye tracker is used to obtain and record the eye movement data of the ultrasonic doctor within the monitor image range, that is, record the fixation coordinates x and y of each sampling of the eye tracker, the time stamp t of the corresponding ultrasonic image frame, and the displacement size s; Save the eye movement data with s lower than the preset displacement threshold, and assign values to the corresponding coordinate positions in the matrix E' with a size of X * Y * T according to its coordinates (x, y, t) to create an eye movement data matrix E' with the same size as the observed image;

[0030] Generating a mask matrix: The matrix value E'(x, y, t) at the position (x, y, t) where the fixation data point is located is 1, and the values at other positions are 0. According to the distance D from the doctor's eyes to the screen and the DPI of the monitor, calculate the visual attention radius In the eye movement data matrix E', taking each fixation data point as the center, assign the points within the range corresponding to the visual attention radius r to 1, thereby generating the mask matrix E;

[0031] Generate the fixation heat map: In a matrix with the same size as the observed image, set the value M'(x, y, t) at the position (x, y, t) of the fixation data point to 255, and the values at other positions to 0, generating the fixation data matrix M'. Then, perform convolution calculation on the fixation data matrix M' using a Gaussian convolution kernel G with a standard deviation of 0.5 and a mean of 1 corresponding to the size of the visual attention radius r, thereby generating the fixation heat map. Where: is the symbol for the convolution operation;

[0032] Data augmentation: Use the mask matrix E generated from the fixation data for binary mask processing with the original images O in the original training set, that is, multiply the mask matrix E with the original image O, and add the generated new image E·O to the training set N' to obtain the new training set N = {N', E·O}, and send them together into the model for training, thereby realizing data augmentation for the images in the data set;

[0033] Start training: For each training sample Z in N, the size of Z is a three-dimensional matrix of X*Y*T, where: X*Y is the corresponding resolution of the ultrasound image frame, and T is the size of the ultrasound image frames in this segment. Divide it into Z = {Z 1 , Z 2 , …, Z t , … Z T} in the order of frames, where t ∈ [1, T]. Among them: Z t is the t-th two-dimensional matrix of size X*Y obtained by splitting the three-dimensional matrix Z along the third dimension. Then, send it into the selected feature extraction convolutional neural network C (such as Vgg, Resnet, etc.) in sequence. The function of the neural network can be abstracted as a function C(z). According to the formula Ф t = C(Z t ) to obtain the three-dimensional matrix Ф = {Ф 1 , Ф 2 , …, Ф t , …, Ф T}. That is, send the two-dimensional matrices Z 1 , Z 2 , …, Z t , … Z T into the neural network C, and output to obtain the two-dimensional matrices Ф 1 , Ф 2 , …, Ф t , …, Ф T . The sizes of these two-dimensional matrices are X c *Y c , X c and Yc Determined by the selected convolutional neural network and combined into a three-dimensional matrix Ф in the order of the timestamp information t, with the size of the three-dimensional matrix being X c *Y c *T. Then, the three-dimensional matrix Ф is arranged in the order of the forward timestamp as the matrix Ф + ={Ф 1 , Ф 2 , …, Ф t , …, Ф T} and fed into the recurrent neural network R(Ф). According to the formula H + =R(Ф + ), the forward feature three-dimensional matrix H + is obtained. The size of the forward feature three-dimensional matrix is X r *Y r *T r , X r , Y r , T r is determined by the selected recurrent neural network and decomposed into forward feature two-dimensional matrices H r , H 1+ , …, H 2+ , …, H t+ , …, H T+ along the T - dimension; the three-dimensional matrix Ф is arranged in the order of the reverse timestamp as the matrix Ф T ={Ф t , …, Ф 1} and fed into the recurrent neural network R(Ф). According to the formula H - =R(Ф - ), the reverse feature three-dimensional matrix H - is obtained. The size of this reverse feature three-dimensional matrix is X r *Y r *T r , X r , Y r , T r is determined by the selected recurrent neural network and decomposed into reverse feature two-dimensional matrices {H r , H 1- , …, H 2- , …, H t- , …, H T-} along the T + dimension. Then, H - and H r are combined along the T + dimension, that is, the feature three-dimensional matrix H={H - , H 1+}={H 2+ , H t+ , …, H T+ , H 1- , H2- , …, H t- , …, H T-} to obtain the feature three-dimensional matrix H = {H 1 , H 2 , …, H t , …H 2*Tr} The value range of t is (1, 2*T r ), and the size of the feature three-dimensional matrix H is X r * Y r * (2*T r ). Subsequently, the feature two-dimensional matrix H t corresponding to the t-th frame is input into 3 convolutional layers and two adjustment layers to obtain the output matrix R t , and the size of R t is X*Y*M R , where X*Y is the same as the size of the input video image frame, and M R is the number of channels of the set convolutional kernel; after the adjustment layer, there are global average pooling and fully connected layers to classify the image frame at this moment; the weight parameters of the fully connected layer are extracted, multiplied by R t , and added in the M R size direction to obtain the corresponding CAM (Class Attention Map);

[0034] Early termination of training: After each round of training, the obtained CAM is compared with the fixation hotspot map M corresponding to the video sample to calculate the loss. When the average loss reaches the previously set threshold, the training can be stopped.

[0035] The above is only the preferred embodiment of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and retouches can be made, and these improvements and retouches should also be regarded as the protection scope of the present invention.

Claims

1. A method for improving the accuracy of an ultrasonic image recognition model through doctors' eye movement data, characterized in that: It includes the following steps: collecting doctors' eye movement data, creating an eye movement data matrix, generating a mask matrix, generating a fixation heat map, performing data augmentation, starting model training, and stopping training according to the fixation heat map; Eye movement data collection and processing: Use an eye tracker to collect the eye movement data of professional ultrasonic doctors when observing the ultrasonic images in the dataset, and distinguish the data points into fixation data and saccade data according to the displacement length of each collected data point. The data points within the preset step threshold range are fixation data points, and the positions where the fixation data points are located are recorded and saved as (x, y, t), where x and y are the position information of the fixation data points, and t is the time stamp information of the current image frame; Create an eye movement data matrix: Create an eye movement data matrix E', with a size of X * Y * T, where: X * Y is the corresponding resolution of the ultrasonic image frame, and T is the size of the ultrasonic image frames in this segment; Generate a mask matrix: Look at the matrix value E at the position (x, y, t) where the data point is located ’ (x, y, t) = 1, and the values at other positions are 0. According to the distance D from the doctor's eyes to the screen and the DPI of the monitor, calculate the visual attention radius And in the eye movement data matrix E', taking each fixation data point as the center, assign the value 1 to the points within the corresponding range of the visual attention radius r, so as to generate the mask matrix E; Generate a fixation heat map: In a matrix with the same size as the observed image, set the value M at the position (x, y, t) where the fixation data point is located ’ (x, y, t) = 255, and the values at other positions are 0, to generate a fixation data matrix M ’ , and then perform a convolution calculation on the fixation data matrix M' with a Gaussian convolution kernel G corresponding to the visual attention radius r, with a standard deviation of 0.5 and a mean of 1, to generate a fixation heat map Where: is the convolution operation symbol; Data augmentation: Use the mask matrix E generated from the fixation data for binary mask processing with the images O in the original training set, that is, multiply the mask matrix E by the original image O, and add the generated new image E·O to the training set N', to obtain a new training set N = {N', E·O}, and send them together into the model for training, so as to achieve data augmentation for the images in the dataset; Start training: For each training sample Z in N, the size of Z is a three-dimensional matrix of X*Y*T, where: X*Y is the corresponding resolution of the ultrasonic image frame, and T is the size of the ultrasonic image frames in this segment. Sequentially divide it into Z = {Z 1 , Z 2 , …, Z t , … Z T}, t ∈ [1, T], where: Z t is the t-th two-dimensional matrix of size X*Y obtained by splitting the three-dimensional matrix Z along the third dimension. Then, sequentially send it into the selected feature extraction convolutional neural network C. The role of the neural network is abstracted as a function C(z). According to the formula Ф t = C(Z t ), obtain the three-dimensional matrix Ф = {Ф 1 , Ф 2 , …, Ф t , …, Ф T}, that is, send the two-dimensional matrices Z 1 , Z 2 , …, Z t , … Z T into the neural network C, and output to obtain the two-dimensional matrices Ф 1 , Ф 2 , …, Ф t , …, Ф T . The size of these two-dimensional matrices is X c *Y c , and they are combined into a three-dimensional matrix Ф in the order of the timestamp information t. The size of the three-dimensional matrix is X c *Y c *T. Then, send the three-dimensional matrix Ф in the forward timestamp order matrix Ф + = {Ф 1 , Ф 2 , …, Ф t , …, Ф T} into the recurrent neural network R(Ф). According to the formula H + = R(Ф + ), obtain the forward feature three-dimensional matrix H + . The size of the forward feature three-dimensional matrix is X r *Y r *T r , and X r , Y r , T r are determined by the selected recurrent neural network, and it is decomposed into forward feature two-dimensional matrices H 1+ , H 2+ , …, H t+ , …, H T+ along the T r dimension; Send the three-dimensional matrix Ф in the reverse timestamp order matrix Ф - ={Ф T ,…,Ф t ,…,Ф 1 } is sent to the recurrent neural network R(Ф), according to the formula H - =R(Ф - ) to obtain the reverse characteristic three-dimensional matrix H - , the reverse characteristic three-dimensional matrix size is X r *Y r *T r , and press T r Dimensionality is decomposed into an inverse feature 2D matrix {H 1- ,H 2- ,…,H t- ,…,H T- }, then H + With H - Press T r Dimension combination, that is, the characteristic three-dimensional matrix H={H + , H - }={H 1+ ,H 2+ ,…,H t+ ,…,H T+ ,H 1- ,H 2- ,…,H t- ,…,H T- }Get the characteristic three-dimensional matrix H={H 1 ,H 2 ,…,H t ,…H 2*Tr }, the value range of t is (1,2*T r ), the feature three-dimensional matrix H has a size of X r *Y r *(2*T r ), and then the feature two-dimensional matrix H corresponding to the t-th frame t Input into 3 convolutional layers and two adjustment layers to get the output matrix R t , R t The size is X*Y*M R , X*Y is the same as the input video image frame size, M R is the number of channels of the set convolution kernel; the adjustment layer is followed by the global average pooling and full connection layer to classify the image frame at that moment; the weight parameters of the full connection layer are extracted and compared with R t After clicking on the multiplication, press M R Add the dimensions together to get the corresponding CAM; Early termination of training: After each round of training, calculate the loss by comparing the obtained CAM with the fixation heat map M corresponding to the video sample. When the average loss reaches the threshold set in advance, the training can be stopped.

2. The method for improving the accuracy of an ultrasonic image recognition model through doctors' eye movement data according to claim 1, characterized in that: When collecting the eye movement data, the ultrasonic doctor sits in front of the monitor and the eye tracker. The eye tracker is placed D cm below the monitor screen. After the eye tracker is calibrated, the ultrasonic images in the training set are presented on the monitor. After the ultrasonic doctor observes the image segment, an image classification judgment is made.

3. The method for improving the accuracy of an ultrasonic image recognition model through doctors' eye movement data according to claim 2, characterized in that: When making the image classification judgment, during the analysis of each image, use the eye tracker to obtain and record the eye movement data of the ultrasonic doctor within the monitor image range, that is, record the fixation coordinates x, y of each sampling of the eye tracker, the time stamp t of the corresponding ultrasonic image frame, and the displacement size s; save the eye movement data with s lower than the preset displacement threshold, and assign values to the corresponding coordinate positions in the matrix E' with the same size X * Y * T according to its coordinates (x, y, t), to create an eye movement data matrix E' with the same size as the observed image.

Images