An eye movement attention joint optimization method and system capable of improving the recognition accuracy of an ultrasound image

Abstract

The application discloses an eye movement attention joint optimization method and system capable of improving the recognition accuracy of an ultrasonic image. Eye movement data of an ultrasonic doctor observing an ultrasonic image is collected, a multi-layer learnable convolution module is used to generate an attention heat map, eye movement attention loss is calculated, a convolution network classification loss is jointly adjusted, and training is stopped according to a preset condition. The model obtained by the application does not require additional doctor eye movement data in the use process, and the output includes not only a recognition result but also accurate lesion feature attention, so that the recognition accuracy of an existing ultrasonic diagnosis and treatment system is improved, and the training cost is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of eye movement attention, and in particular to an eye movement attention joint optimization method and system for improving the accuracy of ultrasonic image recognition. BACKGROUND

[0002] Compared with X-ray imaging and CT imaging, ultrasonic imaging does not cause radioactive damage to patients because it uses mechanical waves, but the organ structure and lesions displayed by the generated images are not as clear as CT imaging, and it is difficult for ultrasonic doctors without years of experience to quickly analyze and diagnose. A major reason why ultrasonic devices cannot be widely used in today's clinical medicine is that ultrasonic doctors need to be highly trained to ensure high-quality and high-accuracy diagnostic images. Therefore, it is very meaningful to study computer-aided ultrasonic diagnosis and treatment systems.

[0003] In recent years, with the rapid development of deep learning models, the accuracy of computer vision and image recognition has been greatly improved, accelerating the application of deep learning in the field of medical image analysis and providing technical support for ultrasonic auxiliary diagnosis and treatment systems. The training of deep models requires a large number of data samples with good labels. For natural images, sample collection and classification are good, but in the medical image field, it is difficult to obtain training samples with good labels due to patient privacy and the need for professional imaging doctors to participate in image classification. For deep deep learning models, too few training sets will inevitably lead to overfitting of the trained model, insufficient generalization ability, and decreased model accuracy. Misdiagnosis can be fatal in the medical field. At the same time, another problem with deep learning models is that it is difficult to determine the basis for the model's judgment from the recognition results, i.e., it is difficult to understand in depth where the deep learning model made such a judgment in the image, which should also be avoided in the field of medical image analysis.

[0004] Existing research generates eye movement attention from doctors' eye movement information and integrates it into deep learning models to allow deep learning models to better understand the experience of doctors in recognizing ultrasonic images. However, existing methods have the following three problems: 1) Different doctors have different thinking patterns and different eye movement search patterns. A unified eye movement attention modeling method will lose the experience of professional ultrasonic doctors. 2) Existing methods integrate eye movement information by modifying the structure of deep learning models, which has poor generalization ability. 3) Existing models that integrate eye movement information still require eye movement data during use, which is costly. SUMMARY

[0005] The technical problem to be solved by the present application is to provide an eye movement attention joint optimization method for improving the recognition accuracy of ultrasound images, which does not require additional doctor eye movement data in use, and outputs both recognition results and accurate lesion feature attention, thereby improving the recognition accuracy of existing ultrasound diagnosis and treatment systems and reducing the training cost.

[0006] To solve the above technical problems, the present application provides an eye movement attention joint optimization method for improving the recognition accuracy of ultrasound images, comprising the following steps:

[0007] Step 1, eye movement data acquisition; using an eye tracker to obtain and record the eye movement data of an ultrasound doctor within the image range of a display;

[0008] Step 2, eye movement attention modeling; using an eye movement attention generation module to generate a gaze data matrix G according to the eye movement data obtained in step 1, and calculating an eye movement model heat map M attn according to a convolution layer;

[0009] Step 3, training process; sending an ultrasound image into a convolutional neural network for training, using a class activation mapping to generate a saliency map M cam and comparing it with the heat map M attn of the eye movement attention module, calculating the attention loss; comparing the output of the convolutional neural network with the true value prediction result, calculating the classification loss of the model;

[0010] Step 4, forward feedback; adjusting the convolution layer parameters in the eye movement attention module according to the attention loss, generating a heat map, and calculating the intersection over union (IOU); comparing the IOU and the loss, taking the optimal convolution layer parameters for saving and used for the next round of model training; adjusting the convolutional neural network model parameters according to the classification loss.

[0011] Preferably, in step 1, in the eye movement data acquisition, the eye tracker is used to obtain and record the eye movement data of the ultrasound doctor within the image range of the display during the analysis of each image, i.e. recording the gaze coordinates x, y, the corresponding ultrasound image frame timestamp t and the displacement size s of each sampling of the eye tracker; saving the eye movement data with a displacement size s lower than a pre-set displacement threshold, and assigning values to the corresponding coordinate positions in a matrix E' with a size of X*Y*T corresponding to the coordinates (x, y, t), to create an eye movement data matrix E' with the same size as the observation image.

[0012] Preferably, in step 2, in the matrix with the same size as the observation image, the value M'(x, y, t) of the position (x, y, t) where the gaze data point is located is set to 255, and the values of the remaining positions are set to 0, to generate a gaze data matrix G, which is calculated by the eye movement attention module to obtain the output feature The eye movement attention generation module is a multi-layer learnable convolution structure, which is a stack of convolution layers of different sizes, and the parameters of the convolution layers are dynamically adjusted in the forward feedback process. There are various structures as model hyperparameters for selection, which better simulate the visual attention of the sonographer.

[0013] Preferably, in step 3, for each training sample image X, the size of X is a two-dimensional image of i*j, where i and j are the resolution corresponding to the ultrasound image, and the image size is reshaped to 224*224 and input into the convolutional neural network for training. The neural network is divided into a convolution feature extraction layer, a pooling layer, a full connection layer, and an output layer. Each network layer is abstracted as a function. After the image is subjected to convolution feature extraction Conv(x), the output feature C is obtained, and the size of C is H*W*K, H and W are feature sizes, and K is a feature dimension. The model class activation mapping (CAM) M is calculated according to the output feature C cam , which is used to compare the attention effect of the model. The CAM calculation formula is as follows:

[0014]

[0015] C k is the kth feature, is the weight of the kth feature corresponding to the cth class, and Relu is the Relu function, and the formula is as follows:

[0016] Relu(+)=max(0,+)

[0017] The output feature C is input into the remaining network layer in turn to obtain the training result 01(+ (t) ) of the tth round.

[0018] Preferably, in step 4, after obtaining the result of each round of training, the overall loss of the model is calculated. The loss function is composed of two parts, i.e., eye movement attention loss and classification loss, and the formula is as follows

[0019]

[0020] A and B are model hyperparameters, A is the type of convolutional neural network, B is the type of multi-layer learnable kernel, M is the attention true value result, 8 is the classification true value result, G is the visual attention heat map, is the classification loss, is the eye movement attention loss. The calculation formula of is as follows:

[0021]

[0022] The calculation formula of is as follows:

[0023]

[0024] Based on the overall loss of the model, the network model uses forward feedback to jointly adjust the parameters of the eye movement attention module and the convolutional neural network. When the training reaches the specified round or the loss is less than the average value, the model training is terminated. During the use phase, the trained model only needs to input ultrasound images to obtain classification results and lesion feature attention maps to achieve the purpose of auxiliary diagnosis.

[0025] Accordingly, an eye movement and attention joint optimization system that can be used to improve the accuracy of ultrasound image recognition includes: an eye movement data acquisition module, an eye movement attention generation module, a joint training optimization module and a forward feedback module; the eye movement data acquisition module uses an eye tracker to obtain and record the eye movement data of the ultrasound doctor within the image range of the display, the eye movement attention generation module generates a gaze data matrix G based on the obtained eye movement data, the joint training optimization module jointly optimizes the eye movement data and ultrasound data, and the forward feedback module calculates the overall loss of the model after obtaining the results of each round of training.

[0026] The beneficial effects of the present invention are as follows: the present invention uses the eye movement data collected from ultrasound doctors when observing ultrasound images to generate an attention heat map, calculates the eye movement attention loss, and jointly adjusts it with the classification loss of the convolutional network. The training is stopped according to preset conditions. The obtained model does not require additional doctor's eye movement data during use, and the output includes both recognition results and accurate lesion feature attention, thereby improving the recognition accuracy of existing ultrasound diagnosis and treatment systems and reducing training costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 Schematic diagram of the model network structure of the present invention.

[0028] Figure 2 Schematic diagram of four multi-layer learnable convolutional structures of the present invention.

[0029] Figure 3 This is a schematic diagram of the flow chart of the ultrasound image recognition model used in the present invention. DETAILED DESCRIPTION

[0030] like Figure 1 As shown in FIG, a joint optimization method of eye movement and attention for improving the accuracy of ultrasound image recognition includes the following steps:

[0031] Step 1, eye movement data acquisition; during the analysis process of each image, the eye tracker is used to obtain and record the eye movement data of the sonographer within the display image range, i.e. to record the eye tracker's sampling gaze coordinates x, y, corresponding to the ultrasound image frame timestamp t and displacement size s; save the eye movement data with s below the pre-set displacement threshold, and assign values to the corresponding coordinate positions in the matrix E' with size X*Y*T corresponding to the coordinates (x, y, t), to create an eye movement data matrix E' with the same size as the observation image.

[0032] Step 2, eye movement attention modeling; in the matrix with the same size as the observation image, set the value of the position (x, y, t) where the gaze data point is located as M'(x, y, t) = 255, and the values of the remaining positions as 0, to generate a gaze data matrix G, which is input into the eye movement attention modeling module for calculation, to obtain the output feature As Figure 2 shown, the eye movement attention modeling module is a multi-layer learnable convolution structure, which is a stack of convolution layers with different sizes, and the parameters of the convolution layers are dynamically adjusted in the forward feedback process. There are various structures as model hyperparameters for selection, which better simulate the visual attention of the sonographer.

[0033] Step 3, training process; for each training sample image X, the size of X is i*j two-dimensional image, where i and j are the resolution corresponding to the ultrasound image, the image size is reshaped to 224x224 and normalized to the [0, 1] interval, and then input into the convolutional neural network for training; the neural network is divided into convolution feature extraction layer, pooling layer, full connection layer and output layer, each network layer is abstracted as a function, after the image is subjected to convolution feature extraction Conv(x), the output feature C is obtained, the size of C is HxWxK, H and W are feature sizes, K is feature dimension, and the model class activation mapping (CAM) M cam is calculated according to the output feature C, which is used to compare the attention effect of the model, and the CAM calculation formula is as follows:

[0034]

[0035] C k is the kth feature, is the weight of the kth feature corresponding to the cth class, and Relu is the Relu function, and the formula is as follows:

[0036] Relu(+) = max(0, +)

[0037] The output feature C is input into the remaining network layers in turn to obtain the training result 01(+ (t) ) of the tth round.

[0038] Step 4, forward feedback; after obtaining the result of each round of training, the overall loss of the model is calculated, and the loss function is composed of two parts of eye movement attention loss and classification loss, and the formula is as follows

[0039]

[0040] A and B are model hyperparameters, A is a convolutional neural network category, B is a multi-layer learnable kernel category, M is an attention true value result, 8 is a classification true value result, and G is a visual attention heat map, is a classification loss, is an eye movement attention loss; The calculation formula is as follows:

[0041]

[0042] The calculation formula is as follows:

[0043]

[0044] According to the overall loss of the model, the network model is forwardly fed back to jointly adjust the parameters of the eye movement attention module and the convolutional neural network, and the training is stopped when the specified number of rounds is reached or the loss is less than the average value; as shown in the formula: Figure 3 The trained model is used in the use stage, that is, the classification result and the lesion feature attention map can be obtained by inputting the ultrasound image, so as to achieve the purpose of auxiliary diagnosis.

[0045] Correspondingly, an eye movement attention joint optimization system for improving the recognition accuracy of an ultrasound image comprises an eye movement data acquisition module, an eye movement attention generation module, a joint training optimization module and a forward feedback module; the eye movement data acquisition module obtains and records the eye movement data of an ultrasound doctor within the image range of a display by using an eye tracker, the eye movement attention generation module generates a gaze data matrix G according to the obtained eye movement data, the joint training optimization module jointly optimizes the eye movement data and ultrasound data, and the forward feedback module calculates the overall loss of the model after obtaining the result of each round of training.

[0046] The eye movement data of the ultrasound doctor observing the ultrasound image is collected, the attention heat map is generated through the multi-layer learnable convolution module, the eye movement attention loss is calculated, the classification loss of the convolution network is jointly adjusted, the training is stopped according to the preset condition, the obtained model does not need additional doctor eye movement data in the use process, and the output includes the recognition result and the accurate lesion feature attention, so that the recognition accuracy of the existing ultrasound diagnosis and treatment system is improved, and the training cost is reduced.

Claims

1. An eye movement attention joint optimization method capable of improving the recognition accuracy of an ultrasound image, characterized in that, Comprising the following steps: Step 1, eye movement data acquisition; use an eye tracker to obtain and record the eye movement data of the sonographer within the display image range; Step 2, eye movement attention modeling; generate gaze data matrix G using eye movement attention generation module according to the eye movement data obtained in step 1, and calculate eye movement model heat map M according to the convolution layer attn ; Step 3, training process; send the ultrasound image into the convolutional neural network for training, and use the class activation mapping to generate the saliency map M for the convolutional feature extraction layer cam , and the heat map M of the eye movement attention module attn Comparison, calculate the attention loss; compare the output of the convolutional neural network with the true value prediction result, and calculate the classification loss of the model; Step 4, forward feedback; according to the attention loss, adjust the convolution layer parameters in the eye movement attention module, generate a heat map, and calculate the intersection over union IOU; compare IOU and loss, take the optimal convolution layer parameters, and save them for the next round of model training; according to the classification loss, adjust the convolutional neural network model parameters; after obtaining the results of each round of training, calculate the overall loss of the model, and the loss function consists of two parts: eye movement attention loss and classification loss, as follows θ and φ are model hyperparameters, θ is a convolutional neural network species, φ is a multi-layer learnable kernel species, M is an attention ground truth result, y is a classification ground truth result, G is a visual attention heat map, is a classification loss, is an eye movement attention loss; The calculation formula is as follows: The calculation formula is as follows: According to the overall loss of the model, the network model forward feedback adjusts the parameters of the eye movement attention module and the convolutional neural network, and the training reaches the specified number of rounds or the loss is less than the average value, and the model terminates training; the trained model only inputs the ultrasound image in the use stage, and the classification result and lesion feature attention map can be obtained to achieve the purpose of auxiliary diagnosis.

2. The eye movement attention joint optimization method for improving the recognition accuracy of an ultrasound image according to claim 1, wherein, In step 1, in the eye movement data acquisition, the eye tracker is used to obtain and record the eye movement data of the sonographer within the display image range during the analysis of each image, that is, the gaze coordinates x, y of each sampling of the eye tracker are recorded, corresponding to the ultrasound image frame timestamp t and the displacement size s; the eye movement data with s below the pre-set displacement threshold is saved, and the coordinates (x, y, t) are assigned to the corresponding coordinate position of the matrix E' with the same size as the observation image, that is, the eye movement data matrix E' with the same size as the observation image is created.

3. The eye movement attention joint optimization method for improving the recognition accuracy of an ultrasound image according to claim 1, wherein, In step 2, in the same matrix as the observation image size, let the value of the gaze data point at the position (x, y, t) M'(x, y, t) = 255, and the values of the remaining positions are 0, to generate a gaze data matrix G, and the gaze data matrix is input into the eye movement attention module The output feature is calculated as 4. The eye movement attention joint optimization method for improving the recognition accuracy of an ultrasound image according to claim 1, wherein, In step 3, for each training sample image X, the size of X is i*j two-dimensional image, where i and j are the resolution of the ultrasound image, and the image size is reshaped to 224x224 and sent to the convolutional neural network for training.

5. The eye movement attention joint optimization method for improving the recognition accuracy of an ultrasound image according to claim 4, characterized in that, The neural network is divided into a convolution feature extraction layer, a pooling layer, a full connection layer, and an output layer. Each network layer is abstracted into a function. After image convolution feature extraction Conv(x), an output feature C is obtained. The size of C is HxWxK. H and W are feature sizes, and K is a feature dimension. A model class activation map (CAM) M is calculated according to the output feature C cam , which is used to compare the attention effect of the model. The CAM calculation formula is as follows: C k is the kth feature, is the weight of the kth feature corresponding to the cth class, and Relu is the Relu function, which is as follows: Relu(x) = max(0, x) The output feature C is sequentially input to the remaining network layer to obtain the training result f of the tth round θ (x (t) ).

6. A system for applying the eye movement attention joint optimization method for improving the recognition accuracy of an ultrasound image according to claim 1, characterized in that, Comprising: An eye movement data acquisition module, an eye movement attention generation module, a joint training optimization module, and a forward feedback module; The eye movement data acquisition module uses an eye tracker to obtain and record the eye movement data of the sonographer within the display image range, the eye movement attention generation module generates a gaze data matrix G according to the obtained eye movement data, the joint training optimization module optimizes the eye movement data and ultrasound data jointly, and the forward feedback module calculates the overall loss of the model after obtaining the results of each round of training.

7. The eye movement attention joint optimization system for improving the recognition accuracy of ultrasound images of claim 6, wherein, According to the overall loss of the model, the network model forward feedback adjusts the parameters of the eye movement attention module and the convolutional neural network, and the training reaches the specified number of rounds or the loss is less than the average value, and the model terminates training; the trained model only inputs the ultrasound image in the use stage, and the classification result can be obtained to achieve the purpose of auxiliary diagnosis.

Images