Multi-attention mechanism network model and thyroid ultrasound contrast focus segmentation method and system

Abstract

The invention discloses a multi-attention mechanism network model and a thyroid ultrasound contrast focus segmentation method and system based on a multi-attention mechanism network. The multi-attention mechanism network model adopts an RFM-ResNeXt module to extract spatial and temporal feature information of an input video; aggregating the three-dimensional feature maps into a two-dimensional feature map through a bidirectional convolution LSTM module; inputting the two-dimensional feature map into a double attention module (DAM) to obtain an attention feature map; and carrying out convolution and up-sampling on the attention feature map to obtain a binary segmentation result image. The time-space information of the ultrasound contrast video can be effectively extracted by utilizing three attention mechanisms of channel attention, time attention and space attention, and the ablation area in the thyroid ultrasound contrast image can be automatically and accurately segmented.

Description

Technical Field

[0001] The present invention relates to the field of intelligent diagnosis technology, and more specifically, to a multi-attention mechanism network model, a thyroid ultrasound angiography lesion segmentation method and system. Background Art

[0002] In recent years, microwave ablation technology has gradually become an effective means of treating thyroid nodules. In order to prevent nodule hyperplasia and parathyroid dysfunction in the ablation area after thyroid ablation, postoperative follow-up of the thyroid ablation area is very important. At the same time, due to differences in regional medical levels, many local hospitals cannot accurately distinguish between the lesion area after ablation and the calcified thyroid nodules, which can easily lead to misjudgment. Therefore, it is very important to propose an automatic segmentation algorithm for thyroid ablation lesions based on ultrasound contrast images for clinical auxiliary diagnosis.

[0003] Contrast-enhanced ultrasound (CEUS) imaging is a commonly used diagnostic tool for follow-up after thyroid ablation surgery. Clinicians can diagnose the recovery of the ablation area based on CEUS images. Image segmentation technology can achieve accurate automatic segmentation of the thyroid ablation area in CEUS images to assist doctors in diagnosis. Currently, some image segmentation methods based on convolutional neural networks (CNNs) have been proposed for segmenting organs and lesions in ultrasound images. However, these methods cannot effectively model the spatiotemporal information between adjacent frames of continuous image sequences and do not consider the different importance of different features on the image for image segmentation. Summary of the invention

[0004] In order to solve the problem of difficulty in segmenting lesions in thyroid ultrasound contrast images, the present invention designs a multi-attention mechanism network model and a thyroid ultrasound contrast lesion segmentation method and system based on the multi-attention mechanism network according to the changes of contrast agent around the ablation area after thyroid ablation, so as to automatically and accurately segment the ablation area in thyroid ultrasound contrast images.

[0005] To achieve the above objectives, in a first aspect, the present invention provides a Multi-Attention Mechanism Network (MAM-Net) model, characterized in that it is a segmentation network structure:

[0006] A 3D convolutional neural network is used as the encoder, and a 2D convolutional neural network is used as the decoder. The input data is a 3D video clip, and the output result is a 2D segmentation result image.

[0007] The RFM-ResNeXt module is used to extract the spatiotemporal feature information of the input video; the three-dimensional feature map is aggregated into a two-dimensional feature map through a bidirectional convolution LSTM (Long Short-Term Memory) module; the two-dimensional feature map is input into a dual attention module DAM to obtain an attention feature map; the attention feature map is convolved and upsampled to obtain a binary segmentation result image;

[0008] The RFM-ResNeXt module includes five submodules: conv1, res1, res2, res3, and res4. Conv1 consists of two consecutive three-dimensional convolutions, group normalization, and ReLU activation functions; res1, res2, res3, and res4 consist of 3, 4, 6, and 3 consecutive residual fusion modules RFM, respectively;

[0009] The dual attention module DAM includes a spatial attention module PAM and a channel attention module CAM.

[0010] The MAM-Net model includes channel attention, temporal attention and spatial attention. First, the spatial and temporal features of the ultrasound contrast imaging sequence are extracted by integrating the three-dimensional CNN with the channel attention mechanism, and then the three-dimensional feature map is given temporal attention using the bidirectional convolutional long short-term memory network and converted into a two-dimensional feature map. Then, the DAM module is used to apply spatial attention and channel attention to the output two-dimensional feature map, and finally, the output two-dimensional feature map is convolved and upsampled to obtain the segmentation result. The spatial attention submodule can solve the global spatial position dependency in the two-dimensional image, so that the current pixel can establish a connection with the pixel at a farther distance. The channel attention submodule solves the global channel dependency and can establish a connection between each channel, so as to segment the ablation zone more accurately.

[0011] Furthermore, the RFM module is expressed as ,in Represents the direct mapping part; Represents the residual part, which consists of three layers of three-dimensional convolution with kernels of 1, 3, and 1 respectively; represents the fusion module; this module will and The result of the addition Y is input to the CAM module. The CAM module includes global and local channel attention. The global channel attention performs two convolution operations with a convolution kernel of 1 after global average pooling on the input Y. The local channel attention directly performs two convolution operations with a convolution kernel of 1 on the input Y. Then, the output results of the global channel attention and the local channel attention are added together, and then the added result is passed through the Sigmoid activation function. Finally, the output results of the activation function are respectively compared with and After multiplication, we get The residual block uses a fusion module RFM based on global and local channel attention to perform feature fusion in the residual and direct mapping, which can aggregate the local and global context features of the feature map, and to some extent make up for the defect that CNN cannot model global semantic information well. When downsampling the three-dimensional convolution, the time dimension is not reduced, but only the spatial dimension is reduced, which avoids the information damage in the time dimension caused by the pooling operation.

[0012] Furthermore, the bidirectional convolutional LSTM module includes five basic units, namely, input gate, forget gate, output gate, cell state and hidden state, and then 16 basic components are connected end to end to form a loop structure. For the input three-dimensional data, the module adds time attention on the time dimension through the gated signal, which can better learn the spatial information along the time dimension, and obtain a two-dimensional feature map by splicing the last layer of the forward LSTM and the reverse LSTM on the channel.

[0013] Furthermore, the PAM module first inputs After three two-dimensional convolutions with a kernel of 1, we get three feature maps Q, K, and V, and transform Q into Then transpose to get Feature map of size (N is the product of H and W), K is transformed into Feature map of size , and then the obtained and After matrix multiplication, we get Feature map of size ; Transform V into Feature map After After multiplying the transposed matrix of The feature map of size is then transformed into Feature map of size ,Will and Add to get the feature map ; The CAM module will input The deformation size is Feature map and and size Feature map ,Will and After matrix multiplication, a layer of softmax function is used to obtain a size of Feature map ;Will After multiplying with the B matrix, we get The feature map is then transformed into Feature map of size ,Will and After adding, we get the feature map .

[0014] In a second aspect, the present invention provides a thyroid ultrasound contrast imaging lesion segmentation method, characterized in that the lesion segmentation in the thyroid ultrasound contrast imaging image is performed using the trained multi-attention mechanism network model as described above, comprising the following steps:

[0015] S1. The ablation area of ​​the patient's thyroid after microwave ablation has no biological activity. The patient's thyroid ablation ultrasound angiography data is preprocessed to obtain a data set: first, the image is annotated, and then the spatial dimension is cropped and the time dimension is divided;

[0016] S2, divide the data set obtained in S1 into a training set and a validation set in a ratio of 8:2;

[0017] S3, input the training set and validation set in S2 into the multi-attention mechanism network model as described above to perform model training, and obtain the network parameters that optimize the model performance;

[0018] S4. Use the multi-attention mechanism network model with the best performance obtained in S3 to segment the real-time thyroid ultrasound angiography video to achieve the discrimination and analysis of thyroid nodules and ablation areas.

[0019] Furthermore, the step S1 includes the following sub-steps:

[0020] S1.1. Clinical experts annotate the ablation area of ​​thyroid lesions to obtain the corresponding binary image annotation file;

[0021] S1.2, using the threshold method to crop the spatial dimension of each frame of the ultrasound contrast imaging video and its annotated file;

[0022] S1.3, dividing the ultrasound contrast imaging video processed in S1.2 and the corresponding annotation file into video segments with a fixed frame length of a certain number of frames; if the number of video frames in the last division of the video is insufficient, the division is removed to obtain a certain number of video segments and corresponding binary annotation images.

[0023] Furthermore, in step S3, the dice loss error between the predicted image and the real annotation of the multi-attention mechanism network model is back-propagated through the back-propagation algorithm, and the optimal network model is obtained through continuous iteration; the calculation formula of the dice loss function is expressed as: , where X represents the predicted result image and Y represents the expert labeled image.

[0024] In the last aspect, the present invention provides a thyroid ultrasound contrast imaging lesion segmentation system, characterized in that it comprises:

[0025] The data acquisition module collects the patient's real-time thyroid ultrasound contrast video; the ablation area of ​​the patient's thyroid after microwave ablation has no biological activity;

[0026] The multi-attention mechanism network model module described above uses the data set to train the model until the network parameters are optimal;

[0027] The lesion segmentation module inputs the video collected by the data acquisition module into the trained multi-attention mechanism network model, and outputs a two-dimensional image showing the segmented lesions.

[0028] Compared with the prior art, the present invention has the following technical effects:

[0029] (1) The present invention proposes a multi-attention mechanism network model for lesion segmentation of thyroid ultrasound contrast imaging. By utilizing three attention mechanisms, namely channel attention, temporal attention and spatial attention, the spatiotemporal information of ultrasound contrast imaging videos can be effectively extracted. The spatial attention submodule can solve the global spatial position dependency in the two-dimensional image, so that the current pixel can establish a connection with pixels at a longer distance; the channel attention submodule adjusts the importance of different channels in the feature map, so that the network model can focus on more important features, thereby more accurately segmenting the ablation area; the bidirectional convolutional LSTM network module adds temporal attention in the temporal dimension through the gating signal, and can learn spatial information along the temporal dimension like CNN. Finally, the two-dimensional feature map is obtained by splicing the last layer of the forward convolutional LSTM and the reverse convolutional LSTM on the channel.

[0030] (2) The present invention can perform end-to-end training and prediction on the multi-attention mechanism network, thereby accurately and automatically segmenting the thyroid lesion ablation area in the thyroid ultrasound angiography image. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is an overall structural diagram of a multi-attention mechanism network module in one embodiment of the present invention.

[0032] Figure 2 Schematic diagram of the structure of a residual fusion module in one embodiment of the present invention.

[0033] Figure 3 Schematic diagram of the structure of the dual attention module in one embodiment of the present invention.

[0034] Figure 4The figure is a flow chart of a method for segmenting thyroid ultrasound angiography lesions in one embodiment of the present invention.

[0035] Figure 5 It is a schematic diagram of the effect of automatic segmentation of thyroid ultrasound contrast imaging lesions in one embodiment of the present invention. DETAILED DESCRIPTION

[0036] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, but they are not intended to limit the present invention.

[0037] In the following detailed description, many specific details are set forth to provide a more thorough understanding of the present invention. However, it is obvious to those skilled in the art that well-known algorithms and models (such as three-dimensional CNN, two-dimensional CNN and LSTM network) are not shown in detail to avoid obscuring the subject matter of the present invention.

[0038] Example 1

[0039] See also Figure 1 , this embodiment provides a multi-attention mechanism network (MAM-Net) model, which is a segmentation network structure, which is constructed based on a three-dimensional CNN, a two-dimensional CNN and a bidirectional convolutional LSTM network structure: a three-dimensional convolutional neural network is used as an encoder, a two-dimensional convolutional neural network is used as a decoder, the input data is a three-dimensional video clip, and the output result is a two-dimensional segmentation result image; an RFM-ResNeXt module is used to extract the spatiotemporal feature information of the input video; a three-dimensional feature map is aggregated into a two-dimensional feature map through a bidirectional convolutional LSTM module; the two-dimensional feature map is input into a dual attention module DAM to obtain an attention feature map; the attention feature map is convolved and upsampled to obtain a binary segmentation result image; the RFM-ResNeXt module includes five submodules, namely, conv1, res1, res2, res3 and res4, wherein conv1 is composed of two consecutive three-dimensional convolutions, group normalization and ReLU activation functions; res1, res2, res3 and res4 are respectively composed of 3, 4, 6 and 3 consecutive residual fusion modules RFM; the dual attention module DAM includes a spatial attention module PAM and a channel attention module CAM.

[0040] The following is a more specific construction principle and training process of the MAM-Net model.

[0041] (1) Combining the residual fusion module (RFM) and the ResNeXt network, a three-dimensional CNN network RFM-ResNeXt module is proposed to extract the spatiotemporal feature information of the input video. The RMF-ResNeXt module contains five three-dimensional convolution blocks, namely conv1, res1, res2, res3 and res4. Conv1 consists of two consecutive three-dimensional convolutions, group normalization and ReLU activation functions; res1, res2, res3 and res4 are composed of 3, 4, 6 and 3 consecutive RFM modules respectively. Figure 2 is the RFM structure diagram. The RFM module can be expressed as ,in Represents the direct mapping part; Represents the residual part, which consists of three layers of three-dimensional convolution with kernels of 1, 3, and 1 respectively. The convolution with kernel 3 uses group convolution with group 32. Represents the fusion module, which and The result of the addition, Y, is input into the Channel Attention Module (CAM). The CAM module includes global and local channel attention. The global channel attention performs two convolution operations with a kernel of 1 after global average pooling on the input Y. The local channel attention directly performs two convolution operations with a kernel of 1 on the input Y. Then, the output of the global channel attention and the output of the local channel attention are added together, and then the result of the addition is passed through the Sigmoid activation function. Finally, the output of the activation function is respectively compared with and After multiplication, we get The network parameters of the RFM-ResNeXt module are shown in Table 1. Considering that the time length is small, downsampling it will cause loss of time information. Therefore, the time dimension is not downsampled in each convolution module, and only the spatial resolution is downsampled by one time. After the image sequence of size 16x256x256 input from S2 passes through the RFM-ResNeXt module, the outputs corresponding to the five submodules of conv1, res1, res2, res3, and res4 are obtained. , where the value of b is 1, indicating the number of batches, and t is 16, indicating 16 frames of continuous images; They are 64, 128, 256, 512, 1024 respectively; and They are 128, 64, 32, 16, and 8, respectively, indicating the spatial resolution of the output of each layer.

[0042] Table 1 Detailed parameters of RFM-ResNeXt module

[0043]

[0044] (2) Replace res1 in step (1) 4 three-dimensional feature maps obtained by res4 Input BiConvLSTM1 with corresponding number of channels and spatial resolution respectively BiConvLSTM4 module. BiConvLSTM stands for Bi-directional Convolutional Long Short-Term Memory network, which includes a forward convolutional LSTM network and a reverse convolutional LSTM network. They are both composed of convolutional LSTM network modules. The reverse convolutional LSTM inputs the input data of the forward LSTM in reverse order into the convolutional LSTM. The bidirectional convolutional LSTM module includes five basic components consisting of input gate, forget gate, output gate, cell state and hidden state, and then 16 basic components are connected end to end to form a loop structure. Convolutional LSTM replaces the Hadamard product in the LSTM network with a convolution calculation, that is:

[0045]

[0046] in represents a two-dimensional convolution, represents the Hadamard product, Represents input data, represents the input gate, represents the forget gate, represents the output gate, Indicates the cell state, Represents the hidden layer state. After passing through the BiConvLSTM-i module, a two-dimensional feature map with the corresponding number of channels and resolution is obtained. . Where b takes the value of 1, They are 128, 256, 512, 1024 respectively. and They are 64, 32, 16, and 8 respectively.

[0047] (3) The four two-dimensional feature maps obtained in step (2) Input into the dual attention module DAM1 respectively In DAM4, the dual attention module (DAM) includes the position attention module (PAM) and the channel attention module (CAM). The schematic diagram of its structure is shown in Figure 3 Schematic diagram of the dual attention module.

[0048] The PAM submodule first inputs After three two-dimensional convolutions with a kernel of 1, we get three feature maps Q, K, and V, and transform Q into Then transpose to get Feature map of size (N is the product of H and W), K is transformed into Feature map of size , and then the obtained and After matrix multiplication, we get Feature map of size ; Transform V into Feature map After After multiplying the transposed matrix of The feature map of size is then transformed into Feature map of size ,Will and Add to get the feature map .

[0049] The CAM submodule will input The deformation size is Feature map and and size Feature map ,Will and After matrix multiplication, a layer of softmax function is used to obtain a size of Feature map .Will After multiplying with the B matrix, we get The feature map is then transformed into Feature map of size ,Will and After adding, we get the feature map .

[0050] The DAM module converts the feature map and After adding, we get the feature map . Input data After passing through the corresponding DAM module, a feature map of the same size is obtained .

[0051] (4) The feature map obtained in step (3) Input to decoder module Decoder4 In Decoder1, the prediction result is obtained after passing through the UpConv and OutConv modules. The structure of Decoder is as follows Figure 2 As shown in the Decoder module in, it contains a layer Convolution, one layer Convolution and one layer Convolution, where each convolution layer is followed by a group normalization layer and a ReLU activation function layer. Decoder4 The corresponding input channels of Decoder1 are 1024, 512, 256, 128 and the corresponding output channels are 512, 256, 128, 64. After adding, we get ;Will After the output of Decoder3 and Add together to get ;Will After the output of Decoder2 Add together to get ;Will After Decoder1, we get , and then After an UpConv module, the feature map is obtained , where the UpConv module consists of an upsampling layer and a convolution layer with a convolution kernel of 1. After a convolution layer OutConv with a convolution kernel of 1, a feature map with a channel number of 1 is obtained, which is input into the Sigmoid function to obtain the predicted segmentation result image.

[0052] (5) The error loss between the predicted image and the true annotation in step (4) is back-propagated through the back-propagation algorithm. After multiple rounds of iterations, the network model parameters with the minimum error are finally obtained. The error loss function used is the Dice loss function, and its calculation formula is expressed as , where X represents the predicted result image and Y represents the expert labeled image.

[0053] The segmentation results of the multi-attention mechanism network model are tested through the test set, and the evaluation index of the segmentation is calculated to evaluate the performance of the model segmentation. The evaluation indexes used in this embodiment include the dice coefficient (DSC), the intersection over union coefficient (IOU), the precision rate (Precision) and the recall rate (Recall), and the calculation formulas are: , , , ,in Indicates the correct part of the thyroid ablation area prediction. It indicates that the part which is not actually the ablation area is predicted to be the ablation area. This indicates the fact that the ablation area is predicted to be part of the black background.

[0054] Example 2

[0055] See also Figure 4 This embodiment provides a thyroid ultrasound contrast imaging lesion segmentation method, which uses the trained multi-attention mechanism network model as described in Example 1 to perform lesion segmentation in thyroid ultrasound contrast imaging images, including the following steps:

[0056] S1. The ablation area after microwave ablation of the patient's thyroid gland is biologically inactive, that is, the contrast agent cannot flow in the ablation area but can flow in normal organs and tissues. Therefore, by observing the perfusion of microbubble contrast agent next to the ablation area, the ablation area and the non-ablation area can be distinguished. The patient's thyroid ablation ultrasound contrast data were preprocessed to obtain a data set: first, the image was annotated, and then the spatial dimension was cropped and the time dimension was divided.

[0057] More specifically, step S1 includes the following sub-steps:

[0058] S1.1. Clinical experts use Pair annotation software to annotate the ablation area of ​​thyroid lesions to obtain the corresponding binary image annotation file.

[0059] S1.2. Use the threshold method to crop the spatial dimension of each frame of the ultrasound contrast imaging video and its annotated file. Specifically, average the pixel values ​​of the original image along the x-axis and y-axis, remove the rows and columns with an average value less than 5, and perform corresponding operations on the annotated binary image. Then deform the cropped image to obtain an image with a fixed spatial resolution of 256×256 pixels.

[0060] S1.3, divide the ultrasound contrast imaging video processed in S1.2 and the corresponding annotation file into video segments with a fixed frame length of several frames (here 16 frames); if the number of video frames in the last division of the video is less than 16 frames, remove the division. For the 16 frames of annotation data obtained from each division, since they are 16 frames of binary annotated images with the same content, and only one two-dimensional annotation is required as a label in network training, the first frame annotation is taken as the label. After the above processing, several 16×256×256 video segments and corresponding 256×256 binary annotated images are obtained.

[0061] S2. Divide the data set obtained in S1 into a training set and a validation set in a ratio of 8:2. In order to increase the robustness of the model, the data in the training set are horizontally flipped, rotated 15 degrees, and normalized.

[0062] S3, input the training set and validation set in S2 into the multi-attention mechanism network model as described above to perform model training, and obtain the network parameters that optimize the model performance. Figure 1 As shown, similar to Example 1, MAM-Net first uses the RFM-ResNeXt network module based on three-dimensional CNN to extract the spatiotemporal features of the input data, and then uses the bidirectional convolutional LSTM network module to pay time attention to the three-dimensional feature map to convert it into a two-dimensional feature map. Then, the DAM module is used to pay attention to the space and channel of the two-dimensional feature map, and finally the output feature map is decoded by the decoder module to obtain the predicted segmentation result.

[0063] (a) The RFM-ResNeXt network module based on 3D CNN contains a convolutional block conv1 and four residual blocks res1 res4, the residual block uses the fusion module RFM based on global and local channel attention to perform feature fusion in the residual and direct mapping. According to the experience of clinicians, it is usually necessary to observe the dynamic changes of the contrast agent around the thyroid ablation area to obtain the boundary of the ablation zone. Therefore, the segmentation of the ablation area needs to consider both temporal and spatial features. Three-dimensional convolution can extract temporal and spatial feature information at the same time. Using local and global channel attention in the RFM module can aggregate the local and global contextual features of the feature map, which to some extent makes up for the defect that CNN cannot model global semantic information well. Since the time dimension is only 16 frames, the time dimension is not reduced when the three-dimensional convolution is downsampled, and the dimension is only reduced in the spatial dimension, which avoids the information damage in the time dimension caused by the pooling operation.

[0064] (b) Replace res1 in (a) 3D feature map output by res4 Input time attention module BiConvLSTM1 BiConvLSTM4 gets Two-dimensional feature map. The LSTM network has a memory function, which controls the transmission state through the gate controller, remembers the information that needs to be remembered for a long time in the time sequence, and forgets the unimportant information. The convolutional LSTM network replaces the Hadamard product in the LSTM network with a convolution calculation, which can process the time sequence relationship of the image sequence. The bidirectional LSTM includes the forward convolutional LSTM and the reverse convolutional LSTM. The forward LSTM models the input data sequentially in the time dimension, and the reverse convolutional LSTM models the input data in reverse order in the time dimension. For the input three-dimensional data, this module adds time attention in the time dimension through the gating signal, which can better learn the spatial information along the time dimension. The two-dimensional feature map is obtained by splicing the last layer of the forward LSTM and the reverse LSTM on the channel.

[0065] (c) The two-dimensional feature map output in (b) Input into the dual attention module DAM1 respectively Get the two-dimensional feature map in DAM4 The DAM module includes a spatial attention submodule and a channel attention submodule. The spatial attention submodule can solve the global spatial position dependency in the two-dimensional image, so that the current pixel can establish a connection with pixels at a farther distance. The channel attention submodule solves the global channel dependency and can establish a connection between each channel, thereby segmenting the ablation area more accurately.

[0066] (d) The feature map obtained in (c) The input is sent to the decoder module, which upsamples the two-dimensional feature map layer by layer to obtain a predicted segmentation image with the same spatial resolution as the input data.

[0067] (e) The dice loss error between the predicted image and the true annotation in (d) is back-propagated through the back-propagation algorithm, and the optimal network model is obtained through continuous iteration.

[0068] S4. See Figure 5 , the best-performing multi-attention network model obtained in S3 is used to segment the real-time thyroid ultrasound contrast video, and the discrimination and analysis of thyroid nodules and ablation areas are achieved. The segmentation results are very similar to the results of the expert gold standard, with high segmentation accuracy. The three-dimensional CNN and bidirectional convolutional LSTM can model the temporal and spatial information of continuous frames of CEUS video, and segment the static thyroid ablation area lesions surrounded by contrast agent from a segment of CEUS video.

[0069] Example 3

[0070] This embodiment provides a thyroid ultrasound contrast imaging lesion segmentation system, including:

[0071] The data acquisition module collects the patient's real-time thyroid ultrasound contrast video; the ablation area of ​​the patient's thyroid after microwave ablation has no biological activity;

[0072] The multi-attention mechanism network model module described in Example 1 uses a data set to train the model until the network parameter performance is optimal;

[0073] The lesion segmentation module inputs the video collected by the data acquisition module into the trained multi-attention mechanism network model, and outputs a two-dimensional image showing the segmented lesions.

[0074] The above-mentioned multi-attention mechanism network (MAM-Net) model, thyroid ultrasound contrast imaging lesion segmentation method and system can be stored in a computer-readable storage medium if they are implemented in the form of software functional units and sold or used as independent products. Therefore, the essence of this technical solution or the part that contributes to the prior art or the part of this technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for an electronic device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0075] In summary, the present invention provides a multi-attention mechanism network model and a thyroid ultrasound angiography lesion segmentation method and system based on the multi-attention mechanism network. The multi-attention mechanism network model uses the RFM-ResNeXt module to extract the spatiotemporal feature information of the input video; the three-dimensional feature map is aggregated into a two-dimensional feature map through the bidirectional convolution LSTM module; the two-dimensional feature map is input into the dual attention module DAM to obtain the attention feature map; the attention feature map is convolved and upsampled to obtain a binary segmentation result image. The present invention uses three attention mechanisms, namely channel attention, time attention and spatial attention, to effectively extract the spatiotemporal information of ultrasound angiography videos, and automatically and accurately segment the ablation area in the thyroid ultrasound angiography image.

[0076] Those skilled in the art should understand that those skilled in the art can implement variations by combining the prior art and the above embodiments, which will not be described in detail here. Such variations do not affect the essential content of the present invention, and will not be described in detail here.

[0077] The above describes the preferred embodiments of the present invention. It should be understood that the present invention is not limited to the above-mentioned specific embodiments, and the devices and structures that are not described in detail should be understood to be implemented in a common manner in the art; any technician familiar with the art can use the above-disclosed methods and technical contents to make many possible changes and modifications to the technical solutions of the present invention without departing from the scope of the technical solutions of the present invention, or modify them into equivalent embodiments of equivalent changes, which does not affect the essential content of the present invention. Therefore, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solutions of the present invention are still within the scope of protection of the technical solutions of the present invention.

[0078] Those skilled in the art will appreciate that the units, i.e., algorithm steps, of the various examples described in conjunction with this embodiment can be implemented in electronic hardware or in a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

Claims

1. A multi-attention mechanism network model, characterized in that: It is a segmentation network structure: A 3D convolutional neural network is used as the encoder, and a 2D convolutional neural network is used as the decoder. The input data is a 3D video clip, and the output result is a 2D segmentation result image. The RFM-ResNeXt module is used to extract the spatiotemporal feature information of the input video; the three-dimensional feature map is aggregated into a two-dimensional feature map through the bidirectional convolution LSTM module; the two-dimensional feature map is input into the dual attention module DAM to obtain an attention feature map; the attention feature map is convolved and upsampled to obtain a binary segmentation result image; The RFM-ResNeXt module includes five submodules: conv1, res1, res2, res3, and res4. Conv1 consists of two consecutive three-dimensional convolutions, group normalization, and ReLU activation functions; res1, res2, res3, and res4 consist of 3, 4, 6, and 3 consecutive residual fusion modules RFM, respectively; The dual attention module DAM includes a spatial attention module PAM and a channel attention module CAM.

2. The multi-attention mechanism network model according to claim 1, characterized in that: The RFM module is represented as ,in Represents the direct mapping part; Represents the residual part, which consists of three layers of three-dimensional convolution with kernels of 1, 3, and 1 respectively; represents the fusion module; this module will and The result of the addition Y is input to the CAM module. The CAM module includes global and local channel attention. The global channel attention performs two convolution operations with a convolution kernel of 1 after global average pooling on the input Y. The local channel attention directly performs two convolution operations with a convolution kernel of 1 on the input Y. Then, the output results of the global channel attention and the local channel attention are added together, and then the added result is passed through the Sigmoid activation function. Finally, the output results of the activation function are respectively compared with and After multiplication, we get .

3. The multi-attention mechanism network model according to claim 2, characterized in that: The bidirectional convolutional LSTM module includes basic components consisting of five basic units: input gate, forget gate, output gate, cell state and hidden state, and then 16 basic components are connected end to end to form a loop structure.

4. The multi-attention mechanism network model according to claim 1 or 3, characterized in that: The PAM module first inputs After three two-dimensional convolutions with a convolution kernel of 1, we get three feature maps Q, K, and V, and transform Q into Then transpose to get Feature map of size (N is the product of H and W), K is transformed into Feature map of size , and then the obtained and After matrix multiplication, we get Feature map of size ; Transform V into Feature map After After multiplying the transposed matrix of The feature map of size is then transformed into Feature map of size ,Will and Add to get the feature map ; The CAM module will input The deformation size is Feature map and and size Feature map ,Will and After matrix multiplication, a layer of softmax function is used to obtain a size of Feature map ;Will After multiplying with the B matrix, we get The feature map is then transformed into Feature map of size ,Will and After adding, we get the feature map .

5. A thyroid ultrasound contrast imaging lesion segmentation method, characterized in that: Using the trained multi-attention mechanism network model according to any one of claims 1 to 4 to perform lesion segmentation in thyroid ultrasound contrast images comprises the following steps: S1. The ablation area of ​​the patient's thyroid after microwave ablation has no biological activity. The patient's thyroid ablation ultrasound contrast data is preprocessed to obtain a data set: first, the image is annotated, and then the spatial dimension is cropped and the time dimension is divided; S2, divide the data set obtained in S1 into a training set and a validation set in a ratio of 8:2; S3, inputting the training set and the validation set in S2 into the multi-attention mechanism network model as described in any one of claims 1 to 4 to perform model training, and obtaining network parameters that optimize the model performance; S4. Use the multi-attention mechanism network model with the best performance obtained in S3 to segment the real-time thyroid ultrasound angiography video to achieve the discrimination and analysis of thyroid nodules and ablation areas.

6. The method for segmenting thyroid ultrasound contrast-enhanced lesions according to claim 5, characterized in that: The step S1 comprises the following sub-steps: S1.

1. Clinical experts annotate the ablation area of ​​thyroid lesions to obtain the corresponding binary image annotation file; S1.2, using the threshold method to crop the spatial dimension of each frame of the ultrasound contrast imaging video and its annotated file; S1.3, dividing the ultrasound contrast imaging video processed in S1.2 and the corresponding annotation file into video segments with a fixed frame length of a certain number of frames; if the number of video frames in the last division of the video is insufficient, the division is removed to obtain a certain number of video segments and corresponding binary annotation images.

7. The thyroid ultrasound contrast imaging lesion segmentation method according to claim 5, characterized in that: In step S3, the dice loss error between the predicted image and the real annotation of the multi-attention mechanism network model is back-propagated through the back-propagation algorithm, and the optimal network model is obtained through continuous iteration; the calculation formula of the dice loss function is expressed as: , where X represents the predicted result image and Y represents the expert labeled image.

8. A thyroid ultrasound contrast imaging lesion segmentation system, characterized in that: include: Data acquisition module, collecting patients' real-time thyroid ultrasound contrast video; The ablation area after microwave ablation of the patient's thyroid gland had no biological activity; The multi-attention mechanism network model module according to any one of claims 1 to 4, wherein the model is trained using a data set to meet the optimal performance of network parameters; The lesion segmentation module inputs the video collected by the data acquisition module into the trained multi-attention mechanism network model, and outputs a two-dimensional image showing the segmented lesions.

Images