Ultrasound carotid intima-media segmentation method based on class activation mapping feature fusion

Abstract

The application discloses an ultrasound carotid artery intima-media segmentation method based on class activation mapping feature fusion, and the method comprises the following steps: acquiring a to-be-detected CAUS image and inputting the to-be-detected CAUS image into a pre-constructed segmentation model; performing feature extraction on the to-be-detected CAUS image based on a feature extraction network to obtain feature relationship image information; generating an activation map and activating a global object region according to the feature relationship image information based on a region activation iterative generation module; generating local features and global features according to the global object region based on a feature generation module; and fusing the global features and the local features based on a feature fusion module to obtain a final segmentation map. Through the use of the application, the carotid artery intima-media boundary can be automatically, quickly and accurately segmented. The application can be widely applied to the field of image segmentation as an ultrasound carotid artery intima-media segmentation method based on class activation mapping feature fusion.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image segmentation, and in particular to an ultrasound carotid intima-media segmentation method based on class activation mapping feature fusion. BACKGROUND

[0002] Accurate detection and labeling of plaque formation in clinic can control and monitor the progression of atherosclerosis. Carotid intima-media thickness (CIMT) estimates the width of the two deepest layers of the arterial wall, which is the most common sign of atherosclerosis development and an important reference index for guiding clinical diagnosis. In clinic, carotid artery ultrasound (CAUS) technology can display the carotid artery wall accordingly as a non-invasive, safe and cost-effective method, so CAUS images are widely used in clinic to detect carotid intima-media.

[0003] In most cases, CA demarcation is performed manually by medical experts, but manual segmentation has poor repeatability, is prone to errors, and has significant variability between different experts, devices and datasets. Therefore, it is necessary to use a fully automatic method for robust and rigorous carotid intima-media measurement. However, there is currently no algorithm for the technical problems caused by the large local differences of carotid physiological specificity and the noise and artifacts of imaging devices. SUMMARY

[0004] The purpose of the present application is to provide an ultrasound carotid intima-media segmentation method based on class activation mapping feature fusion, which can automatically and quickly and accurately segment the carotid intima-media boundary, and solve the problems caused by the large local differences of carotid and the imaging devices.

[0005] The first technical solution adopted by the present application is: an ultrasound carotid intima-media segmentation method based on class activation mapping feature fusion, comprising the following steps:

[0006] Obtaining a to-be-tested CAUS image and inputting the to-be-tested CAUS image into a pre-constructed segmentation model;

[0007] The pre-constructed segmentation model comprises a feature extraction network, a region activation iterative generation module, a feature generation module and a feature fusion module;

[0008] Based on the feature extraction network, the feature relationship image information of the to-be-tested CAUS image is extracted;

[0009] Based on the region activation iterative generation module, the activation map is generated according to the feature relationship image information, and the global object region is activated;

[0010] Based on the feature generation module, the local feature and the global feature are generated according to the global object region;

[0011] The global feature and the local feature are fused based on the feature fusion module to obtain a final segmentation map.

[0012] Further, before the step of acquiring the to-be-tested CAUS image and inputting the to-be-tested CAUS image into the pre-constructed segmentation model, the method further comprises the following steps of:

[0013] The pre-constructed segmentation model is trained, tested and verified by constructing a data set.

[0014] Further, the feature extraction of the to-be-tested CAUS image based on the feature extraction network is iterative extraction, and the formula is as follows:

[0015]

[0016] In the above formula, denotes the input of the i-th residual module, denotes the ResNet module, denotes the input and the internal weight are operated, denotes the identity branch, denotes the input to obtain the identity mapping.

[0017] Further, the step of generating an activation map and activating a global object region according to feature relationship image information based on the region activation iterative generation module specifically comprises the following steps of:

[0018] Based on the region activation iterative generation module, the feature relationship image information is activated in the last convolution layer according to the class activation mapping principle to obtain activation information;

[0019] The activation information is multiplied by the weight from the adaptive average pooling to obtain a region-level convolution activation map;

[0020] According to the iterative inference of the pixel-level label, new object regions are gradually activated to obtain a global object region.

[0021] Further, the calculation formula of the region-level convolution activation map is as follows:

[0022]

[0023]

[0024] In the above formula, denotes the activation map, is equivalent to performing linear operation and full connection layer on the image feature ​denotes the image is divided adaptive average pooling of the part, is an activation function, the activation map is input to the adaptive average pooling and the classification confidence is finally generated using the activation function .

[0025] Further, the feature-based generation module generates local features and global features according to the global object region, which specifically includes:

[0026] The feature-based generation module calculates the local feature center according to the global object region and constructs the local feature relationship through the local feature center;

[0027] The local feature relationship is integrated to obtain the global feature;

[0028] The global feature is processed by the classification confidence to suppress irrelevant features.

[0029] Further, the calculation formula of the local feature and the global feature is as follows:

[0030]

[0031]

[0032] wherein denotes the segmented feature relationship image, denotes the segmented activation map, represents the classification confidence, , denotes the input segmented activation map, the sofmax normalization operation is performed along the spatial dimension and multiplied by the feature relationship image matrix, and then suppressed by the classification confidence to obtain the local feature .

[0033] wherein denotes a learnable weight parameter, the convolution layer fuses node information, the linear layer fuses channel information, represents the convolution and linear operation on the local feature and the multiplication and summation with the learnable weight to obtain the global feature .

[0034] Further, the formula according to the feature pixel relationship is as follows:

[0035]

[0036] wherein , , represents a linear conversion matrix to perform a linear transformation, represents a linear transformation along a spatial dimension performing a normalization operation of sofmax, representing that the local features and image features obtained above are subjected to linear transformation, normalization operation and multiplication with the global features subjected to linear transformation, and finally obtaining an enhanced feature pixel relationship .

[0037] The method has the advantages that: the application establishes the corresponding relationship between the CAUS image and the intima-media boundary segmentation result of the carotid artery by using the self-learning ability of the deep learning network, determines the intima-media boundary segmentation result of the carotid artery corresponding to the current image feature, obtains the coarse segmentation through the class activation mapping, and obtains the corresponding local feature and global feature in combination with the latent feature relationship image information, thereby effectively avoiding the problem of imperfect segmentation caused by large local differences of the carotid artery due to factors such as movement, position and equipment, and improving the segmentation precision through the attention global-local fusion feature. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 is a step flow chart of an ultrasound carotid intima-media segmentation method based on class activation mapping feature fusion according to the application;

[0039] Figure 2 is a structure block diagram of a deep learning network according to a specific embodiment of the application;

[0040] Figure 3 is a segmentation result schematic diagram according to a specific embodiment of the application;

[0041] Figure 4 is a structure schematic diagram of a region activation iteration generation module according to a specific embodiment of the application;

[0042] Figure 5 is a structure schematic diagram of a local and global feature generation module according to a specific embodiment of the application.

[0043] Figure 6 is a structure schematic diagram of a feature fusion module according to a specific embodiment of the application. DETAILED DESCRIPTION

[0044] The application will be further described in detail below in combination with the drawings and specific embodiments. For the step numbers in the following embodiments, only the setting is for the convenience of description, and the order between the steps is not limited in any way, and the execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0045] As Figure 1As shown, the application provides an ultrasound carotid intima-media segmentation method based on class activation mapping feature fusion, which comprises the following steps:

[0046] S0, constructing a data set to train, test and verify the pre-constructed segmentation model.

[0047] S0.1, collect CAUS images of different health conditions and corresponding carotid intima-media segmentation results as a data set;

[0048] S0.2, divide the data set into corresponding training, testing and verification data sets, and use the training data set for training;

[0049] S0.3, input the CAUS images of different health conditions, and output the corresponding carotid intima-media segmentation results, train the deep learning network, and test it with the test data set;

[0050] S0.4, adjust the network parameters of the deep learning network until the error rate reaches the preset range, and obtain the trained deep learning network, i.e. the pre-constructed segmentation model in this scheme.

[0051] The network parameters of the deep learning network include the number of convolution layers, the number of normalization times, the number of BN layers, the type of activation function, the size and number of convolution kernels, the number of pooling layers, the number of up-sampling layers, the initial weight, the bias value and all the hyperparameters in the network.

[0052] S1, obtaining the CAUS image to be tested and inputting the CAUS image to be tested into the pre-constructed segmentation model;

[0053] The pre-constructed segmentation model comprises a feature extraction network, a region activation iterative generation module, a feature generation module and a feature fusion module;

[0054] S2, feature extraction of the CAUS image to be tested based on the feature extraction network, to obtain feature relationship image information;

[0055] Specifically, based on the classic network architecture ResNet, the potential features of the CAUS image are taken, and the dilated and deepened ResNet-50 (which can be adjusted according to the task) is used to extract the features of the CAUS image to be tested. Where F is the ResNet module, h is the identity branch, and the final ;

[0056] The feature relationship image information of the intima-media and the media-adventitia in the CAUS image is obtained through the skeleton network to obtain the potential intima-media position, size, difference and other multi-level feature relationships.

[0057] S3, a region-based activation iterative generation module, generates an activation map according to feature relationship image information and activates a global object region;

[0058] S3.1, the region-based activation iterative generation module, activates the feature relationship image information according to the class activation mapping principle in the last convolution layer to obtain activation information;

[0059] S3.2, multiply the activation information by the weight from the adaptive average pooling to obtain a region-level convolution activation map;

[0060] S3.3, iteratively infer according to the pixel-level label, gradually activate new object regions, and obtain a global object region, for reference Figure 4 .

[0061] The calculation formula of the region-level convolution activation map is as follows:

[0062]

[0063]

[0064] In the above formula, represents the activation map, is equivalent to a linear operation and a fully connected layer, represents dividing the image into adaptive average pooling of the part, is an activation function, represents the classification confidence.

[0065] S4, a feature-based generation module, generates local features and global features according to the global object region;

[0066] S4.1, the global object region is used to calculate the local feature center in a weighted summation manner, so as to ensure that the local feature center has relevant channel information and node information, and a graph convolution unit is used to construct the interaction of the local feature center to obtain the local feature relationship;

[0067] S4.2, according to the local feature relationship, integrating all the local feature relationships of the regions to obtain a global overall feature, and using classification confidence for weighted fusion to suppress irrelevant features such as background and noise, and ensure that only the required features are collected.

[0068] Specifically, the obtained global feature relationship can improve the inconsistent relationship caused by the large local specificity difference of the carotid artery due to the calculation of the local feature relationship in the region, for reference Figure 5 .

[0069] The calculation formula of the local feature and the global feature is as follows:

[0070]

[0071]

[0072] wherein denotes the segmented feature relationship image, denotes the segmented activation map, , denotes the normalization operation of sofmax along the spatial dimension , denotes the local feature, denotes the learnable weight parameter, the convolutional layer fuses the node information, the linear layer fuses the channel information, denotes the global feature.

[0073] S5, based on the feature fusion module, fusing the global feature and the local feature to obtain the final segmentation map.

[0074] S5.1, calculating the carotid intima-media pixel similarity relationship in each region using the local feature and the intima-media feature relationship image information;

[0075] S5.2, using the global feature and the intima-media pixel similarity relationship to calculate the enhanced feature pixel relationship, and simultaneously performing dimension conversion to restore the original size;

[0076] Specifically, the enhanced feature pixel relationship is obtained by using the non-local method, and the formula is as follows:

[0077]

[0078] Specifically, , , denotes the linear conversion matrix to perform linear transformation, denotes the normalization operation of sofmax along the spatial dimension , denotes the enhanced feature pixel relationship.

[0079] S5.3, splicing and convolving the enhanced feature pixel relationship with channel features and context features with the original feature pixel relationship to generate the final segmentation map, and the structure diagram is referred to Figure 6 .

[0080] An ultrasonic carotid intima-media segmentation device based on class activation mapping feature fusion:

[0081] at least one processor;

[0082] at least one memory for storing at least one program;

[0083] When the at least one program is executed by the at least one processor, the at least one processor implements the method for carotid intima-media segmentation based on class activation mapping feature fusion as described above.

[0084] The content in the above method embodiments is applicable to the device embodiments, the device embodiments specifically implement the same functions as the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0085] A storage medium, wherein the storage medium stores processor-executable instructions, and the processor-executable instructions, when executed by a processor, are used to implement the method for carotid intima-media segmentation based on class activation mapping feature fusion as described above.

[0086] The content in the above method embodiments is applicable to the storage medium embodiments, the storage medium embodiments specifically implement the same functions as the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0087] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the above-mentioned embodiments, and those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the present application, and these equivalent modifications or replacements are all included in the scope defined by the claims of the present application.

Claims

1. A method for segmenting the intima-media layer of the carotid artery based on class activation mapping feature fusion, characterized in that, Includes the following steps: Acquire the CAUS image to be tested and input the CAUS image to be tested into the pre-built segmentation model; The pre-built segmentation model includes a feature extraction network, a region activation iterative generation module, a feature generation module, and a feature fusion module; Feature extraction is performed on the CAUS image under test using a feature extraction network to obtain feature relationship image information; Based on the region activation iterative generation module, an activation map is generated and global object regions are activated according to the feature relationship image information; Based on the feature generation module, local and global features are generated according to the global object region; Based on the feature fusion module, global features and local features are fused to obtain the final segmentation map; The feature extraction of the CAUS image under test based on the feature extraction network is specifically iterative, and the formula is expressed as follows: In the above formula, This represents the input of the i-th residual module. Indicates the ResNet module. Indicates input With internal weights Perform calculations. Indicates the identity branch, Indicates input We obtain the identity mapping; The step of generating an activation map and activating the global object region based on the feature relationship image information by the region activation iterative generation module specifically includes: Based on the region activation iterative generation module, the feature relationship image information is activated in the last convolutional layer according to the class activation mapping principle to obtain activation information; The activation information is multiplied by the weights from adaptive average pooling to obtain a region-level convolutional activation map. Iterative reasoning is performed based on pixel-level labels to gradually activate new object regions and obtain the global object region. The formula for calculating the region-level convolutional activation map is as follows: In the above formula, Indicates that the map is activated. Equivalent to image features Perform linear operations and fully connected layers. This indicates dividing the image. Partial adaptive average pooling, It is an activation function that takes the activation map as input, feeds it into adaptive average pooling, and uses the activation function to finally generate classification confidence. ; The calculation formulas for the local features and global features are as follows: In the above formula, This represents the feature relationship image after segmentation. This represents the active map after segmentation. Represents classification confidence. , This represents the activated map after input segmentation, along the spatial dimension. After performing sofmax normalization and multiplying it with the feature relation image matrix, local features are obtained by further suppression with the classification confidence. , Indicates the learnable weight parameters. Convolutional layers fuse node information. Linear layers fuse channel information. This means that the global features are obtained by performing convolution and linear operations on local features and then multiplying them with learnable weights and summing the results. .

2. The ultrasound carotid intima-media segmentation method based on class activation mapping feature fusion according to claim 1, characterized in that, Before the step of acquiring the CAUS image to be tested and inputting the CAUS image to be tested into the pre-built segmentation model, the following steps are also included: Build a dataset to train, test, and validate the pre-built segmentation model.

3. The ultrasound carotid intima-media segmentation method based on class activation mapping feature fusion according to claim 1, characterized in that, The step of generating local and global features based on the global object region by the feature generation module specifically includes: Based on the feature generation module, local feature centers are calculated according to the global object region, and local feature relationships are constructed using the local feature centers. By integrating the relationships between local features, global features are obtained. Irrelevant features are suppressed by using classification confidence scores to suppress global features.

4. The ultrasound carotid intima-media segmentation method based on class activation mapping feature fusion according to claim 1, characterized in that, The step of fusing global and local features based on the feature fusion module to obtain the final segmentation map specifically includes: Based on the feature fusion module, the similarity relationship of carotid intima-media pixels in each region is calculated according to local features and feature relationship image information. The enhanced feature pixel relationship is calculated based on the global features and the similarity relationship between the inner and middle membrane pixels. At the same time, the enhanced feature pixel relationship is dimensionally transformed to restore the original size. The enhanced feature pixel relationships are concatenated with the original feature pixel relationships and then convolved to generate the final segmentation image.

5. The ultrasound carotid intima-media segmentation method based on class activation mapping feature fusion according to claim 4, characterized in that, The formula for calculating the enhanced feature pixel relationship based on global features and the similarity relationship between inner and middle membrane pixels is expressed as follows: In the above formula, , , Represent a linear transformation matrix to perform a linear transformation. Indicates along spatial dimensions Perform normalization on sofmax. This involves performing linear transformations on local features and image features separately, then normalizing them, and finally multiplying them with the global features after the linear transformation to obtain the final representation of the enhanced feature pixel relationships. .

Images