Carotid plaque segmentation method and device in ultrasonic image, equipment and medium

Abstract

The invention relates to the technical field of image processing, and particularly discloses a carotid plaque segmentation method and device in an ultrasonic image, equipment and a medium, which construct a carotid plaque segmentation network and train and test the carotid plaque segmentation network to realize end-to-end segmentation of a carotid ultrasonic image to be segmented. The carotid plaque segmentation network comprises a feature extraction module, N multi-scale attention modules, a fusion module and N-1 boundary supplement modules; the feature extraction module is used for extracting rich multi-layer features f1 to fN; the multi-scale attention module is used for capturing and integrating multiple layers of features f1 to fN to obtain multiple layers of perception features F1 to FN; the fusion module is used for generating a prediction map SN of the Nth layer according to F2 to FN; and the boundary supplement module generates a prediction map Sa of the current layer according to the sensing feature Fa of the current layer and the sensing feature Fa + 1 and the prediction map Sa + 1 of the adjacent high layer, and the prediction map S1 of the first layer is selected as a final segmentation result. Compared with various mainstream medical image segmentation methods, the method has the advantage that the most excellent segmentation result is obtained.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly to a method, device, equipment and medium for segmenting carotid artery plaques in ultrasonic images. Background Art

[0002] Carotid artery ultrasound is a commonly used non-invasive diagnostic tool that can detect various blood flow parameters, such as blood flow velocity, blood flow width, as well as the thickness and irregularity of the arterial wall. These parameters are of great significance for evaluating arterial health status and potential disease risks. Among them, the intima-media thickness (IMT) of the carotid artery is one of the important indicators reflecting atherosclerosis and cardiovascular disease risks. When the IMT exceeds 1.3 millimeters, carotid artery plaques are usually considered to have formed. The presence of these plaques is closely related to serious diseases such as cerebral infarction and stroke. Therefore, segmenting carotid artery plaques in ultrasonic images can help doctors accurately identify and measure the size, shape and distribution of carotid artery plaques, which plays a crucial role in the early prevention and precise intervention of stroke. However, it is subjective and laborious for ultrasound physicians to manually annotate each pixel on ultrasonic images. Therefore, the task of automatically segmenting carotid artery plaques based on ultrasonic images is of great significance.

[0003] In recent years, with the rapid development of deep learning, deep neural networks have demonstrated powerful capabilities in the field of medical image segmentation and achieved breakthrough progress in many tasks. However, due to the fact that carotid artery plaques usually exhibit high heterogeneity, with complex and variable shapes, and at the same time, the boundaries are blurred and irregular, the performance of some existing general segmentation algorithms in this task is still not ideal, especially when dealing with the complex morphology and blurred boundaries of plaques. Therefore, the segmentation of carotid artery plaques mainly faces the following challenges: (1) In ultrasonic images, due to imaging noise, artifacts and resolution limitations, the boundaries of plaques are often relatively blurred, and the contrast with surrounding tissues is low. In addition, the irregularity of plaque morphology further increases the difficulty of automatic segmentation. (2) Carotid artery plaques may have significant shape differences in different patients or at different stages, and at the same time, they show high heterogeneity in tissue structure and texture distribution. This diversity makes it difficult for the model to capture the general features of plaques, thus affecting the stability and accuracy of segmentation. Summary of the Invention

[0004] The present invention provides a method, device, equipment and medium for segmenting carotid artery plaques in ultrasonic images, and the technical problem to be solved is: how to accurately segment carotid artery plaques in ultrasonic images.

[0005] To solve the above technical problems, the present invention provides a method for segmenting carotid artery plaques in ultrasonic images, including:

[0006] Construct a carotid plaque segmentation network, where the carotid plaque segmentation network includes a feature extraction module, N multi-scale attention modules, a fusion module, and N - 1 boundary supplement modules; the feature extraction module is used to extract features at N scales from the input image to obtain corresponding multi-layer features f with gradually decreasing scales 1 to f N , N ≥ 3; the N multi-scale attention modules are used to extract attention from the multi-layer features f 1 to f N respectively to obtain corresponding multi-layer perception features F 1 to F N ; the fusion module is used to gradually perform upsampling and fusion on the multi-layer perception features F 2 to F N to generate the prediction map S of the Nth layer N ; the boundary supplement module of the current layer is used to generate the prediction map S of the current layer according to the perception feature F of the current layer a and the perception feature F of the adjacent higher layer a+1 and the prediction map S a+1 , a ∈ {1, 2,..., N - 1}, where S a is selected as the final segmentation result; 1

[0007] Use the training set to train the carotid plaque segmentation network and use the test set to test the trained carotid plaque segmentation network;

[0008] Input the carotid ultrasound image to be segmented into the tested carotid plaque segmentation network, and the carotid plaque segmentation network outputs the corresponding carotid plaque segmentation image.

[0009]

[0010] Specifically, the operations performed by the multi-scale attention module are as follows: i Respectively pass the feature f of the ith layer through three e×e convolutions to generate a query matrix Q, a key matrix K, and a value matrix V, i ∈ {1, 2,..., N}, and e is the convolution kernel size;

[0011] Downsample the query matrix Q, the key matrix K, and the value matrix V respectively through three downsampling modules. Each downsampling module performs downsampling according to predefined M downsampling factors and projects them into different scale spaces through M e×e convolutions to obtain the downsampled query matrix Q j , the downsampled key matrix K j and the downsampled value matrix V j , j ∈ {1, 2,..., M};

[0012] According to the downsampled query matrix Q j ​​and the downsampling key matrix K j Generate weight graph

[0013] For the downsampled value matrix V j Fusion to get the value matrix And the value matrix Add to the value matrix V after 1×1 convolution to generate the value matrix V o ;

[0014] The value matrix Add to the value matrix V after 1×1 convolution to generate the value matrix V o ;

[0015] The weight map and value matrix V o Perform element-by-element multiplication to generate the feature map V′ and f i The 1×1 convolution results are added together to finally obtain the multi-layer perception feature F i .

[0016] Further, according to the down-sampling query matrix Q j and the downsampling key matrix K j Generate weight graph The specific process is:

[0017] By downsampling the query matrix Q in the same scale space j and the downsampling key matrix K j Perform e×e convolution and multiply to get the attention map, and then adjust the attention maps of all scale spaces to a uniform size through bilinear interpolation and take the average to get the weight map

[0018] Furthermore, for the downsampled value matrix V j Fusion to get the value matrix The specific process is:

[0019] For the downsampled value matrix V j M scales V 1 To V M Upsample according to the predefined M upsampling factors, and project them to different scale spaces through M e×e convolutions, add them together, and then perform e×e convolution to obtain the value matrix

[0020] Furthermore, the operations performed by the boundary supplementation module are:

[0021] The perceptual feature F of the current layer a And the perceptual features F of the adjacent high-level layers of the upsampled a+1 After e×e convolution processing, they are added to obtain the perceptual features. Upsample the prediction map S of the adjacent high layer a+1 and multiply it with the perceptual feature to obtain the feature map S'. a Perform local average pooling on the feature map S' a consisting of an e×e convolution kernel, and subtract the processed feature from S' a to obtain the boundary clue B 1 ; Concatenate the feature map S' a with the boundary clue B 1 to obtain the feature map

[0022] Upsample the prediction map S of the adjacent high layer a+1 and perform dilation and erosion operations respectively and then subtract to obtain the boundary clue B 2 ; Multiply the boundary clue B 2 with the perceptual feature and then add it to the perceptual feature and generate a feature map through e×e convolution

[0023] Perform e×e convolution on the feature map respectively and then concatenate them, and extract attention through the spatial attention unit and the channel attention unit for the concatenated result, and then perform e×e convolution on the attention and then 1×1 convolution to obtain the prediction map S of the current layer a .

[0024] Furthermore, the feature extraction module adopts a pre-trained PVTv2-B2 network.

[0025] Furthermore, the loss function used for training the carotid plaque segmentation network is S i represents the i-th prediction map, G m represents the ground truth annotation mask, and L() represents the sum of the weighted binary cross-entropy loss L wB and the weighted intersection over union loss L wI .

[0026] The present invention also provides a device for segmenting carotid artery plaques in ultrasonic images, which is applied to the method for segmenting carotid artery plaques in ultrasonic images. The key lies in that the device includes a network construction part, a network training and testing part, and a network application part. The network construction part is used to construct a carotid artery plaque segmentation network; the network training and testing part is used to train the carotid artery plaque segmentation network with a training set and test the trained carotid artery plaque segmentation network with a test set; the network application part is used to input a carotid artery ultrasonic image to be segmented into the carotid artery plaque segmentation network that has completed testing, and the carotid artery plaque segmentation network outputs a corresponding carotid artery plaque segmentation image.

[0027] The present invention also provides a device for segmenting carotid artery plaques in ultrasonic images, which is characterized by including:

[0028] One or more processors;

[0029] A memory for storing one or more programs;

[0030] When the one or more programs are executed by the one or more processors, the one or more processors implement the method for segmenting carotid artery plaques in ultrasonic images.

[0031] The present invention also provides a storage medium, which is characterized in that: a computer program is stored thereon, and when the computer program is executed by a processor, the method for segmenting carotid artery plaques in ultrasonic images is implemented.

[0032] The method, device, equipment and medium for segmenting carotid artery plaques in ultrasonic images provided by the present invention construct a carotid artery plaque segmentation network and train and test it to achieve end-to-end segmentation of carotid artery ultrasonic images to be segmented. The carotid artery plaque segmentation network includes a feature extraction module, N multi-scale attention modules, a fusion module and N - 1 boundary supplement modules; the feature extraction module is used to extract rich multi-layer features f 1 to f N ; the multi-scale attention modules are used to capture and integrate the multi-layer features f 1 to f N , and obtain corresponding multi-layer perception features F 1 to F N , so as to accurately identify carotid artery plaques with different sizes and shapes; the fusion module is used to generate the Nth layer of prediction map S 2 to F N ; the boundary supplement module combines average pooling and dilation-erosion to enhance the boundary representation, and according to the perception feature F N of the current layer, as well as the perception feature F a of the adjacent high layer and the prediction map S a+1 and S a+1Generate the prediction map S of the current layer a , and the prediction map S of the first layer 1 is selected as the final segmentation result. In addition, the present invention considers calculating the combined loss for the prediction maps at each scale, which helps to improve the segmentation accuracy. Compared with a variety of mainstream medical image segmentation methods, the present invention has achieved the best segmentation result and realized the accurate segmentation of carotid plaques in ultrasonic images. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 is the structural diagram of the carotid plaque segmentation network provided by the embodiment of the present invention;

[0034] Figure 2 is the structural diagram of the multi-scale attention module provided by the embodiment of the present invention;

[0035] Figure 3 is the structural diagram of the fusion module provided by the embodiment of the present invention;

[0036] Figure 4 is the structural diagram of the boundary supplement module provided by the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0037] The following specifically illustrates the implementation manner of the present invention in conjunction with the drawings. The given embodiments are only for illustrative purposes and should not be construed as limiting the present invention. The drawings are only for reference and explanation, and do not constitute a limitation on the protection scope of the present invention patent. Because many changes can be made to the present invention without departing from the spirit and scope of the present invention.

[0038] Aiming at the problem of carotid plaque segmentation in ultrasonic images, the present invention first proposes a high-performance multi-scale fusion and boundary enhancement method for carotid plaque segmentation in ultrasonic images, aiming to accurately segment the carotid plaques in carotid ultrasonic images to assist clinicians in more quickly and accurately screening and diagnosing cerebral infarction and stroke.

[0039] The method for segmenting carotid plaques in ultrasonic images provided by the embodiment of the present invention includes the steps of:

[0040] Construct a carotid plaque segmentation network;

[0041] Train the carotid plaque segmentation network using a training set and test the trained carotid plaque segmentation network using a test set;

[0042] Input the carotid ultrasonic image to be segmented into the tested carotid plaque segmentation network, and the carotid plaque segmentation network outputs the corresponding carotid plaque segmentation image.

[0043] The BSNet proposed in the present invention follows an end-to-end framework, that is, only data needs to be input to obtain the predicted output, and no other operations are required in the middle.

[0044] The architecture of the carotid plaque segmentation network constructed by the present invention (referred to as BSNet in this example) is as follows: Figure 1 As shown, the carotid plaque segmentation network includes a feature extraction module, N multi-scale attention modules (MASMs), a fusion module (PD), and N-1 boundary supplementation modules (BSMs);

[0045] The feature extraction module is used to extract N scale features of the input image and obtain the corresponding multi-layer features f 1 to f N , N ≥ 3;

[0046] N multi-scale attention modules are used to focus on the multi-layer features f 1 to f N Perform attention extraction respectively to obtain the corresponding N-scale multi-layer perception features F 1 To F N ;

[0047] The fusion module is used to integrate the multi-layer perception features F 2 To F N Upsampling and fusion are performed step by step to generate the prediction graph S of the Nth layer N ;

[0048] The boundary supplement module of the current layer is used to calculate the perceptual features F of the current layer. a and the perceptual features F of the adjacent high-level a+1 and prediction graph S a+1 Generate the prediction map S of the current layer a , a∈{1,2,...,N-1}, where S 1 is selected as the final segmentation result.

[0049] This embodiment adopts the PVTv2-B2 network (pre-trained on ImageNet-1 K) as the feature extraction module because it can extract multi-level features and performs well in various visual tasks. Figure 1 Taking N=4 as an example, the multi-layer feature f generated by the feature extraction module 1 to f 4 Expressed as Where i represents the i-th layer of the feature extraction module, there are 4 layers in total, H and W are the length and width of the input image respectively, and C i (i∈{1,2,3,4}) represents the channel dimension of the feature of the i-th layer of the feature extraction module, The vector dimension representing the multi-scale feature is C i ×H / 2i ×W / 2 i Of course, the feature extraction module can also adopt other multi-level feature extraction networks that perform well in various visual tasks.

[0050] In ultrasound images, carotid plaques usually exhibit significant variations in size, shape, and texture, which pose a key challenge to accurate segmentation. In the present invention, this embodiment proposes a multi-scale attention module (MSAM) to capture and integrate multi-scale features, thereby accurately identifying carotid plaques with different sizes and shapes.

[0051] To perform attention extraction on the multi-level features f 1 to f N respectively, the structure of each multi-scale attention module is as Figure 1 shown. The N multi-scale attention modules perform one-to-one extraction on the multi-level features f 1 to f N to obtain the multi-level perceptual features F 1 to F N . On the basis of Figure 1 taking N = 4 as an example, the structure of each multi-scale attention module is as Figure 2 shown.

[0052] As Figure 2 shown, the operations performed by the multi-scale attention module are:

[0053] 1. The features f of the i-th layer i are respectively passed through three e×e (where e is the convolution kernel size, 1 < e < 6. The value of e should not be too small as the convolution effect will be poor, and the value of e should not be too large as the computational complexity will be high. In this example, e is preferably 3) convolutions to generate the query matrix Q, the key matrix K, and the value matrix V.

[0054] 2. The query matrix Q, the key matrix K, and the value matrix V are respectively downsampled through three downsampling modules (DS). Each downsampling module is downsampled according to a predefined M downsampling factors (in this example, 2 < M < 6. The value of M should not be too small as the downsampling effect will be poor, and the value of M should not be too large as the computational complexity will be high. In this example, M is preferably 4, and the corresponding downsampling factors are 8, 4, 2, 1 respectively) and projected into different scale spaces through M e×e convolutions to obtain the downsampled query matrix Q j , the downsampled key matrix K j , and the downsampled value matrix V j , where j ∈ {1, 2, …, M}.

[0055] 3. Generate the weight map j according to the downsampled query matrix Q j and the downsampled key matrix K In each scale space, by downsampling the query matrix Q in the same scale space j and the downsampled key matrix K j perform an e×e convolution and multiply to obtain the attention map, and then adjust the attention maps of all scale spaces to the same size by bilinear interpolation (upsampling) and take the average (avg) to obtain the weight matrix

[0056]

[0057] where σ(·) represents the normalization operation, d is the dimension of the query matrix Q and the key matrix K, ↑(·) represents the upsampling operation, Conv e (·) represents the e×e convolution, and T represents the matrix transpose.

[0058] 4. Fuse the downsampled value matrix V j to obtain the value matrix For the M scales V j to V 1 of the downsampled value matrix V M perform upsampling respectively according to the predefined M upsampling factors (in this example, preferably M = 4, and the corresponding upsampling factors are 1, 2, 4, 8 respectively), project them into different scale spaces through M e×e convolutions respectively, then add them up and perform an e×e convolution to obtain the value matrix

[0059]

[0060] 5. Add the value matrix to the value matrix V processed by a 1×1 convolution to generate the value matrix V o :

[0061]

[0062] where the adjustment factor α is used to automatically adjust the contribution of the value matrix during training, Conv 1 (·) represents the 1×1 convolution, represents element-wise addition.

[0063] 6. Multiply the weight map element-wise with the value matrix V o and add the result to the 1×1 convolution result of f i to finally obtain the feature map F i :

[0064]

[0065] The fusion module (PD module) processes the multi-layer perception feature F2 to F N Perform upsampling and fusion step by step to generate the prediction map S of the Nth layer N . The PD block performs an element-wise multiplication operation on the multi-layer perceptron features F 2 to F N and their respective high-level features (if any). F N is concatenated with the multiplied features along the channel dimension and a prediction map S is generated through an e×e convolution block N . The reason for excluding F 1 in the PD block is that F 1 contributes less to performance improvement but consumes more computational resources. Taking N = 4 as an example, the structure of the PD module is as Figure 3 shown

[0066] Since a single method may not comprehensively highlight boundary features, this embodiment combines average pooling and dilation-erosion to enhance boundary representation. This comprehensive method can effectively strengthen the extraction and representation of boundary features, thereby improving the overall performance. As Figure 4 shown, the boundary supplement module (BSM) takes the perceptual feature F of the current layer a and the perceptual feature F of the adjacent high layer a+1 and the prediction map S a+1 as inputs to generate the prediction map S of the current layer a . Specifically, this process includes the steps:

[0067] 1. Add the perceptual feature F of the current layer a and the upsampled perceptual feature F of the adjacent high layer a+1 (with the same scale as the perceptual feature F of the current layer a ) after being processed by e×e convolution respectively to obtain the perceptual feature

[0068]

[0069] 2. Multiply the upsampled prediction map S of the adjacent high layer a+1 (with the same scale as the perceptual feature ) by the perceptual feature to obtain the feature map S′ a :

[0070]

[0071] 3. Process the feature map S′ a by local average pooling (Avg(·)) composed of an e×e convolution kernel, and subtract the processed feature from the feature map S′ a to obtain the boundary clue B 1 :

[0072]

[0073] 4. To highlight the boundary region, the feature map S′ a is concatenated with the boundary clue B 1 to obtain the feature map

[0074] 5. The prediction maps S a+1 of adjacent high-level layers are respectively subtracted after dilation and erosion operations to obtain the boundary clue B 2 ; the boundary clue B 2 is multiplied by the perceptual feature and then added to the perceptual feature and an e×e convolution is performed to generate the feature map

[0075]

[0076] 6. The feature maps are respectively subjected to e×e convolutions and then concatenated, and the concatenated result is passed through a spatial attention (SA) unit and a channel attention (CA) unit to extract key features. After being processed by an e×e convolution, a 1×1 convolution is performed to generate the prediction map S a of the current layer. The spatial attention (SA) unit and the channel attention (CA) unit are used to extract key features to remove irrelevant information, where the spatial attention (SA) unit and the channel attention (CA) unit come from the widely used CBAM module.

[0077] During the process of training the carotid plaque segmentation network, a combined loss function L is adopted, which combines the weighted binary cross-entropy loss L wB and the weighted intersection over union loss L wI , that is, L = L wB + L wI . To obtain better segmentation results, a deep supervision mechanism is also applied, and L is applied to each prediction map. The total loss function is expressed as follows:

[0078]

[0079] where S i represents Figure 1 the i-th prediction map in m , and G i represents the ground truth annotation mask. It should be noted that before calculating the loss, the resolution of S m is adjusted to be consistent with that of G.

[0080] In the present invention, in this embodiment, a dataset named CP-Seg is collected for the segmentation of carotid artery plaques in ultrasound images. The CP-Seg dataset contains 3,500 ultrasound images for model training and their corresponding per-pixel segmentation masks, and another 500 ultrasound images for model testing and their corresponding segmentation masks. The image resolution is 352×352, and the images are normalized to remove unnecessary background information to ensure privacy protection.

[0081] In this embodiment, the Adam optimizer is used to optimize BSNet, and the initial learning rate is 1e-4. The training process of BSNet lasts for 100 epochs, the batch size is 16, and the learning rate decays every 20 epochs with a decay factor of 0.1. To mitigate overfitting, various data augmentation strategies are adopted, including random horizontal flipping and rotation. In addition, the size of the input images is adjusted to 352×352, and a multi-scale training method is used. This strategy adjusts the image size by randomly selecting scales from {0.75, 1, 1.25}.

[0082] In this embodiment, two evaluation metrics widely recognized in the field of medical image segmentation are used to measure the network performance. These metrics include the Dice similarity coefficient (Dice) and the intersection over union (IoU). Generally, the larger the Dice and IoU, the better the performance.

[0083] The network proposed in the present invention is compared with eight mainstream medical image segmentation methods, including U-Net, U-Net++, CE-Net, U-Net3+, CaraNet, BATFormer, I2U-Net, and SHFormer. The test results on the CP-Seg dataset are shown in Table 1. For easy observation, the best results are shown in bold. As can be seen from Table 1, the method proposed in this embodiment achieves the best results and is superior to the eight mainstream medical image segmentation methods in both evaluation metrics.

[0084] Table 1 Experimental Results

[0085] Method Dice IoU U-Net 0.695 0.648 U-Net++ 0.791 0.675 CE-Net 0.795 0.675 U-Net3+ 0.749 0.623 CaraNet 0.787 0.680 BATFormer 0.783 0.671 I2U-Net 0.730 0.605 SHFormer 0.780 0.661 BSNet 0.814 0.698

[0086] An embodiment of the present invention also provides a carotid plaque segmentation device in an ultrasonic image, which is applied to the carotid plaque segmentation method in the above ultrasonic image. The device includes a network construction part, a network training and testing part, and a network application part. The network construction part is used to construct a carotid plaque segmentation network; the network training and testing part is used to train the carotid plaque segmentation network using a training set and test the trained carotid plaque segmentation network using a test set; the network application part is used to input a carotid ultrasound image to be segmented into the carotid plaque segmentation network that has completed testing, and the carotid plaque segmentation network outputs a corresponding carotid plaque segmentation image.

[0087] An embodiment of the present invention also provides a carotid plaque segmentation device in an ultrasonic image, which includes:

[0088] One or more processors;

[0089] A memory for storing one or more programs;

[0090] When one or more programs are executed by one or more processors, the carotid plaque segmentation method in the ultrasonic image is implemented. The processor can be various general and / or special processing components with processing and computing capabilities. Some examples of the processor include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc.

[0091] An embodiment of the present invention also provides a storage medium, on which a computer program is stored. When the computer program is executed by a processor, the carotid plaque segmentation method in the above ultrasonic image is implemented. In some embodiments, the carotid plaque segmentation method in the ultrasonic image can be implemented as a computer program, which is tangibly included in a computer-readable storage medium, such as a storage unit.

[0092] In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device via a ROM and / or a communication unit. When the computer program is loaded into the RAM and executed by the processor, one or more steps of the carotid plaque segmentation method described above can be executed. Alternatively, in other embodiments, the processor can be configured to execute the carotid plaque segmentation method in any other suitable manner (for example, by means of firmware).

[0093] The various embodiments of the systems and techniques described above in this example can be implemented in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special-purpose or general-purpose programmable processor that receives data and instructions from a storage system, at least one input device, and at least one output device, and transmits the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0094] The computer programs for implementing the methods of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer programs are executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer programs can be executed entirely on the machine, partially on the machine, as a stand-alone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0095] In the context of the present invention, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer-readable storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, the computer-readable storage medium can be a machine-readable signal medium. More specific examples of the machine-readable storage medium would include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, compact disc read-only memory (CD ROM), optical storage device, magnetic storage device, or any suitable combination of the foregoing.

[0096] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor; and a keyboard and a pointing device, such as a mouse or a trackball, by which the user can provide input to the electronic device. Other kinds of devices can also be used to provide interaction with the user, and the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0097] The embodiments described in the present invention can be implemented in a computing system including a backend component (e.g., as a data server), or a computing system including a middleware component (e.g., an application server), or a computing system including a frontend component (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system including any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected to each other by digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include: local area network (LAN), wide area network (WAN), blockchain network, and the Internet.

[0098] The computing system can include a client and a server. The client and the server are generally far from each other and usually interact through a communication network. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system, and solves the defects of difficult management and weak business scalability existing in traditional physical hosts and VPS services.

[0099] It should be understood that various forms of the processes shown above can be used, with steps reordered, added, or deleted. For example, the steps recited in the present invention can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved, and this embodiment is not limited herein.

[0100] In summary, the carotid plaque segmentation method, device, equipment and medium provided by the embodiments of the present invention construct a carotid plaque segmentation network, train and test it to achieve end-to-end segmentation of the carotid ultrasound image to be segmented. The carotid plaque segmentation network includes a feature extraction module, N multi-scale attention modules, a fusion module and N-1 boundary supplementation modules. The feature extraction module is used to extract rich multi-layer features f 1 to f N , and the multi-scale attention module is used to capture and integrate f 1 to f N to obtain the corresponding multi-layer perception features F 1 to F N , so as to accurately identify carotid plaques with different sizes and shapes; the fusion module can generate the prediction map S 2 to S N of the Nth layer according to F N ; the boundary supplementation module combines average pooling and dilation-erosion to enhance the boundary representation, and generates the prediction map S a of the current layer according to the perception feature F a+1 of the current layer, the perception feature F a+1 of the adjacent high layer and the prediction map S a , and the prediction map S 1 of the first layer is selected as the final segmentation result. In addition, the present invention considers calculating the combined loss for the prediction map of each scale, which helps to improve the segmentation accuracy. Compared with a variety of mainstream medical image segmentation methods, the present invention has achieved the best segmentation results and realized accurate segmentation of carotid plaques in ultrasound images.

[0101] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited by the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications made without departing from the spirit and principle of the present invention shall be equivalent replacement methods and are all included in the protection scope of the present invention.

Claims

1. A method for segmenting carotid artery plaque in an ultrasound image, characterized in that: include: Construct a carotid plaque segmentation network, which includes a feature extraction module, N multi-scale attention modules, a fusion module and N-1 boundary supplement modules; the feature extraction module is used to extract features from the input image to obtain corresponding multi-layer features f1 to f N , N ≥ 3; N multi-scale attention modules are used to analyze the multi-layer features f1 to f N Attention extraction is performed respectively to obtain the corresponding multi-layer perception features F1 to F N The fusion module is used to combine the multi-layer perception features F2 to F N Upsampling and fusion are performed step by step to generate the prediction graph S of the Nth layer N The boundary supplement module of the current layer is used to supplement the current layer according to the perceptual features F a and the perceptual features F of the adjacent high-level a+1 and prediction graph S a+1 Generate the prediction map S of the current layer a , a∈{1,2,,N-1}, where S1 is selected as the final segmentation result; Using a training set to train the carotid plaque segmentation network and using a test set to test the trained carotid plaque segmentation network; The carotid artery ultrasound image to be segmented is input to the tested carotid artery plaque segmentation network, and the carotid artery plaque segmentation network outputs the corresponding carotid artery plaque segmentation image.

2. The method for segmenting carotid artery plaque in an ultrasonic image according to claim 1, characterized in that: The operations performed by the multi-scale attention module are: The feature f of the i-th layer i Generate the query matrix Q, key matrix K and value matrix V through three e×e convolutions respectively, i∈{1,2,,N}, e is the convolution kernel size; The query matrix Q, key matrix K and value matrix V are downsampled by three downsampling modules respectively. Each downsampling module downsamples by a predefined M downsampling factor and projects them to different scale spaces through M e×e convolutions to obtain the downsampled query matrix Q. j , downsampling key matrix K j and the downsampled value matrix V j ,j∈{1,2,,M}; According to the down-sampling query matrix Q j and the downsampling key matrix K j Generate weight graph For the downsampled value matrix V j Fusion to get the value matrix And the value matrix Add to the value matrix V after 1×1 convolution to generate the value matrix V o ; The value matrix Add to the value matrix V after 1×1 convolution to generate the value matrix V o ; The weight map and value matrix V o Perform element-by-element multiplication to generate the feature map V′ and f i The 1×1 convolution results are added together to finally obtain the multi-layer perception feature F i .

3. The method for segmenting carotid artery plaque in an ultrasonic image according to claim 2, characterized in that: According to the down-sampling query matrix Q j and the downsampling key matrix K j Generate weight graph The specific process is: By downsampling the query matrix Q in the same scale space j and the downsampling key matrix K j Perform e×e convolution and multiply to get the attention map, and then adjust the attention maps of all scale spaces to a uniform size through bilinear interpolation and take the average to get the weight map 4. The method for segmenting carotid artery plaque in an ultrasonic image according to claim 3, characterized in that: For the downsampled value matrix V j Fusion to get the value matrix The specific process is: For the downsampled value matrix V j M scales V1 to V M Upsample according to the predefined M upsampling factors, and project them to different scale spaces through M e×e convolutions, add them together, and then perform e×e convolution to obtain the value matrix 5. The method for segmenting carotid artery plaque in an ultrasonic image according to claim 1, characterized in that: The operations performed by the boundary supplementation module are: The perceptual feature F of the current layer a And the perceptual features F of the adjacent high-level layers of the upsampled a+1 After e×e convolution processing, they are added to obtain the perceptual features. The prediction graph S of the adjacent high-level a+1 After upsampling and perceptual features Multiply them together to get the feature map S a ′; the feature map S a ′ is processed by a local average pooling composed of an e×e convolution kernel, and the processed features are combined with S a ′, obtain the boundary clue B1; the feature map S a ′ is concatenated with the boundary clue B1 to obtain the feature map The prediction graph S of the adjacent high-level a+1 The boundary clue B2 is obtained by dilation and corrosion operations and then subtraction; the boundary clue B2 is combined with the perceptual feature After multiplication, the perceptual features Add and generate feature maps through e×e convolution The feature map and After performing e×e convolutions respectively, they are concatenated, and the concatenated results are passed through the spatial attention unit and the channel attention unit to extract the attention, and then the attention is passed through the e×e convolution and then through the 1×1 convolution to obtain the prediction map S of the current layer. a .

6. The method for segmenting carotid artery plaque in an ultrasonic image according to claim 1, characterized in that: The feature extraction module uses a pre-trained PVTv2-B2 network.

7. The method for segmenting carotid artery plaque in an ultrasonic image according to any one of claims 1 to 6, characterized in that: The loss function used to train the carotid plaque segmentation network is S i represents the i-th prediction graph, G m represents the true annotation mask, and L() represents the weighted binary cross entropy loss L wB and weighted intersection-over-union loss L wI sum.

8. A device for segmenting carotid artery plaques in an ultrasonic image, applied to a method for segmenting carotid artery plaques in an ultrasonic image according to any one of claims 1 to 7, characterized in that: The device includes a network construction part, a network training and testing part and a network application part, wherein the network construction part is used to construct a carotid plaque segmentation network; the network training and testing part is used to train the carotid plaque segmentation network with a training set and to test the trained carotid plaque segmentation network with a test set; the network application part is used to input a carotid artery ultrasound image to be segmented into the tested carotid plaque segmentation network, and the carotid plaque segmentation network outputs a corresponding carotid plaque segmentation image.

9. A device for segmenting carotid artery plaque in an ultrasound image, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method for segmenting carotid plaque in an ultrasound image as described in any one of claims 1 to 7.

10. A storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the method for segmenting carotid artery plaque in an ultrasound image as claimed in any one of claims 1 to 7 is implemented.

Images