Carotid artery plaque segmentation method and device in ultrasound image, equipment and medium

Abstract

The application relates to the technical field of image processing, and particularly discloses a carotid plaque segmentation method and device in an ultrasonic image, equipment and a medium, which constructs a carotid plaque segmentation network and trains and tests the carotid plaque segmentation network to realize end-to-end segmentation of a to-be-segmented carotid ultrasonic image, the carotid plaque segmentation network comprises a feature extraction module, N multi-scale attention modules, a fusion module and N-1 boundary supplement modules; the feature extraction module is used for extracting rich multi-layer features f1 to f N ; the multi-scale attention module is used for capturing and integrating the multi-layer features f1 to f N , to obtain multi-layer perception features F1 to F N ; the fusion module is used for generating a prediction map S N of the Nth layer according to F2 to F N ; the boundary supplement module generates a prediction map S a of the current layer according to a perception feature F a of the current layer, a perception feature F a+1 of an adjacent high layer and a prediction map S a+1 of the adjacent high layer, and the prediction map S1 of the first layer is selected as the final segmentation result. Compared with various mainstream medical image segmentation methods, the application achieves the best segmentation result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a carotid plaque segmentation method in an ultrasound image, a device, equipment and a medium. BACKGROUND

[0002] Carotid ultrasound is a commonly used non-invasive diagnostic tool that can detect various blood flow parameters such as blood flow velocity, blood flow width, and arterial wall thickness and irregularities. These parameters are of great significance for assessing arterial health and potential disease risk. Among them, the carotid intima-media thickness (IMT) is one of the important indicators reflecting atherosclerosis and cardiovascular disease risk. When the IMT exceeds 1.3 millimeters, it is generally considered that carotid plaques have formed. The presence of these plaques is closely related to serious diseases such as cerebral infarction and stroke. Therefore, by segmenting carotid plaques in ultrasound images, doctors can accurately identify and measure the size, shape and distribution of carotid plaques, which plays a crucial role in the early prevention and precise intervention of stroke. However, manual pixel-by-pixel annotation of ultrasound images by ultrasound physicians is subjective and laborious. Therefore, it is of great significance to perform automatic carotid plaque segmentation tasks based on ultrasound images.

[0003] In recent years, with the rapid development of deep learning, deep neural networks have shown great ability in the field of medical image segmentation and have made breakthroughs in many tasks. However, due to the high heterogeneity of carotid plaques, their complex shapes, and the blurred and irregular boundaries, the performance of some existing general segmentation algorithms in this task is still not ideal, especially when dealing with complex plaque morphology and blurred boundaries. Therefore, carotid plaque segmentation mainly faces the following challenges: (1) In ultrasound images, due to imaging noise, artifacts and resolution limitations, the boundaries of plaques are often blurred and have low contrast with surrounding tissues. In addition, the irregularity of plaque morphology further increases the difficulty of automatic segmentation. (2) Carotid plaques in different patients or different stages can have significant shape differences, and show high heterogeneity in tissue structure and texture distribution. This diversity makes it difficult for the model to capture general features of plaques, thereby affecting the stability and accuracy of segmentation. SUMMARY

[0004] The present application provides a carotid plaque segmentation method in an ultrasound image, device, equipment and medium, which solves the technical problem of how to accurately segment carotid plaques in ultrasound images.

[0005] To solve the above technical problems, the present application provides a carotid plaque segmentation method in an ultrasound image, comprising:

[0006] The carotid plaque segmentation network comprises a feature extraction module, N multi-scale attention modules, a fusion module and N-1 boundary supplement modules; the feature extraction module is used for performing feature extraction on an input image in N scales to obtain corresponding multi-layer features f1 to f N , N≥3; the N multi-scale attention modules are used for performing attention extraction on the multi-layer features f1 to f N respectively to obtain corresponding multi-layer perception features F1 to F N in N scales; the fusion module is used for performing upsampling and fusion on the multi-layer perception features F2 to F N step by step to generate a prediction map S N of the Nth layer; the boundary supplement module of the current layer is used for generating a prediction map S a of the current layer according to the perception feature F a+1 of the current layer and the perception features F a+1 and the prediction map S a of the adjacent higher layer, a∈{1,2,…,N-1}, wherein S1 is selected as the final segmentation result;

[0007] The carotid plaque segmentation network is trained by using a training set and tested by using a test set;

[0008] The carotid plaque segmentation network outputs a corresponding carotid plaque segmentation image.

[0009] Further, the multi-scale attention module performs the following operations:

[0010] The feature f i of the ith layer is respectively generated into a query matrix Q, a key matrix K and a value matrix V through three e×e convolutions, i∈{1,2,…,N}, and e is a convolution kernel size;

[0011] The query matrix Q, the key matrix K and the value matrix V are respectively down-sampled through three down-sampling modules, each down-sampling module is down-sampled according to a pre-defined M down-sampling factor and is projected into different scale spaces through M e×e convolutions to obtain a down-sampled query matrix Q j , a down-sampled key matrix K j and a down-sampled value matrix V j , j∈{1,2,…,M};

[0012] The down-sampled query matrix Q j and the down-sampled key matrix K j are used to generate a weight map

[0013] For the downsampled value matrix V j The value matrix is ​​obtained by fusion. and value matrix Add the result to the value matrix V after 1×1 convolution to generate the value matrix V. o ;

[0014] Value matrix Add the result to the value matrix V after 1×1 convolution to generate the value matrix V. o ;

[0015] Weight graph The sum-value matrix V o Element-wise multiplication is performed, and the resulting feature map V′ is compared with f. i The 1×1 convolution results are added together to finally obtain the multilayer perceptual feature F. i .

[0016] Furthermore, based on the downsampling query matrix Q j and the downsampling key matrix K j Generate weight graph The specific process is as follows:

[0017] By downsampling the query matrix Q in the same scale space j and the downsampling key matrix K j An e×e convolution is performed and multiplied to obtain an attention map. Then, bilinear interpolation is used to adjust the attention maps of all scales to a uniform size, and the average value is taken to obtain the weight map.

[0018] Furthermore, for the downsampled value matrix V j The value matrix is ​​obtained by fusion. The specific process is as follows:

[0019] For the downsampled value matrix V j M scales V1 to V M Upsampling is performed using M predefined upsampling factors, and then each upsampling is projected onto different scale spaces through M e×e convolutions. The samples are then summed and followed by an e×e convolution to obtain the value matrix.

[0020] Furthermore, the boundary compensation module performs the following operations:

[0021] The perceptual features F of the current layer a Perceptual features F of adjacent high layers and upsampled a+1 The perceptual features are obtained by performing e×e convolutions on each component and then summing them. The prediction map S of adjacent high-rise buildings a+1 After upsampling and perceptual features S' = S * S a ; S' = S - S a Local average pooling processing composed of an e x e convolution kernel, and the processed feature and S' a are subtracted to obtain boundary clues B1; the feature map S' a is spliced with the boundary clues B1 to obtain a feature map

[0022] The prediction map S a+1 of the adjacent high layer is subtracted after being expanded and eroded respectively to obtain boundary clues B2; the boundary clues B2 are multiplied with the perception feature , and then added to the perception feature , and an e x e convolution is performed to generate a feature map

[0023] The feature map is spliced after being subjected to e x e convolution respectively, and the spliced result is extracted through a spatial attention unit and a channel attention unit, and then through e x e convolution and 1 x 1 convolution, a prediction map S a of the current layer is obtained.

[0024] Further, the feature extraction module adopts a pre-trained PVTv2-B2 network.

[0025] Further, the loss function used for training the carotid plaque segmentation network is S i represents the i-th prediction map, G m represents the real label mask, L() represents the weighted binary cross-entropy loss L wB and the weighted intersection over union loss L wI .

[0026] The application also provides a carotid plaque segmentation device in an ultrasound image, which is applied to the carotid plaque segmentation method in the ultrasound image, and the key lies in that the device comprises a network construction part, a network training and testing part and a network application part, the network construction part is used for constructing a carotid plaque segmentation network; the network training and testing part is used for training the carotid plaque segmentation network by using a training set and testing the trained carotid plaque segmentation network by using a test set; and the network application part is used for inputting a carotid ultrasound image to be segmented into the tested carotid plaque segmentation network, and the carotid plaque segmentation network outputs a corresponding carotid plaque segmentation image.

[0027] The application also provides a carotid plaque segmentation device in an ultrasound image, and the key lies in that the device comprises:

[0028] One or more processors;

[0029] Memory, used to store one or more programs;

[0030] When the one or more programs are executed by the one or more processors, the one or more processors implement the carotid plaque segmentation method in the ultrasound image.

[0031] The present invention also provides a storage medium, the key feature of which is that a computer program is stored thereon, which, when executed by a processor, implements the carotid plaque segmentation method in the ultrasound image.

[0032] The present invention provides a method, apparatus, device, and medium for segmenting carotid artery plaques in ultrasound images. It constructs a carotid artery plaque segmentation network and trains and tests it to achieve end-to-end segmentation of the ultrasound image of the carotid artery to be segmented. The carotid artery plaque segmentation network includes a feature extraction module, N multi-scale attention modules, a fusion module, and N-1 boundary completion modules. The feature extraction module is used to extract rich multi-layer features f1 to f2. N The multi-scale attention module is used to capture and integrate multi-layer features f1 to f2. N The corresponding multilayer perceptual features F1 to F2 are obtained. N This allows for the precise identification of carotid artery plaques of varying sizes and shapes; the fusion module adjusts the plaque size based on F2 to F2. N Generate the prediction map S of the Nth layer N The boundary enhancement module combines average pooling and dilation-erosion to enhance the boundary representation, based on the perceptual features F of the current layer. a and the perception features F of adjacent high-rise buildings a+1 And prediction map S a+1 Generate the prediction map S of the current layer a The predicted image S1 from the first layer is selected as the final segmentation result. Furthermore, this invention considers calculating the combined loss of the predicted images at each scale, which helps improve segmentation accuracy. Compared with various mainstream medical image segmentation methods, this invention achieves the best segmentation results, realizing accurate segmentation of carotid artery plaques in ultrasound images. Attached Figure Description

[0033] Figure 1 This is a structural diagram of the carotid plaque segmentation network provided in an embodiment of the present invention;

[0034] Figure 2 This is a structural diagram of the multi-scale attention module provided in an embodiment of the present invention;

[0035] Figure 3 This is a structural diagram of the fusion module provided in an embodiment of the present invention;

[0036] Figure 4is a structural diagram of the boundary supplement module provided by the embodiment of the application. DETAILED DESCRIPTION

[0037] The embodiments of the application are described below in detail with reference to the drawings, the embodiments are given only for illustrative purposes and cannot be understood as limiting the application, the accompanying drawings are only for reference and illustration and do not constitute a limitation on the patent protection scope of the application, because many changes can be made to the application without departing from the spirit and scope of the application.

[0038] The application is directed to the carotid plaque segmentation problem in an ultrasound image, first, a high-performance multiscale fusion and boundary enhancement carotid plaque segmentation method in an ultrasound image is proposed, aiming at accurately segmenting the carotid plaque in the carotid ultrasound image to assist clinicians to more quickly and accurately screen and diagnose cerebral infarction and stroke.

[0039] The carotid plaque segmentation method in an ultrasound image provided by the embodiment of the application comprises the steps of:

[0040] Constructing a carotid plaque segmentation network;

[0041] Training the carotid plaque segmentation network using a training set and testing the trained carotid plaque segmentation network using a test set;

[0042] Inputting the carotid ultrasound image to be segmented into the tested carotid plaque segmentation network, and the carotid plaque segmentation network outputs the corresponding carotid plaque segmentation image.

[0043] The BSNet proposed by the application follows an end-to-end framework, that is, only the data needs to be input to obtain the predicted output, and no other operations are needed in the middle.

[0044] The architecture of the carotid plaque segmentation network (referred to as BSNet in this example) constructed by the application is as shown in Figure 1 The carotid plaque segmentation network includes a feature extraction module, N multiscale attention modules (MASM), a fusion module (PD) and N-1 boundary supplement modules (BSM);

[0045] The feature extraction module is used for feature extraction of the input image in N scales to obtain corresponding multi-layer features f1 to f N , N>=3;

[0046] The N multiscale attention modules are used for attention extraction of the multi-layer features f1 to f N respectively to obtain corresponding multi-layer perception features F1 to F N in N scales;

[0047] The fusion module is used for fusing the multi-layer perception features F2 to FN Upsampling and fusion are performed progressively to generate the prediction map S of the Nth layer. N ;

[0048] The current layer boundary completion module is used to supplement the current layer's perceptual features F a and the perception features F of adjacent high-rise buildings a+1 And prediction map S a+1 Generate the prediction map S of the current layer a , a∈{1,2,...,N-1}, where S1 is selected as the final segmentation result.

[0049] This embodiment uses the PVTv2-B2 network (pre-trained on ImageNet-1K) as the feature extraction module because it can extract multi-level features and performs well in various visual tasks. Figure 1 Taking N=4 as an example, the multi-layer features f1 to f4 generated by the feature extraction module are represented as follows: Where i represents the i-th layer of the feature extraction module, there are a total of 4 layers, H and W are the length and width of the input image, respectively, and C i (i∈{1,2,3,4}) represents the channel dimension of the features in the i-th layer of the feature extraction module. The vector dimension representing this multi-scale feature is C. i ×H / 2 i ×W / 2 i Of course, the feature extraction module can also employ other multi-layered feature extraction networks that perform well in various visual tasks.

[0050] In ultrasound images, carotid artery plaques typically exhibit significant variations in size, shape, and texture, posing a critical challenge to accurate segmentation. In this invention, this embodiment proposes a multi-scale attention module (MSAM) to capture and integrate multi-scale features, thereby accurately identifying carotid artery plaques with different sizes and shapes.

[0051] To process multi-layer features f1 to f2 at N scales N Attention is extracted separately, and the structure of each multi-scale attention module is as follows: Figure 1 As shown, N multi-scale attention modules attend to multi-layer features f1 to f2 at N scales. N One-to-one extraction is performed to obtain multilayer perceptual features F1 to F2 at N scales. N .exist Figure 1 Taking N=4 as an example, the structure of each multi-scale attention module is as follows: Figure 2 As shown.

[0052] like Figure 2 As shown, the operations performed by the multi-scale attention module are as follows:

[0053] 1, the feature f of the i-th layer is denoted as f i The query matrix Q, the key matrix K, and the value matrix V are respectively generated by three e x e (e is the size of the convolution kernel, 1 < e < 6, the value of e is not too small, the convolution effect is not good, the value of e is not too large, the calculation complexity is high, and in this example, e is preferably 3) convolutions.

[0054] 2, the query matrix Q, the key matrix K, and the value matrix V are respectively down-sampled by three down-sampling modules (DS), each down-sampling module is down-sampled according to a pre-defined M down-sampling factor (in this example, 2 < M < 6, the value of M is not too small, the down-sampling effect is not good, the value of M is not too large, the calculation complexity is high, and in this example, M is preferably 4, and the corresponding down-sampling factors are 8, 4, 2, and 1), and are respectively projected into different scale spaces by M e x e convolutions to obtain a down-sampled query matrix Q j , a down-sampled key matrix K j , and a down-sampled value matrix V j , j e {1, 2, …, M}.

[0055] 3, a weight map W is generated according to the down-sampled query matrix Q j and the down-sampled key matrix K j In each scale space, e x e convolutions are performed on the down-sampled query matrix Q j and the down-sampled key matrix K j of the same scale space and multiplied to obtain an attention map, and then the attention maps of all scale spaces are adjusted to a uniform size by bilinear interpolation (up-sampling) and averaged (avg) to obtain a weight matrix W

[0056]

[0057] , where σ (·) represents a normalization operation, d is the dimension of the query matrix Q and the key matrix K, ↑ (·) represents an up-sampling operation, Conv e (·) represents e x e convolution, and T represents matrix transposition.

[0058] 4, the down-sampled value matrix V j is fused to obtain a value matrix V The M scales V1 to V M of the down-sampled value matrix V j are respectively up-sampled according to a pre-defined M up-sampling factor (in this example, M is preferably 4, and the corresponding up-sampling factors are 1, 2, 4, and 8), and are respectively projected into different scale spaces by M e x e convolutions, added, and then e x e convolved to obtain a value matrix V

[0059] ​

[0060] 5. Value matrix Add the result to the value matrix V after 1×1 convolution to generate the value matrix V. o :

[0061]

[0062] The adjustment factor α is used to automatically adjust the value matrix during training. The contribution, Conv1(·) represents 1×1 convolution, This indicates element-wise addition.

[0063] 6. Weight the graph The sum-value matrix V o Element-wise multiplication is performed, and the resulting feature map V′ is compared with f. i The 1×1 convolution results are summed to obtain the feature map F. i :

[0064]

[0065] The fusion module (PD module) integrates multilayer sensing features F2 to F... N Upsampling and fusion are performed progressively to generate the prediction map S of the Nth layer. N PD blocks for multilayer sensing features F2 to F N And their respective high-level features (if present) are multiplied element-wise. F N The features multiplied by the multiplier are concatenated along the channel dimension and then passed through an e×e convolutional block to generate a prediction map S. N F1 is excluded from the PD block because it contributes little to performance improvement but consumes a lot of computational resources. Taking N=4 as an example, the structure of the PD module is as follows: Figure 3 As shown.

[0066] Since a single method may not fully capture boundary features, this embodiment combines average pooling and dilation-erosion to enhance boundary representation. This comprehensive approach effectively strengthens the extraction and representation of boundary features, thereby improving overall performance. Figure 4 As shown, the Boundary Completion Module (BSM) uses the perceptual features F of the current layer. a and the perception features F of adjacent high-rise buildings a+1 And prediction map S a+1 As input, generate the prediction map S for the current layer. a Specifically, this process includes the following steps:

[0067] 1. Transfer the perceptual features F of the current layer a Perceptual features F of adjacent high layers and upsampled a+1(are added after e×e convolution processing respectively, to obtain the perception feature a The same scale) are added after e×e convolution processing respectively, to obtain the perception feature

[0068]

[0069] 2、The prediction map S a+1 is up-sampled (to the same scale as the perception feature ) and multiplied by the perception feature , to obtain the feature map S' a :

[0070]

[0071] 3、The feature map S' a is processed by a local average pooling (Avg(·)) composed of an e×e convolution kernel, and the processed feature is subtracted from the feature map S' a , to obtain the boundary clue B1:

[0072]

[0073] 4、In order to highlight the boundary area, the feature map S' a is spliced with the boundary clue B1, to obtain the feature map

[0074] 5、The prediction map S a+1 of the adjacent high layer is subtracted after being dilated and eroded respectively, to obtain the boundary clue B2; the boundary clue B2 is multiplied by the perception feature , and then added to the perception feature , and an e×e convolution is performed to generate the feature map

[0075]

[0076] 6、The feature map is spliced after e×e convolution respectively, and the spliced result is extracted by a spatial attention (SA) unit and a channel attention (CA) unit to obtain key features, which are processed by e×e convolution and then by 1×1 convolution to generate the prediction map S a of the current layer. The spatial attention (SA) unit and the channel attention (CA) unit are used to extract key features to remove irrelevant information, wherein the spatial attention (SA) unit and the channel attention (CA) unit come from the widely used CBAM module.

[0077] In the process of training the carotid plaque segmentation network, a combined loss function L is adopted, which combines a weighted binary cross-entropy loss L wBand weighted intersection over union loss L wI i.e., L = L wB + L wI To obtain better segmentation results, a deep supervision mechanism is also applied, which applies L to each prediction map. The total loss function is expressed as follows:

[0078]

[0079] where S i represents the i-th prediction map in S m , and G i represents the true annotation mask. It should be noted that before calculating the loss, the resolution of S m is adjusted to be consistent with G N .

[0080] In the present application, the present embodiment collects a dataset named CP-Seg for the segmentation of carotid plaque in ultrasound images. The CP-Seg dataset contains 3,500 ultrasound images and their corresponding pixel-wise segmentation masks for model training, and additionally 500 ultrasound images and their corresponding segmentation masks for model testing. The image resolution is 352x352, and the images are standardized to remove unnecessary background information and ensure privacy protection.

[0081] The present embodiment uses the Adam optimizer to optimize the BSNet, with an initial learning rate of 1e-4. The training process of the BSNet lasts for 100 epochs, with a batch size of 16, and the learning rate is decayed once every 20 epochs with a decay factor of 0.1. To alleviate overfitting, various data augmentation strategies are used, including random horizontal flipping and rotation. In addition, the size of the input image is adjusted to 352x352, and a multi-scale training method is used. This strategy adjusts the image size by randomly selecting a scale from {0.75, 1, 1.25}.

[0082] The present embodiment uses two widely recognized evaluation indicators in the field of medical image segmentation to measure network performance. These indicators include the Dice similarity coefficient (Dice) and the intersection over union (IoU). Generally, the larger the Dice, IoU, the better the performance.

[0083] The network proposed in the present application is compared with eight mainstream medical image segmentation methods, including U-Net, U-Net++, CE-Net, U-Net3+, CaraNet, BATFormer, I2U-Net and SHFormer. The test results on the CP-Seg dataset are shown in Table 1, and the best results are shown in bold for easy observation. As can be seen from Table 1, the method proposed in the present embodiment achieves the best results, and is superior to the eight mainstream medical image segmentation methods in two evaluation indicators.

[0084] Table 1 Experimental results

[0085] Method Dice IoU U-Net 0.695 0.648 U-Net++ 0.791 0.675 CE-Net 0.795 0.675 U-Net3+ 0.749 0.623 CaraNet 0.787 0.680 BATFormer 0.783 0.671 I2U-Net 0.730 0.605 SHFormer 0.780 0.661 BSNet 0.814 0.698

[0086] The present embodiment also provides an apparatus for carotid plaque segmentation in an ultrasound image, which is applied to the carotid plaque segmentation method in the ultrasound image described above. The apparatus comprises a network construction part, a network training and testing part and a network application part. The network construction part is used to construct a carotid plaque segmentation network. The network training and testing part is used to train the carotid plaque segmentation network using a training set and test the trained carotid plaque segmentation network using a test set. The network application part is used to input the carotid ultrasound image to be segmented into the tested carotid plaque segmentation network, and the carotid plaque segmentation network outputs the corresponding carotid plaque segmentation image.

[0087] The present embodiment also provides an apparatus for carotid plaque segmentation in an ultrasound image, which comprises:

[0088] one or more processors;

[0089] a memory for storing one or more programs;

[0090] When the one or more programs are executed by the one or more processors, the one or more processors implement the carotid plaque segmentation method in the ultrasound image. The processor can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor include but are not limited to a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc.

[0091] The present embodiment also provides a storage medium having a computer program stored thereon, which is executed by a processor to implement the carotid plaque segmentation method in the ultrasound image described above. In some embodiments, the carotid plaque segmentation method in the ultrasound image can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit.

[0092] In some embodiments, portions or all of the computer program can be loaded onto an electronic device via the ROM and / or the communication unit. When the computer program is loaded onto the RAM and executed by the processor, one or more steps of the carotid plaque segmentation method in an ultrasound image described above can be performed. Alternatively, in other embodiments, the processor can be configured to perform the carotid plaque segmentation method by other any suitable means, for example, by means of firmware.

[0093] Various implementations of the systems and techniques described above in this embodiment can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (PLD), a computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0094] Computer programs used to implement the methods of the present application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the computer program, when executed by the processor of the machine, implements the functions / acts specified in the flow diagrams and / or block diagrams. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0095] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer-readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of a machine-readable storage medium will include one or more lines of a program of instructions in a transitory signal, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0096] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0097] Embodiments of the application described can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here, or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0098] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. A server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of large management difficulty and weak business scalability in traditional physical hosts and VPS services.

[0099] It should be understood that the steps shown above can be reordered, added to, or deleted from. For example, each step described in the present application can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions of the present application can be achieved, and the present embodiment is not limited herein.

[0100] To sum up, the carotid plaque segmentation method, device, equipment and medium in the ultrasound image provided by the embodiment of the present application construct a carotid plaque segmentation network and train and test it to realize end-to-end segmentation of the carotid ultrasound image to be segmented. The carotid plaque segmentation network includes a feature extraction module, N multi-scale attention modules, a fusion module and N-1 boundary supplement modules. The feature extraction module is used to extract rich multi-layer features f1 to f N The multi-scale attention module is used to capture and integrate f1 to f N to obtain corresponding multi-layer perception features F1 to F N , so as to accurately identify carotid plaques with different sizes and shapes; the fusion module is used to generate the prediction map S N of the Nth layer according to F2 to F N ; the boundary supplement module combines average pooling and dilation-erosion to enhance the boundary representation, generates the prediction map S a of the current layer according to the perception feature F a+1 of the current layer and the perception features F a+1 and the prediction map S a of the adjacent higher layer, the prediction map S1 of the first layer is selected as the final segmentation result. In addition, the present application considers the calculation of the loss of the combination of the prediction map of each scale, which helps to improve the segmentation accuracy. Compared with a variety of mainstream medical image segmentation methods, the present application achieves the best segmentation result and realizes accurate segmentation of carotid plaques in ultrasound images.

[0101] The above embodiment is a preferred embodiment of the present application, but the embodiments of the present application are not limited by the above embodiment, and any changes, modifications, substitutions, combinations, simplifications made without departing from the spirit and principles of the present application shall be equivalent replacement methods and shall be included in the protection scope of the present application.

Claims

1. A method of carotid plaque segmentation in ultrasound images, characterized in that, The method comprises the following steps: The carotid plaque segmentation network comprises a feature extraction module, N multi-scale attention modules, a fusion module and N-1 boundary supplement modules; the feature extraction module is used for feature extraction of an input image to obtain corresponding multi-layer features to , N≥3; the N multi-scale attention modules are used for attention extraction of the multi-layer features to respectively to obtain corresponding multi-layer perception features to ; the fusion module is used for step-by-step upsampling and fusion of the multi-layer perception features to to generate a prediction map of the Nth layer ; the boundary supplement module of the current layer is used for generating a prediction map of the current layer according to the perception feature of the current layer and the perception features and the prediction map of an adjacent higher layer , wherein is selected as the final segmentation result; training the carotid plaque segmentation network by using a training set and testing the trained carotid plaque segmentation network by using a test set; inputting a carotid ultrasound image to be segmented into the tested carotid plaque segmentation network, and the carotid plaque segmentation network outputs a corresponding carotid plaque segmentation image; The operation performed by the multi-scale attention module is: The features of the i-th layer are The query matrix Q, the key matrix K and the value matrix V are respectively generated by three e x e convolutions, , e is the size of the convolution kernel The query matrix Q, the key matrix K and the value matrix V are respectively down-sampled by three down-sampling modules, each down-sampling module is down-sampled according to a predefined M down-sampling factor and is respectively projected to different scale spaces by M e x e convolution, to obtain a down-sampled query matrix , a down-sampled key matrix and a down-sampled value matrix , ; According to the down-sampling query matrix and the down-sampling key matrix generating a weight map ; down-sampling value matrix fusion to obtain a value matrix , and adding the value matrix to a value matrix processed by 1x1 convolution, to generate a value matrix ; Weight graph AND-value matrix Element-wise multiplication generates a feature map. and The 1×1 convolution results are added together to obtain the multilayer perceptual features. ; According to the down-sampling query matrix and the down-sampling key matrix to generate the weight map The specific process is as follows: Query matrix by down-sampling the same scale space and down-sampling key matrix Perform e x e convolution and multiply to get attention map, and then take the average value after adjusting the attention map of all scale spaces to the same size by bilinear interpolation to get the weight map .

2. The carotid plaque segmentation method in ultrasound images according to claim 1, characterized in that, down-sampled value matrix fusion to obtain a value matrix The specific process is as follows: M scales of down-sampled value matrix Up-sampled by predefined M up-sampling factors respectively, and added after projected to different scale spaces by M e x e convolutions respectively, and then e x e convolution to get value matrix .​​ 3. The carotid plaque segmentation method in ultrasound images according to claim 1, characterized in that, The operation performed by the boundary supplement module is: the perception feature of the current layer is added to the perception feature of the adjacent high layer after being up-sampled the prediction map of the adjacent high layer is multiplied by the perception feature after being up-sampled ​​​​​​​​​​​ Predictions of adjacent high-rise buildings Boundary clues are obtained by subtracting the expansion and corrosion operations respectively. ; Boundary clues With perceptual features Multiply and then combine with the perceived features The sums are then processed through an e×e convolution to generate a feature map. ; The feature maps and are respectively subjected to e×e convolution, spliced, and the spliced result is subjected to attention extraction through a spatial attention unit and a channel attention unit, and then the attention is subjected to e×e convolution and 1×1 convolution to obtain a prediction map of the current layer .

4. The carotid plaque segmentation in ultrasound images method of claim 1, wherein: The feature extraction module adopts a pre-trained PVTv2-B2 network.

5. The carotid plaque segmentation method in ultrasound images according to any one of claims 1 to 4, characterized in that, The loss function used in training the carotid artery plaque segmentation network is , represents the first prediction map, represents the true labeled mask, ( ) represents the sum of the weighted binary cross-entropy loss and the weighted intersection over union loss .

6. The apparatus for carotid plaque segmentation in ultrasound images according to any one of claims 1 to 5, applied to the method for carotid plaque segmentation in ultrasound images according to any one of claims 1 to 5, characterized in that: The device comprises a network construction part, a network training and testing part and a network application part, the network construction part is used for constructing a carotid plaque segmentation network; the network training and testing part is used for training the carotid plaque segmentation network by using a training set and testing the trained carotid plaque segmentation network by using a test set; and the network application part is used for inputting a carotid ultrasound image to be segmented into the tested carotid plaque segmentation network, and the carotid plaque segmentation network outputs a corresponding carotid plaque segmentation image.

7. Carotid plaque segmentation apparatus in an ultrasound image, characterized by, The device comprises a network construction part, a network training and testing part and a network application part, the network construction part is used for constructing a carotid plaque segmentation network; the network training and testing part is used for training the carotid plaque segmentation network by using a training set and testing the trained carotid plaque segmentation network by using a test set; and the network application part is used for inputting a carotid ultrasound image to be segmented into the tested carotid plaque segmentation network, and the carotid plaque segmentation network outputs a corresponding carotid plaque segmentation image. one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the carotid plaque segmentation method in an ultrasound image as claimed in any one of claims 1 to 5.

8. A storage medium characterized by: A computer program is stored thereon, and the computer program is executed by a processor to implement the carotid plaque segmentation method in an ultrasound image as claimed in any one of claims 1 to 5.

Images