Ultrasonic image processing method and device, electronic equipment and storage medium

Abstract

The invention provides an ultrasonic image processing method and device, electronic equipment and a storage medium. The ultrasonic image processing method comprises the following steps: acquiring an intravascular ultrasonic image; the intravascular ultrasound image is input into an image segmentation model to obtain a segmented image, the image segmentation model is used for carrying out image segmentation on the calcified plaque of the intravascular ultrasound image, and the image segmentation model comprises a prediction module and a residual error refining module which are connected with each other; the prediction module comprises at least one convolution block attention module; and determining an area occupied by the calcified plaque based on the calcified plaque in the segmented image. According to the scheme, the area occupied by the calcified plaque can be determined more automatically and more accurately according to the intravascular ultrasound image.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an ultrasound image processing method, device, electronic device, storage medium and computer program product. Background Art

[0002] Intravascular ultrasound (IVUS) can be used to examine the inner walls of blood vessels. Specifically, IVUS can assess the morphology, severity, and extent of intravascular plaque, quantitatively analyze the degree of vascular stenosis, and measure the minimum lumen area. In addition to the extent of calcification, IVUS examinations can also accurately measure the length of calcified lesions.

[0003] Typically, IVUS image sequences can contain hundreds or even thousands of images, making it time-consuming and labor-intensive for clinicians to identify and delineate calcified plaques. Currently, clinicians' IVUS-based diagnosis of calcified lesions relies on qualitative analysis and clinical experience, lacking precise data support, which severely impacts the efficiency of intravascular ultrasound diagnosis. Summary of the Invention

[0004] The present invention has been made in view of the above-mentioned problems.

[0005] According to a first aspect of the present invention, a method for processing ultrasound images is provided. The method comprises: acquiring an intravascular ultrasound image; inputting the intravascular ultrasound image into an image segmentation model to obtain a segmented image, wherein the image segmentation model is configured to segment calcified plaques in the intravascular ultrasound image, the image segmentation model comprising an interconnected prediction module and a residual refinement module, the prediction module comprising at least one convolutional block attention module; and determining an area occupied by the calcified plaques based on the calcified plaques in the segmented image.

[0006] Exemplarily, the prediction module includes M-level downsampling blocks connected in series and M-level upsampling blocks connected in series, where M is a positive integer, and downsampling blocks of the same level correspond to upsampling blocks of the same level. Each of the at least one convolution block attention modules is directly connected to a first-level upsampling block or a first-level downsampling block; inputting the intravascular ultrasound image into the image segmentation model to obtain a segmented image includes: downsampling the intravascular ultrasound image in sequence using the M-level downsampling blocks to obtain multiple downsampling results respectively; upsampling using the M-level upsampling blocks based on the multiple downsampling results to obtain a target feature map of the intravascular ultrasound image; inputting the target feature map into the residual refinement module for refinement to obtain the segmented image; wherein each of the convolution block attention modules is used to perform channel attention calculation and / or spatial attention calculation on the downsampling result output by the downsampling block directly connected to the convolution block attention module or the upsampling result output by the upsampling block directly connected to the convolution block attention module.

[0007] Exemplarily, there are at least M convolution block attention modules, and each level of downsampling block is directly connected to a convolution block attention module. The intravascular ultrasound image is sequentially downsampled using the M-level downsampling blocks to obtain multiple downsampling results, including: inputting the intravascular ultrasound image into the first-level downsampling block in the M-level downsampling blocks to obtain the first downsampling feature map of the intravascular ultrasound image, and inputting the first downsampling feature map into the convolution block attention module directly connected to the first-level downsampling block to obtain the first downsampling result; inputting the i-1th downsampling result corresponding to the i-1th level downsampling block in the M-level downsampling blocks into the i-th level downsampling block to obtain the i-th downsampling feature map of the intravascular ultrasound image, and inputting the i-th downsampling feature map into the convolution block attention module directly connected to the i-th level downsampling block to obtain the i-th downsampling result, wherein i is a positive integer, and 1 <i≤M。

[0008] Exemplarily, there are at least M convolution block attention modules. A convolution block attention module is directly connected behind each upsampling block at each level. The upsampling using the M-level upsampling blocks based on the multiple downsampling results to obtain the target feature map of the intravascular ultrasound image includes: determining the M-th upsampled feature map using the M-th upsampling block in the M-level upsampling blocks according to the M-th downsampling result corresponding to the M-th downsampling block in the M-level downsampling blocks; inputting the M-th upsampled feature map into the convolution block attention module directly connected to the M-th upsampling block to obtain the M-th upsampling result; splicing the j-th downsampling result corresponding to the j-th downsampling block in the M-level downsampling blocks and the (j + 1)-th upsampled feature map corresponding to the (j + 1)-th upsampling block in the M-level upsampling blocks, and inputting the splicing result into the j-th upsampling block to obtain the j-th upsampled feature map of the intravascular ultrasound image, and inputting the j-th upsampled feature map into the convolution block attention module directly connected to the j-th upsampling block to obtain the j-th upsampling result, where j is a positive integer and 1 ≤ j < M; the 1st upsampling result is the target feature map of the intravascular ultrasound image.

[0009] Exemplarily, the prediction module further includes an atrous spatial pyramid pooling module, which is connected between the M-th downsampling block in the M-level downsampling blocks and the M-th upsampling block in the M-level upsampling blocks. The upsampling using the M-level upsampling blocks based on the multiple downsampling results to obtain the target feature map of the intravascular ultrasound image includes: inputting the M-th downsampling result corresponding to the M-th downsampling block into the atrous spatial pyramid pooling module for atrous convolution operation, splicing the M-th downsampling result and the downsampling result after the atrous convolution operation to obtain the spliced downsampling result, and inputting the spliced downsampling result into the M-th upsampling block to obtain the M-th upsampling result; according to the M-th upsampling result and the downsampling results other than the M-th downsampling result in the multiple downsampling results, using the (M - 1)-th upsampling block to the 1st upsampling block in the M-level upsampling blocks for upsampling to obtain the target feature map.

[0010] Exemplarily, the prediction module further includes at least one activation function, and each activation function is connected behind the corresponding convolution block attention module, and each activation function is used to perform a non-linear mapping on the corresponding downsampling result or upsampling result.

[0011] Exemplarily, after determining the area occupied by the calcified plaque based on the calcified plaque in the segmented image, the method further includes: for each calcified plaque in the segmented image, determining two tangents of the area occupied by the calcified plaque passing through the center point of the segmented image, and determining the angle between the two tangents to obtain the calcification angle corresponding to the calcified plaque; and determining the total angle of the calcified plaque in the segmented image based on the calcification angles corresponding to all calcified plaques in the segmented image.

[0012] Exemplarily, after determining the total angle of the calcified plaque in the segmented image, the method further includes: determining the calcification level according to a correspondence between the total angle and the calcification level.

[0013] Exemplarily, the method further includes: displaying the segmented image and geometric marks of the calcified plaque in the segmented image, wherein the geometric marks include at least one of the tangent, the calcification angle corresponding to at least one calcified plaque in the segmented image, the image outer contour corresponding to the angle between the two tangents, and the total angle.

[0014] According to a second aspect of the present invention, there is also provided an ultrasonic image processing device, comprising:

[0015] An image acquisition module, used for acquiring intravascular ultrasound images;

[0016] an image segmentation module, configured to input the intravascular ultrasound image into an image segmentation model to obtain a segmented image, wherein the image segmentation model is configured to perform image segmentation on calcified plaques in the intravascular ultrasound image, the image segmentation model comprising an interconnected prediction module and a residual refinement module, the prediction module comprising at least one convolutional block attention module;

[0017] The first determining module is configured to determine an area occupied by the calcified plaque based on the calcified plaque in the segmented image.

[0018] According to a third aspect of the present invention, an electronic device is further provided, comprising: a processor and a memory, wherein the memory stores computer program instructions, and the computer program instructions are used by the processor to execute the above-mentioned ultrasound image processing method when executed.

[0019] According to a fourth aspect of the present invention, a storage medium is further provided, on which program instructions are stored. The program instructions are used to execute the above-mentioned ultrasound image processing method when running.

[0020] According to a fifth aspect of the present invention, a computer program product is further provided, comprising computer program instructions, wherein the computer program instructions are used to execute the above-mentioned ultrasound image processing method when run.

[0021] In the above technical solution, an intravascular ultrasound image is acquired and input into an image segmentation model including a convolutional block attention module to produce a segmented image. Based on the calcified plaques in the segmented image, the area occupied by the calcified plaques is determined. This allows for more automatic and accurate determination of the area occupied by calcified plaques based on the intravascular ultrasound image, thereby providing medical personnel with a more precise picture of the distribution of calcified plaques within the blood vessels.

[0022] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are specifically listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The above and other objects, features, and advantages of the present invention will become more apparent through a more detailed description of the embodiments of the present invention with reference to the accompanying drawings. The accompanying drawings are provided to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and are not intended to limit the present invention. In the drawings, the same reference numerals generally represent the same components or steps.

[0024] Figure 1 A schematic flow chart of an ultrasound image processing method according to an embodiment of the present invention is shown;

[0025] Figure 2 A schematic diagram showing an intravascular ultrasound image and a corresponding segmented image according to an embodiment of the present invention is shown;

[0026] Figure 3 A schematic flow chart of inputting an intravascular ultrasound image into an image segmentation model to obtain a segmented image according to one embodiment of the present invention is shown;

[0027] Figure 4 A schematic diagram of a convolutional block attention module according to one embodiment of the present invention is shown;

[0028] Figure 5 A schematic diagram showing image segmentation of an intravascular ultrasound image using an image segmentation model according to one embodiment of the present invention is shown;

[0029] Figure 6 A schematic flow chart of downsampling to obtain multiple downsampling results according to one embodiment of the present invention is shown;

[0030] Figure 7 A schematic flow chart of upsampling to obtain a target feature map of an intravascular ultrasound image according to one embodiment of the present invention is shown;

[0031] Figure 8 A schematic flowchart of upsampling using an M-level upsampling block to obtain a target feature map of an intravascular ultrasound image according to one embodiment of the present invention is shown;

[0032] Figure 9 A schematic diagram of performing a dilated convolution operation using a dilated spatial convolution pyramid pooling module according to one embodiment of the present invention is shown;

[0033] Figure 10 A schematic diagram of a convolution kernel of a dilated convolution operation according to one embodiment of the present invention is shown;

[0034] Figure 11 A schematic flowchart of determining the total angle of a calcified plaque based on segmenting the calcified plaque in an image according to one embodiment of the present invention is shown;

[0035] Figure 12 shows a schematic diagram of displaying a segmented image according to one embodiment of the present invention;

[0036] Figure 13 shows a schematic block diagram of an ultrasonic image processing apparatus according to an embodiment of the present invention;

[0037] Figure 14 A schematic block diagram of an electronic device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solutions and advantages of the present invention more apparent, exemplary embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments of the present invention, and it should be understood that the present invention is not limited to the exemplary embodiments described herein. Based on the embodiments of the present invention described in the present invention, all other embodiments obtained by those skilled in the art without creative work should fall within the scope of protection of the present invention.

[0039] In order to at least partially solve the above problems, the ultrasound image processing method provided by the present invention uses an image segmentation model including a convolutional block attention module to segment the intravascular ultrasound image, and determines the area occupied by the calcified plaque based on the segmented image.

[0040] Figure 1 FIG. 1 shows a schematic flow chart of an ultrasound image processing method according to an embodiment of the present invention. Figure 1 As shown, the ultrasound image processing method may include steps S1100 to S1300.

[0041] In step S1100 , an intravascular ultrasound image is acquired.

[0042] The intravascular ultrasound image can be any suitable image acquired using an ultrasound device. Exemplarily, the intravascular ultrasound image can be an RGB image or a grayscale image. The intravascular ultrasound image can be a static image or any video frame in a dynamic video. The intravascular ultrasound image can be an image of any suitable size and resolution. The intravascular ultrasound image can be the original image directly acquired by the microprobe of the ultrasound device in the blood vessel, or it can be an image obtained by performing a preprocessing operation on the original image, for example, it can be a callback image. The preprocessing operation can include all operations to improve the visual effect of the intravascular ultrasound image, increase its clarity, or highlight certain features in the image. Exemplarily and non-limitingly, the preprocessing operation can include operations such as digitization, geometric transformation, normalization, and filtering of the original image. The intravascular ultrasound image can also be a synthesized image without affecting subsequent image processing.

[0043] In step S1200, the intravascular ultrasound image is input into an image segmentation model to obtain a segmented image, wherein the image segmentation model is used to perform image segmentation on calcified plaques in the intravascular ultrasound image, and the image segmentation model includes an interconnected prediction module and a residual refinement module, and the prediction module includes at least one convolution block attention module.

[0044] The image segmentation model may be a trained neural network model, which can perform image segmentation on the input intravascular ultrasound image with respect to the calcified plaques therein to generate a segmented image corresponding to the intravascular ultrasound image.

[0045] Image segmentation is the technique and process of dividing an image into several specific regions with unique properties and proposing desired targets. It can be understood that an image segmentation model can perform image segmentation using the region corresponding to the calcified plaque in an intravascular ultrasound image as the segmentation target, and the resulting segmented image highlights the image information of the region corresponding to the calcified plaque. The segmented image can be a binary image. In the segmented image, the pixel values of the region corresponding to the calcified plaque are all equal and equal to a first value; the pixel values of the region outside the calcified plaque are also all equal and equal to a second value, with the first value and the second value being different. Specifically, the segmented image is an image that highlights the image information of the region occupied by the calcified plaque.

[0046] It is understandable that the intravascular ultrasound image may or may not include the area occupied by the calcified plaque. When the intravascular ultrasound image includes the area occupied by the calcified plaque, the area occupied by the calcified plaque can be highlighted in the segmented image after subsequent image segmentation.

[0047] The image segmentation model may include an interconnected prediction module and a residual refinement module. After the intravascular ultrasound image is input into the image segmentation model, the input intravascular ultrasound image may be segmented by the prediction module in the image segmentation model to highlight the area occupied by the calcified plaque, thereby obtaining a preliminary segmented image. The prediction module may include at least one convolutional block attention module (CBAM). In an embodiment of the present application, the intravascular ultrasound image is processed to automatically obtain its calcification level information. With respect to the intravascular ultrasound image, the tissue structure information of the target object such as the human body, such as the inner and outer membranes, the middle membrane and the inner membrane, etc., is not concerned, but only the area occupied by the calcified plaque therein is concerned. The prediction module added with the convolutional block attention module provides spatial attention and channel attention for the image information corresponding to the calcified plaque in the intravascular ultrasound image, so that the image segmentation model can better identify the area occupied by the calcified plaque. This initial segmented image is then fed into the Residual Refinement Module (RRM) within the image segmentation model. This module performs prediction based on the residuals to reduce segmentation errors, ultimately yielding a segmented image corresponding to the intravascular ultrasound image. In other words, the Residual Refinement Module refines the initial segmented image, producing a segmented image with more accurate boundaries and a more complete representation of the internal area occupied by the calcified plaque.

[0048] The basic structure of the prediction module can adopt a neural network structure, such as a neural network structure including a pooling layer and a convolution layer, and the prediction module can include at least one convolutional block attention module. The convolutional block attention module can include a channel attention module and / or a spatial attention module, so that the convolutional block attention module can perform channel attention calculation and / or spatial attention calculation on the feature map of the intravascular ultrasound image. The residual refinement module can adopt a residual refinement module or a residual refinement module with structural adjustment based on the residual refinement module, such as a residual refinement module based on the U-net network structure.

[0049] In step S1300 , based on the segmented calcified plaque in the image, the area occupied by the calcified plaque is determined.

[0050] After obtaining the segmented image, one or more parameters of the calcified plaque, such as its length, width, and area, can be determined. The area occupied by the calcified plaque in the segmented image can then be determined based on these parameters. This, in turn, allows the area occupied by the calcified plaque in the intravascular ultrasound image to be determined. Parameters such as the length, width, and area of the calcified plaque can be determined, for example, based on the pixel coordinates or number of pixels of the calcified plaque in the segmented image.

[0051] In the above technical solution, after acquiring an intravascular ultrasound image, the image is input into an image segmentation model that includes a convolutional block attention module for image segmentation to produce a segmented image. The area occupied by the calcified plaque in the segmented image is then determined based on the calcified plaque. In this technical solution, the image segmentation model with the convolutional block attention module can better identify calcified plaques in intravascular ultrasound images, improving the accuracy of calcified plaque identification, thereby increasing the efficiency of pathological diagnosis for medical personnel and saving them time and effort.

[0052] Figure 2 A schematic diagram of an intravascular ultrasound image and a corresponding segmented image according to an embodiment of the present invention is shown.

[0053] like Figure 2 As shown in the figure, calcified plaques in intravascular ultrasound images vary in size and shape, including dots, strips, or rings. Using only ordinary network models cannot achieve the desired effect. The above ultrasound image processing method can produce a more accurate segmentation image that highlights calcified plaques.

[0054] Exemplarily, the prediction module includes M levels of downsampling blocks connected in series and M levels of upsampling blocks connected in series, where M is a positive integer. Downsampling blocks of the same level correspond to upsampling blocks of the same level. Each of the convolution block attention modules is directly connected to the back of a level upsampling block or a level downsampling block. The back is in the direction of data flow, that is, for each convolution block attention module, the output data of the upsampling block or downsampling block to which it is connected is input into the convolution block attention module. Each downsampling block or upsampling block can be composed of one or more convolution layers or convolution modules, respectively, and the types of multiple convolution layers or convolution modules can be different, such as a convolution module (ResBlock) containing a jump connection, a conventional convolution layer (ConvBlock), etc. The number of levels of downsampling blocks and upsampling blocks in the prediction module is the same and corresponds one to one. Figure 3 FIG. 1 shows a schematic flow chart of inputting an intravascular ultrasound image into an image segmentation model to obtain a segmented image according to an embodiment of the present invention. Figure 3 As shown, step S1200 may include steps S1210 to S1230.

[0055] In step S1210 , the intravascular ultrasound image is downsampled sequentially using M-level downsampling blocks to obtain a plurality of downsampling results.

[0056] For example, the intravascular ultrasound image can be sequentially downsampled using M-level downsampling blocks to obtain M downsampling results of directly downsampling the intravascular ultrasound image. That is, each downsampling result is a downsampling result obtained using a single-level downsampling block. Alternatively, the intravascular ultrasound image can be input into a first-level downsampling block to obtain a first downsampling result, and then the first downsampling result can be input into a second-level downsampling block to obtain a second downsampling result. The downsampling result obtained from the i-th level downsampling block can be input into an i+1-th level downsampling block to obtain a downsampling result from an i+1-th level downsampling block, and so on, until a downsampling result outputted by an M-th level downsampling block is obtained. That is, each downsampling result is a downsampling result obtained using a single-level or multi-level downsampling block. The downsampling can be based on interpolation.

[0057] The downsampling result may be a feature map. Depending on the configuration of the downsampling blocks, the formats of the feature maps output by each level of downsampling blocks may be the same or different. Each level of downsampling blocks may be a downsampling block preset before performing the above-mentioned ultrasound image processing method.

[0058] In step S1220 , upsampling is performed using an M-level upsampling block according to the multiple downsampling results to obtain a target feature map of the intravascular ultrasound image.

[0059] An M-level upsampling block can be used to determine multiple upsampling results based on the multiple downsampling results. This process is similar to the process in step S1210 described above and will not be further described here for the sake of brevity. A target feature map for the intravascular ultrasound image can then be determined based on the multiple upsampling results. For example, the upsampling result obtained by the first-level upsampling block can be used as the target feature map.

[0060] In step S1230, the target feature map is input into the residual refinement module for refinement to obtain a segmented image.

[0061] Refinement refers to the process in which the residual refinement module processes the target feature map according to parameters such as the relationship between pixels in the target feature map and the boundary weight of the area occupied by the calcified plaque, so as to make the area occupied by the calcified plaque more accurate.

[0062] It can be understood that since each of the above-mentioned convolution block attention modules is directly connected to the back of a first-level upsampling block or a first-level downsampling block, after obtaining the downsampling result through the downsampling block of that level or obtaining the upsampling result through the upsampling block of that level, the downsampling result or the upsampling result can be input into the convolution block attention module directly connected to the downsampling block or upsampling block of that level for further processing, thereby obtaining the processed downsampling result or the processed upsampling result, and then using the processed downsampling result or the processed upsampling result as the final downsampling result or upsampling result obtained by the downsampling block or upsampling of that level. Specifically, each of the above-mentioned convolution block attention modules is used to perform channel attention calculation and / or spatial attention calculation on the downsampling result output by the downsampling block directly connected to the convolution block attention module or the upsampling result output by the upsampling block directly connected to the convolution block attention module.

[0063] Figure 4 A schematic diagram of a convolutional block attention module according to an embodiment of the present invention is shown.

[0064] like Figure 4 As shown, the convolutional block attention module can perform channel attention calculation and spatial attention calculation. Alternatively, the convolutional block attention module can also perform only channel attention calculation or spatial attention calculation. In addition, the convolutional block attention module can also perform attention calculation based on other image factors.

[0065] Figure 4 The calculation process of the convolutional block attention module is as follows. First, channel attention is calculated Ac. As shown in Formula 1, the feature matrix F (H*W*C, where H and W represent the height and width of the matrix, respectively, and C represents the number of channels) corresponding to the downsampled feature map obtained in the downsampling step or the upsampled feature map obtained in the upsampling step can be processed with maximum pooling MaxPool and average pooling AvgPool, respectively. A sigmoid function is then applied to obtain the channel attention-calculated feature matrix Ac(F), which can be of size 1*1*C. Then, as shown in Formula 2, the feature matrix F is first element-wise multiplied by the channel attention-calculated feature matrix Ac(F) to obtain F′. Spatial attention calculation As can be performed on F′. As shown in Formula 3, AvgPool and MaxPool are applied, followed by a 7*7 convolution operation. Finally, a sigmoid function is applied to obtain the spatial attention-calculated feature matrix As, which can be of size H*W*1. Finally, as shown in Formula 4, As(F′) is multiplied element-by-element by F′ to obtain the final module output F". F" is the downsampling result or upsampling result corresponding to the input downsampling feature map or upsampling feature map, respectively.

[0066] Formula 1:

[0067] Formula 2:

[0068] Formula 3:

[0069] Formula 4:

[0070] It can be understood that each level of upsampling block and each level of downsampling block can be connected to a convolution block attention module. Alternatively, each of the partial upsampling blocks and / or partial downsampling blocks can be connected to a convolution block attention module to further process the sampling results of the connected upsampling blocks or downsampling blocks. Exemplarily, for the case where the convolution block attention module is directly connected to the back of the downsampling block and the directly connected downsampling block is not the M-th level upsampling block, the downsampling result processed by the convolution block attention module can be further input to the next level downsampling block to continue the downsampling operation. For the case where the convolution block attention module is directly connected to the back of the upsampling module and the directly connected upsampling block is not the 1st level upsampling block, the upsampling result processed by the convolution block attention module can be further input to the next level upsampling block to continue the upsampling operation.

[0071] It can be understood that for a downsampling block or upsampling block that is not directly connected to a convolution block attention module, the corresponding downsampling result or upsampling result is the sampling result directly output by it. For a downsampling block or upsampling block that is directly connected to a convolution block attention module, the downsampling result or upsampling result that is further processed by the convolution block attention module can be called the sampling result corresponding to the downsampling block or upsampling block.

[0072] In the above technical solution, the target feature map of the intravascular ultrasound image is determined by a prediction module composed of a multi-level downsampling block, an upsampling block corresponding to each level of the multi-level downsampling block, and a convolution block attention module. The target feature map is then refined by a residual refinement module, so that the area occupied by the calcified plaque in the segmented image is more accurate.

[0073] Figure 5 FIG. 1 is a schematic diagram showing image segmentation of an intravascular ultrasound image using an image segmentation model according to an embodiment of the present invention. Figure 5 As shown, the image segmentation model may include an interconnected prediction module and a residual refinement module. The prediction module includes M-level downsampling blocks connected in series and M-level upsampling blocks connected in series. Figure 5 In the illustrated embodiment, M = 4. Downsampling blocks at the same level correspond to upsampling blocks at the same level.

[0074] Exemplarily, there are at least M convolutional block attention modules, and each level of downsampling block is directly connected to a convolutional block attention module. That is, when the downsampling block is of M levels, there are at least M convolutional block attention modules. Among these convolutional block attention modules, M convolutional block attention modules correspond one-to-one to the M levels of downsampling blocks, and each of the remaining convolutional block attention modules corresponds to a different level of upsampling blocks. Figure 5 ,After each downsampling block, there is a convolutional block attention module (CBAM).

[0075] Figure 6 FIG. 4 shows a schematic flow chart of downsampling to obtain multiple downsampling results according to an embodiment of the present invention. Figure 6 As shown, step S1210 may include step S1211 and step S1212.

[0076] In step S1211, the intravascular ultrasound image is input into the first-level downsampling block in the M-level downsampling blocks to obtain a first downsampling feature map of the intravascular ultrasound image, and the first downsampling feature map is input into the convolution block attention module directly connected to the first-level downsampling block to obtain a first downsampling result.

[0077] Reference again Figure 5 The intravascular ultrasound image is input into the first-level downsampling block. The intravascular ultrasound image can be a 256*256*1 size intravascular ultrasound image. The first downsampling feature map output by the first-level downsampling block is a feature map that has not undergone channel attention calculation and / or spatial attention calculation by the convolution block attention module. After the convolution block attention module performs channel attention calculation and / or spatial attention calculation on the first downsampling feature map, it can output a first downsampling result, which can be a 256*256*64 size feature map.

[0078] The first-level downsampling block may include a convolution module (Conv) and a basic residual block (Basic resblock). The convolution module (Conv) may include multiple convolution submodules, multiple batch normalization submodules, and an activation function module. The basic residual block may include multiple convolution submodules, multiple batch normalization submodules, and an activation function module. The convolution submodule may perform a convolution operation with a convolution kernel of 1*1 or 3*3.

[0079] In step S1212, the i-1th downsampling result corresponding to the i-1th level downsampling block in the M-level downsampling blocks is input into the i-th level downsampling block to obtain the i-th downsampling feature map of the intravascular ultrasound image, and the i-th downsampling feature map is input into the convolution block attention module directly connected to the i-th level downsampling block to obtain the i-th downsampling result, where i is a positive integer and 1 <i≤M。

[0080] The i-1th level upsampling block is the previous level upsampling block of the i-th level upsampling block.

[0081] The i-1th downsampling result corresponding to the i-1th level downsampling block in the M-level downsampling blocks is input into the i-th level downsampling. That is, for the i-th level downsampling block, the i-1th downsampling result obtained by the convolution block attention module directly connected to the previous level downsampling block of the i-th level is used as its input for downsampling. For example, for the second level downsampling block, the first downsampling result obtained by the convolution block attention module directly connected to the first level downsampling block is used as the input of the second level downsampling block and the downsampling operation is performed on the first downsampling result.

[0082] It can be understood that for a certain level of downsampling blocks other than the first level of downsampling blocks, after the downsampling result obtained by the convolution block attention module directly connected to the previous level of downsampling blocks is input into the downsampling block of the level and the downsampling operation is performed by the downsampling block of the level, the downsampling feature map corresponding to the downsampling block of the level can be obtained. For example, the i-th level downsampling block corresponds to the i-th downsampling feature map. The downsampling feature map corresponding to the downsampling block of the level is then subjected to channel attention calculation and / or spatial attention calculation by the convolution block attention module directly connected to the downsampling block of the level, and the downsampling result corresponding to the downsampling block of the level is obtained, for example, the i-th level downsampling block corresponds to the i-th downsampling result.

[0083] It can be understood that the total number of the first down-sampling result and the i-th down-sampling result is multiple, so the first down-sampling result and all the i-th down-sampling results together constitute the multiple down-sampling results obtained in the above step S1210.

[0084] Reference again Figure 5 The first downsampling result is input into the second-level downsampling block, and the second downsampling result is obtained through the second-level downsampling block and the convolution block attention module directly connected to the second-level downsampling block. In the case where the first downsampling result can be a feature map of size 256*256*64, the second downsampling result can be a feature map of size 128*128*128. It is understandable that the second downsampling result can be input into the third-level downsampling block, and the third downsampling result can be obtained through the third-level downsampling block and the convolution block attention module directly connected to the third-level downsampling block. The third downsampling result can be a feature map of size 64*64*256. It is understandable that the third downsampling result can be input into the fourth-level downsampling block, and the fourth downsampling result can be obtained through the fourth-level downsampling block and the convolution block attention module directly connected to the fourth-level downsampling block. The fourth downsampling result can be a feature map of size 32*32*512. At this time, four downsampling results can be obtained, namely the first downsampling result, the second downsampling result, the third downsampling result, and the fourth downsampling result.

[0085] Similar to the first-level downsampling block, the second-level downsampling block, the third-level downsampling block, and the fourth-level downsampling block can each include a basic res block and a basic res block with downsampling. The basic res block with downsampling can include multiple convolution submodules, multiple batch normalization submodules, and an activation function module. A downsampling block can include one or more basic res blocks, and the number of basic res blocks in downsampling blocks of different levels can be the same or different.

[0086] In the above technical solution, by directly connecting a convolution block attention module behind each level of downsampling block, and performing image processing on the input intravascular ultrasound image using a multi-level downsampling block and a convolution block attention module directly connected to the multi-level downsampling block to obtain multiple downsampling results, the accuracy of the area occupied by calcified plaques in the segmented image output by the image segmentation model can be improved.

[0087] Exemplarily, there are at least M convolutional block attention modules, and each level of upsampling blocks is directly connected to a convolutional block attention module. That is, when there are at least M levels of upsampling blocks, there are at least M convolutional block attention modules, and the M convolutional block attention modules among these convolutional block attention modules correspond one-to-one to the M levels of upsampling blocks. Figure 7 FIG. 5 shows a schematic flow chart of upsampling to obtain a target feature map of an intravascular ultrasound image according to an embodiment of the present invention. Figure 7 As shown, step S1220 may include steps S1221 to S1223.

[0088] In step S1221, according to the Mth downsampling result corresponding to the Mth downsampling block in the M-level downsampling blocks, the Mth upsampling feature map is determined using the Mth upsampling block in the M-level upsampling blocks.

[0089] Exemplarily, the Mth downsampling result can be directly used as the input of the Mth upsampling block. It can be understood that the Mth downsampling result can be calculated by or without the convolution block attention module.

[0090] Alternatively, the Mth downsampling result may be processed through a preset connection module to obtain a processed Mth downsampling result, the processed Mth downsampling result and the unprocessed Mth downsampling result are spliced together, and the spliced result is used as the input of the Mth level upsampling block. The processing performed by a preset connection module may include conventional processing operations such as convolution and pooling, which are not limited here. Figure 5 When M=4, the 4th downsampling result directly output by the 4th downsampling block can be input to the connection module (in Figure 5 shown as ASSPP), to obtain the fourth downsampling result output by the connection module. The fourth downsampling result output by the connection module can be a feature map with a size of 32*32*512. Then, the fourth downsampling result and the fourth downsampling result output by the connection module are concatenated in terms of feature maps, so as to obtain the concatenated fourth result as the input of the fourth-level upsampling block. In Figure 5 the illustrated embodiment, this concatenation operation is performed by a concatenation module (shown as Concatenate in Figure 5 ). Alternatively, this concatenation operation can also be automatically completed by the upsampling block in the prediction module. The concatenated fourth downsampling result is input into the fourth-level upsampling block to determine the fourth upsampled feature map.

[0091] In step S1222, the M-th upsampled feature map is input into a convolutional block attention module directly connected to the M-th level upsampling block to obtain the M-th upsampling result.

[0092] Referring again to Figure 5 , the fourth upsampled feature map output by the fourth-level upsampling block is a feature map that has not undergone channel attention calculation and / or spatial attention calculation by the convolutional block attention module. After the convolutional block attention module performs channel attention calculation and / or spatial attention calculation on the fourth upsampled feature map, the fourth upsampling result can be obtained. The fourth upsampling result can be a feature map with a size of 64*64*256.

[0093] Exemplarily, the fourth-level upsampling block can include modules for convolutional operation, batch normalization operation, and activation function.

[0094] In step S1223, the j-th downsampling result corresponding to the j-th downsampling block in the M-level downsampling block and the (j + 1)-th upsampling result corresponding to the (j + 1)-th upsampling block in the M-level upsampling block are concatenated, and the concatenated result is input into the j-th upsampling block to obtain the j-th upsampled feature map of the intravascular ultrasound image, and the j-th upsampled feature map is input into a convolutional block attention module directly connected to the j-th upsampling block to obtain the j-th upsampling result, where j is a positive integer and 1 ≤ j < M. The first upsampling result is the target feature map of the intravascular ultrasound image.

[0095] The (j + 1)-th level upsampling block is the upper-level upsampling block of the j-th level upsampling block.

[0096] The jth downsampling result corresponding to the jth level downsampling block in the M-level downsampling blocks and the j+1th upsampling result corresponding to the j+1th level upsampling block in the M-level upsampling blocks are concatenated. Common feature map concatenation methods can be used for this concatenation. The concatenated result can be input into the j-level upsampling block. That is, for the j-level upsampling block, its input is the concatenated feature map obtained by concatenating the j+1th upsampling result obtained by the attention module of the convolutional block directly connected to the previous upsampling block and the jth downsampling result corresponding to the j-level downsampling block in the M-level downsampling blocks.

[0097] It can be understood that for upsampling blocks other than the M-th level upsampling block, the concatenation result of the upsampling result obtained by the convolution block attention module directly connected to the previous level upsampling block and the downsampling result corresponding to the same level downsampling block is input into the upsampling block, and the upsampling feature map corresponding to the upsampling block can be obtained, for example, the j-th upsampling feature map corresponding to the j-th level upsampling block. The upsampling feature map corresponding to the upsampling block is then subjected to channel attention calculation and / or spatial attention calculation by the convolution block attention module directly connected to the upsampling block to obtain the upsampling result corresponding to the upsampling block, for example, the j-th upsampling result corresponding to the j-th level upsampling block.

[0098] Reference again Figure 5 The fourth upsampling result and the third downsampling result are concatenated, and the concatenated fourth result is used as the input to the third upsampling block. The fourth concatenated result is input to the third upsampling block, and the third upsampling result is obtained through the third upsampling block and the convolution block attention module directly connected to the third upsampling block. The third upsampling result can be a 128*128*128 feature map. The third upsampling result and the second downsampling result are concatenated, and the third concatenated result is used as the input to the second upsampling block. The third concatenated result is input to the second upsampling block, and the second upsampling result is obtained through the second upsampling block and the convolution block attention module directly connected to the second upsampling block. The second upsampling result can be a 256*256*64 feature map. The second upsampling result and the first downsampling result are concatenated, and the second concatenated result is used as the input to the first upsampling block. The second concatenation result is input into the first-level upsampling block, and the first upsampling result is obtained through the second-level upsampling block and the convolution block attention module directly connected to the first-level upsampling block. This first upsampling result can be a feature map of size 256*256*64. As can be understood, four upsampling results can be obtained at this time.

[0099] The first upsampling result can be used as the target feature map of the intravascular ultrasound image. This is the upsampling result obtained by the attention module of the convolutional block directly connected to the first-level upsampling block. The first-level upsampling block is the last upsampling block in the M-level upsampling block. As can be understood, all upsampling results are feature maps, and the first upsampling result is the feature map that has undergone the most upsampling operations. It can be used as the target feature map of the intravascular ultrasound image.

[0100] The aforementioned second-level upsampling block, third-level upsampling block, and fourth-level upsampling block may each include modules for convolution operations, batch normalization operations, activation functions, and upsampling operations. A processing unit may be composed of a convolution module, a batch normalization module, and an activation function. Each level of upsampling block may include one or more such processing units. The number of processing units in upsampling blocks of different levels may be the same or different.

[0101] In this technical solution, a convolutional attention module is directly connected to each upsampling block. The downsampling results obtained from the downsampling operation are then upsampled using the multi-level upsampling blocks and the convolutional attention modules directly connected to them to generate the target feature map. This can further improve the accuracy of the area occupied by calcified plaques in the segmented image output by the image segmentation model.

[0102] Exemplarily, the prediction module further includes a dilated spatial convolutional pyramid pooling module (ASSPP). The dilated spatial convolutional pyramid pooling module is connected between the M-th level downsampling block in the M-level downsampling blocks and the M-th level upsampling block in the M-level upsampling blocks. Figure 5 A dilated spatial convolutional pyramid pooling module is connected between the M-level downsampling block and the M-level upsampling block. This module can enhance the image segmentation model's ability to recognize calcified plaques of different sizes.

[0103] Figure 8 A schematic flowchart of upsampling using an M-level upsampling block to obtain a target feature map of an intravascular ultrasound image according to an embodiment of the present invention is shown.

[0104] like Figure 8 As shown, the above step 1220 may include step S1225 and step 1226.

[0105] In step S1225, the Mth downsampling result corresponding to the Mth level downsampling block is input into the dilated spatial convolution pyramid pooling module for a dilated convolution operation, the Mth downsampling result and the downsampling result after the dilated convolution operation are spliced to obtain a spliced downsampling result, and the spliced downsampling result is input into the Mth level upsampling block to obtain an Mth upsampling result.

[0106] In step 1226, based on the M-th upsampling result and the downsampling results other than the M-th downsampling result in the multiple downsampling results, upsampling is performed using the M-1-th level upsampling block to the 1-th level upsampling block in the M-level upsampling blocks to obtain a target feature map.

[0107] When the Mth-level downsampling block is not directly connected to the convolution block attention module, the Mth downsampling result is the Mth downsampling feature map directly determined by the Mth-1th downsampling block. When the Mth-level downsampling block is directly connected to the convolution block attention module, the Mth downsampling result is the Mth downsampling result determined by performing channel attention calculation and / or spatial attention calculation on the Mth downsampling feature map by the convolution block attention module directly connected to the Mth downsampling block.

[0108] Reference again Figure 5 A dilated spatial convolution pyramid pooling module is connected between the 4th level downsampling block and the 4th level upsampling block. The dilated spatial convolution pyramid pooling module is used to perform a dilated convolution operation on the 4th downsampling result. The 4th downsampling result output by the attention module of the convolution block directly connected to the 4th level upsampling block and the downsampling result after the dilated convolution operation can be spliced to obtain a spliced downsampling result, and the spliced downsampling result is input into the 4th level upsampling block to obtain a 4th upsampling result. Then, the above-mentioned step S1223 can be performed. For the sake of brevity, it will not be repeated here.

[0109] Understandably, Figure 5 In the illustrated embodiment, each level of upsampling blocks is directly connected to a convolution block attention module, and a dilated spatial convolution pyramid pooling module is connected between the 4th level downsampling block and the 4th level upsampling block. Figure 5 The illustrated embodiments are for illustration only and do not constitute a limitation of the present application. The atrous spatial convolution pyramid pooling module may also be used in embodiments where all or at least one level of upsampling blocks are not directly connected to a convolution block attention module. When the j-th level upsampling block is not directly connected to a convolution block attention module, the j-th upsampling feature map is used as the j-th upsampling result. When the j-th level upsampling block has a directly connected convolution block attention module, the j-th upsampling feature map is input into the convolution block attention module to obtain the j-th upsampling result.

[0110] In the above technical solution, the atrous spatial convolution pyramid pooling module is connected between the M-th downsampling block in the M-level downsampling block and the M-th upsampling block in the M-level upsampling block, and by performing a atrous convolution operation on the M-th downsampling result, the image segmentation model's ability to recognize calcified plaques can be improved, making the area occupied by calcified plaques in the output segmented image more accurate.

[0111] Figure 9A schematic diagram of performing a dilated convolution operation using a dilated spatial convolution pyramid pooling module according to an embodiment of the present invention is shown.

[0112] like Figure 9 As shown in Figure 2, when the dilated spatial convolution pyramid pooling module performs dilated convolution operations on the downsampling results corresponding to the last level downsampling block, it first performs four parallel dilated convolution operations of different scales on the downsampling result F. Figure 9 Here, ①②③④ represent the convolution operation with a 1*1 kernel and a dilation rate of 1 (i.e., Conv2D(1,1) dilation rate = 1), the convolution operation with a 3*3 kernel and a dilation rate of 6 (i.e., Conv2D(3,3) dilation rate = 6), the convolution operation with a 3*3 kernel and a dilation rate of 12 (i.e., Conv2D(3,3) dilation rate = 12), and the convolution operation with a 3*3 kernel and a dilation rate of 18 (i.e., Conv2D(3,3) dilation rate = 18), respectively. Conv2D(1,1) indicates that the convolution kernel is 1*1, and Conv2D(3,3) indicates that the convolution kernel is 3*3. The dilation rate represents the dilation rate of the convolution kernel. Figure 10 Figure 2 shows convolution kernels with different dilation rates according to one embodiment of the present invention. Next, the newly obtained feature maps are fused to obtain a feature map F′ of size 32*32*512. Finally, convolution, activation function, and batch normalization are performed to obtain the final feature map F″.

[0113] It is understood that the image segmentation model of the embodiment of the present application can be constructed based on the BasNet network. During construction, the original connection layer of the BasNet model, as well as the two upsampling blocks and two downsampling blocks close to the connection layer, can be removed first. That is, after the intravascular ultrasound image is downsampled, before it reaches the dilated spatial convolutional pyramid pooling module, the size of the downsampled result is 32*32*512, to avoid insufficient information due to the downsampling result being too small. Then, the above-mentioned dilated spatial convolutional pyramid pooling module is added between the last level downsampling block and the upsampling block.

[0114] Exemplarily, the prediction module further includes at least one activation function, each activation function being connected to the back of the corresponding convolution block attention module. Each activation function is used to perform nonlinear mapping on the downsampling result corresponding to the downsampling block or the upsampling result corresponding to the upsampling block. The downsampling result corresponding to the downsampling block may include the downsampling result corresponding to the downsampling block connected to the convolution block attention module with or without channel attention calculation and / or position attention calculation, and the upsampling result corresponding to the upsampling block includes the upsampling result corresponding to the upsampling block connected to the convolution block attention module with or without channel attention calculation and / or position attention calculation.

[0115] The activation function is used to make the characteristics of the up-sampling result and the down-sampling result consistent, that is, to maintain the nonlinear characteristics of the two. The activation function can use any commonly used activation function if it meets the requirements, and is not limited here.

[0116] In the above technical solution, by connecting the activation function behind the convolution block attention module, the characteristics of the upsampling result and the downsampling result are made consistent, thereby reducing the errors caused by upsampling, downsampling, channel attention calculation and / or position attention, and improving the accuracy of the area occupied by calcified plaques in the segmented image output by the image segmentation model.

[0117] Figure 11 FIG. 4 is a schematic flow chart showing a method for determining the total angle of a calcified plaque based on segmenting the calcified plaque in an image according to an embodiment of the present invention. Figure 11 As shown, step S1300 may include steps S1310 to S1320.

[0118] In step S1310, for each calcified plaque in the segmented image, two tangent lines of the area occupied by the calcified plaque passing through the center point of the segmented image are determined, and the angle between the two tangent lines is determined to obtain the calcification angle corresponding to the calcified plaque.

[0119] When multiple calcified plaques exist in the segmented image, multiple calcification angles can be determined for each calcified plaque. Each calcification angle corresponds to a calcified plaque. Two tangent lines to the area occupied by the calcified plaque, passing through the center point of the segmented image, can be determined by connecting pixels on the edge of the calcified plaque with the center point of the segmented image. It is understood that when determining the angle between the two tangent lines, both the angle information and the angle position information can be determined.

[0120] In step S1320, the total angle of the calcified plaques in the segmented image is determined according to the calcification angles corresponding to all the calcified plaques in the segmented image.

[0121] The sum of the calcification angles corresponding to all calcified plaques can be used as the total angle of the calcified plaque in the segmented image. Alternatively, the total angle of the calcified plaque in the segmented image can be determined by using other calculation methods, such as integration or multiplication, based on the calcification angles corresponding to all calcified plaques. This total angle of the calcified plaque can be used to determine other information related to the calcified plaque, such as calcification grade assessment.

[0122] In the above technical solution, the angle of each calcified plaque is determined based on two tangent lines to the area occupied by each calcified plaque at the center point of the over-segmented image, thereby determining the total angle of the calcified plaque. This allows for more accurate determination of the total angle of the calcified plaque, which helps improve the accuracy of subsequent determination of information related to the calcified plaque, thereby enhancing the efficiency of diagnosis by medical personnel.

[0123] Illustratively, after determining the total angle of the calcified plaque in the segmented image, the above step S1300 may further include step S1330: determining the calcification level according to the correspondence between the total angle and the calcification level.

[0124] Different calcification levels may correspond to different angle ranges, and the calcification level may be directly determined based on the angle range in which the total angle falls. The correspondence between the total angle and the calcification level may be a preset relationship or may be manually set during ultrasound image processing.

[0125] For example, the intravascular ultrasound image can be divided into different calcification levels based on the correspondence between the calcified plaque angle and the calcification level shown in Table 1.

[0126] Table 1

[0127] The total angle A of the calcified plaque Calcification grade A≤90° I 90°<A≤180° II 180°<A≤270° III 270°<A≤360° IV

[0128] As shown in Table 1, when the total angle of the calcified plaque is in different ranges, it corresponds to different calcification grades.

[0129] In the above technical solution, the calcification level is determined based on the correspondence between the total angle and the calcification level, which can more accurately determine the calcification level and save medical staff time in determining the calcification level.

[0130] Exemplarily, the above-mentioned ultrasound image processing method may further include step S1340: displaying the segmented image and geometric marks of the calcified plaques in the segmented image, wherein the geometric marks include at least one of a tangent, a calcification angle corresponding to at least one calcified plaque in the segmented image, an image outer contour corresponding to the angle between two tangents, and a total angle.

[0131] When displaying the segmented image and the geometric markers of the calcified plaques in the segmented image, the segmented image and the geometric markers can be rendered. For example, lines, symbols, or areas of different colors or sizes can be used to represent the above-mentioned geometric markers, including tangents, the calcification angle corresponding to at least one calcified plaque in the segmented image, the image outer contour corresponding to the angle between two tangents, and the total angle, to better distinguish them. It is also possible to display a partial area in the segmented image without affecting the determination of the calcification level. The rendering method can be automatically determined by the system or manually adjusted.

[0132] In the above technical solution, by displaying the segmented image and geometric marks, medical staff can more easily understand the precise data of calcified plaques.

[0133] Figure 12 FIG. 4 is a schematic diagram showing a segmented image according to an embodiment of the present invention.

[0134] like Figure 12 As shown, the segmented calcified plaque, the calcification angle, the image outer contour corresponding to the angle between the two tangents, and the ray from the image center to the calcified edge (i.e., the two tangents of the calcified plaque) can be displayed on the display terminal by rendering. The rendering method can be any method that highlights the calcified plaque, the calcification angle, the outer arc of the calcified area, and the ray from the image center to the calcified edge (i.e., the two tangents of the calcified plaque), for example Figure 12 In the image, the outer arc of the calcification area is thickened for display.

[0135] Figure 13 FIG. 1 shows a schematic block diagram of an ultrasonic image processing apparatus according to an embodiment of the present invention. Figure 13 As shown, the ultrasonic image processing device includes an image acquisition module, an image segmentation module, and a first evaluation module.

[0136] The image acquisition module is used to acquire intravascular ultrasound images.

[0137] The image segmentation module is used to input the intravascular ultrasound image into the image segmentation model to obtain a segmented image, wherein the image segmentation model is used to perform image segmentation on calcified plaques in the intravascular ultrasound image, and the image segmentation model includes an interconnected prediction module and a residual refinement module, and the prediction module includes at least one convolution block attention module.

[0138] The first determination module is used to determine the area occupied by the calcified plaque based on the segmented calcified plaque in the image.

[0139] Exemplarily, the prediction module includes M-level downsampling blocks and M-level upsampling blocks connected in series, where M is a positive integer. Downsampling blocks of the same level correspond to upsampling blocks of the same level. Each of the at least one convolutional block attention modules is directly connected to a first-level upsampling block or a first-level downsampling block. The image segmentation module includes a first downsampling submodule, a first upsampling submodule, and an image refinement submodule. The first downsampling submodule is configured to sequentially downsample the intravascular ultrasound image using the M-level downsampling blocks to obtain multiple downsampling results. The first upsampling submodule is configured to upsample the multiple downsampling results using the M-level upsampling blocks to obtain a target feature map of the intravascular ultrasound image. The image refinement submodule is configured to input the target feature map into a residual refinement module for refinement to obtain a segmented image. Each convolutional block attention module is configured to perform channel attention and / or spatial attention calculations on the downsampling result output by the downsampling block to which it is directly connected, or the upsampling result output by the upsampling block to which it is directly connected.

[0140] Exemplarily, there are at least M convolution block attention modules, and each level of downsampling block is directly connected to a convolution block attention module. The first downsampling submodule includes a first downsampling result determination submodule and a second downsampling result determination submodule. The first downsampling result determination submodule is used to input the intravascular ultrasound image into the first level downsampling block in the M level downsampling blocks to obtain the first downsampling feature map of the intravascular ultrasound image, and input the first downsampling feature map into the convolution block attention module directly connected to the first level downsampling block to obtain the first downsampling result. The second downsampling result determination submodule is used to input the i-1th downsampling result corresponding to the i-1th level downsampling block in the M level downsampling blocks into the i-th level downsampling block to obtain the i-th downsampling feature map of the intravascular ultrasound image, and input the i-th downsampling feature map into the convolution block attention module directly connected to the i-th level downsampling block to obtain the i-th downsampling result, where i is a positive integer, and 1 <i≤M。

[0141] Exemplarily, there are at least M convolutional block attention modules. A convolutional block attention module is directly connected behind each upsampling block at each level. The first upsampling sub-module includes a first upsampled feature map determination sub-module, a first upsampling result determination sub-module, and a second upsampling result determination sub-module. The first upsampled feature map determination sub-module is configured to use the Mth upsampling block in the M-level upsampling block to determine the Mth upsampled feature map according to the Mth downsampling result corresponding to the Mth downsampling block in the M-level downsampling block. The first upsampling result determination sub-module is configured to input the Mth upsampled feature map into the convolutional block attention module directly connected to the Mth upsampling block to obtain the Mth upsampling result. The second upsampling result determination sub-module is configured to splice the jth downsampling result corresponding to the jth downsampling block in the M-level downsampling block and the (j + 1)th upsampling result corresponding to the (j + 1)th upsampling block in the M-level upsampling block, and input the splicing result into the jth upsampling block to obtain the jth upsampled feature map of the intravascular ultrasound image, and input the jth upsampled feature map into the convolutional block attention module directly connected to the jth upsampling block to obtain the jth upsampling result, where j is a positive integer and 1 ≤ j < M, and the first upsampling result is the target feature map of the intravascular ultrasound image.

[0142] Exemplarily, the prediction module further includes an atrous spatial pyramid pooling module. The atrous spatial pyramid pooling module is connected between the Mth downsampling block in the M-level downsampling block and the Mth upsampling block in the M-level upsampling block. The first upsampling sub-module includes a second upsampling result determination module and a second target feature map determination module. The second upsampling result determination module is configured to input the Mth downsampling result corresponding to the Mth downsampling block into the atrous spatial pyramid pooling module for atrous convolution operation, splice the Mth downsampling result and the downsampling result after the atrous convolution operation to obtain the spliced downsampling result, and input the spliced downsampling result into the Mth upsampling block to obtain the Mth upsampling result. The second target feature map determination module is configured to perform upsampling using the (M - 1)th upsampling block to the 1st upsampling block in the M-level upsampling block according to the Mth upsampling result and the downsampling results other than the Mth downsampling result among the multiple downsampling results to obtain the target feature map.

[0143] Exemplarily, the prediction module further includes at least one activation function. Each activation function is connected behind the corresponding convolutional block attention module. Each activation function is configured to perform a non-linear mapping on the corresponding downsampling result or upsampling result.

[0144] Exemplarily, the ultrasound image processing device further includes a second determination module, which includes an angle determination submodule and a total angle determination submodule. After determining the area occupied by the calcified plaques in the segmented image, the angle determination submodule is configured to, for each calcified plaque in the segmented image, determine two tangent lines to the area occupied by the calcified plaque that pass through the center point of the segmented image, and determine the angle between the two tangent lines to obtain the calcification angle corresponding to the calcified plaque. The total angle determination submodule is configured to determine the total angle of the calcified plaques in the segmented image based on the calcification angles corresponding to all calcified plaques in the segmented image.

[0145] Exemplarily, the second determination module further includes a calcification level determination submodule. After determining the total angle of the calcified plaque in the segmented image, the calcification level determination submodule is configured to determine the calcification level according to the corresponding relationship between the total angle and the calcification level.

[0146] Exemplarily, the ultrasound image processing apparatus further includes a display module. The display module is configured to display the segmented image and geometric markers of the calcified plaques in the segmented image, wherein the geometric markers include at least one of a tangent, a calcification angle corresponding to at least one calcified plaque in the segmented image, an image contour corresponding to an angle between two tangents, and a total angle.

[0147] According to another aspect of the present invention, an electronic device is provided. Figure 14 FIG. 1 shows a schematic block diagram of an electronic device according to an embodiment of the present invention. Figure 14 As shown, the electronic device includes a processor and a memory, wherein the memory stores computer program instructions, and the computer program instructions are used by the processor to execute the ultrasound image processing method as described above when the processor is running.

[0148] In addition, according to another aspect of the present invention, a storage medium is provided, on which program instructions are stored. When the program instructions are executed by a computer or processor, the computer or processor performs the corresponding steps of the above-mentioned ultrasound image processing method according to the embodiment of the present invention, and is used to implement the corresponding modules in the above-mentioned ultrasound image processing device according to the embodiment of the present invention or the corresponding modules used in the above-mentioned ultrasound image processing device. The storage medium may include, for example, a memory card of a smartphone, a storage component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), a USB memory, or any combination of the above-mentioned storage media. The computer-readable storage medium may be any combination of one or more computer-readable storage media.

[0149] According to another aspect of the present invention, a computer program product is provided, comprising computer program instructions, wherein the computer program instructions are used to execute the above-mentioned ultrasound image processing method when running.

[0150] A person skilled in the art can understand the specific implementation and beneficial effects of the above-mentioned ultrasonic image processing device, electronic device, storage medium and computer program product by reading the above-mentioned detailed description of the ultrasonic image processing method. For the sake of brevity, they will not be repeated here.

[0151] Although example embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above example embodiments are merely illustrative and are not intended to limit the scope of the present invention. Various changes and modifications may be made therein by those skilled in the art without departing from the scope and spirit of the present invention. All such changes and modifications are intended to be included within the scope of the present invention as claimed in the appended claims.

[0152] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0153] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units described is merely a logical function division. In actual implementation, other division methods may be used, such as combining or integrating multiple units or components into another device, or ignoring or not performing some features.

[0154] In the description provided herein, numerous specific details are described. However, it is understood that embodiments of the present invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques are not shown in detail so as not to obscure the understanding of this description.

[0155] Similarly, it should be understood that in order to streamline the present invention and aid in understanding one or more of the various inventive aspects, in the description of exemplary embodiments of the present invention, the various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, the present method should not be interpreted as reflecting the intention that the claimed invention requires more features than those explicitly recited in each claim. More precisely, as reflected in the corresponding claims, the inventive point is that the corresponding technical problem can be solved with fewer features than all the features of a single disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into the detailed description, with each claim itself serving as a separate embodiment of the present invention.

[0156] It will be understood by those skilled in the art that, except where mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or apparatus disclosed herein may be combined in any combination. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature providing the same, equivalent, or similar purpose.

[0157] Furthermore, those skilled in the art will appreciate that although some embodiments described herein include certain features included in other embodiments but not other features, combinations of features from different embodiments are intended to be within the scope of the present invention and to form different embodiments. For example, in the claims, any of the claimed embodiments may be used in any combination.

[0158] The various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art will appreciate that in practice, a microprocessor or digital signal processor (DSP) may be used to implement some or all of the functions of some modules in the ultrasound image processing apparatus according to an embodiment of the present invention. The present invention may also be implemented as a device program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program implementing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0159] It should be noted that the above embodiments illustrate rather than limit the invention, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between brackets should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising several different elements and by means of appropriately programmed computers. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names.

[0160] The foregoing description is merely a specific embodiment of the present invention or an illustration of a specific embodiment. The scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be readily conceived by a person skilled in the art within the technical scope disclosed in the present invention are intended to be encompassed by the scope of protection of the present invention. The scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A method for processing an ultrasonic image, characterized in that: The method comprises: Acquire intravascular ultrasound images; Inputting the intravascular ultrasound image into an image segmentation model to obtain a segmented image, wherein the image segmentation model is used to perform image segmentation on calcified plaques in the intravascular ultrasound image, the image segmentation model comprising an interconnected prediction module and a residual refinement module, the prediction module comprising at least one convolutional block attention module; Based on the calcified plaque in the segmented image, the area occupied by the calcified plaque is determined.

2. The method according to claim 1, characterized in that The prediction module includes M-level downsampling blocks and M-level upsampling blocks connected in series, where M is a positive integer, and downsampling blocks of the same level correspond to upsampling blocks of the same level. Each of the at least one convolutional block attention modules is directly connected to the back of a level upsampling block or a level downsampling block. Inputting the intravascular ultrasound image into an image segmentation model to obtain a segmented image includes: downsampling the intravascular ultrasound image sequentially using the M-level downsampling blocks to obtain a plurality of downsampling results; performing upsampling using the M-level upsampling block according to the multiple downsampling results to obtain a target feature map of the intravascular ultrasound image; Inputting the target feature map into the residual refinement module for refinement to obtain the segmented image; Among them, each of the convolution block attention modules is used to perform channel attention calculation and / or spatial attention calculation on the downsampling result output by the downsampling block directly connected to the convolution block attention module or the upsampling result output by the upsampling block directly connected to the convolution block attention module.

3. The method according to claim 2, characterized in that There are at least M convolution block attention modules, and each level of downsampling block is directly connected to a convolution block attention module. The downsampling of the intravascular ultrasound image by using the M-level downsampling blocks in sequence to obtain a plurality of downsampling results respectively includes: Inputting the intravascular ultrasound image into a first-level downsampling block in the M-level downsampling blocks to obtain a first downsampling feature map of the intravascular ultrasound image, and inputting the first downsampling feature map into an attention module of a convolutional block directly connected to the first-level downsampling block to obtain a first downsampling result; Inputting the i-1th downsampling result corresponding to the i-1th level downsampling block in the M-level downsampling blocks into the i-th level downsampling block to obtain the i-th downsampling feature map of the intravascular ultrasound image, and inputting the i-th downsampling feature map into the convolution block attention module directly connected to the i-th level downsampling block to obtain the i-th downsampling result, wherein i is a positive integer, and 1 <i≤M。 4. The method according to claim 2, characterized in that There are at least M convolution block attention modules, and each level of upsampling block is directly connected to a convolution block attention module. The step of performing upsampling using the M-level upsampling block according to the multiple downsampling results to obtain a target feature map of the intravascular ultrasound image includes: Determine an Mth upsampling feature map using the Mth upsampling block in the M-level upsampling blocks according to the Mth downsampling result corresponding to the Mth downsampling block in the M-level downsampling blocks; Input the M-th upsampled feature map into the convolutional block attention module directly connected to the M-th upsampling block to obtain the M-th upsampling result; Concatenate the j-th downsampling result corresponding to the j-th downsampling block in the M-th downsampling block and the (j + 1)-th upsampling result corresponding to the (j + 1)-th upsampling block in the M-th upsampling block, and input the concatenated result into the j-th upsampling block to obtain the j-th upsampled feature map of the intravascular ultrasound image. Then input the j-th upsampled feature map into the convolutional block attention module directly connected to the j-th upsampling block to obtain the j-th upsampling result, where j is a positive integer and 1 ≤ j < M, and the 1st upsampling result is the target feature map of the intravascular ultrasound image.

5. The method according to claim 2, characterized in that The prediction module further includes a dilated spatial pyramid pooling module, and the dilated spatial pyramid pooling module is connected between the M-th downsampling block in the M-th downsampling block and the M-th upsampling block in the M-th upsampling block. The upsampling using the M-th upsampling block according to the multiple downsampling results to obtain the target feature map of the intravascular ultrasound image includes: Input the M-th downsampling result corresponding to the M-th downsampling block into the dilated spatial pyramid pooling module for dilated convolution operation, concatenate the M-th downsampling result and the downsampling result after the dilated convolution operation to obtain the concatenated downsampling result, and input the concatenated downsampling result into the M-th upsampling block to obtain the M-th upsampling result; According to the M-th upsampling result and the downsampling results other than the M-th downsampling result among the multiple downsampling results, use the (M - 1)-th upsampling block to the 1st upsampling block in the M-th upsampling block for upsampling to obtain the target feature map.

6. The method according to any one of claims 2 to 5, characterized in that The prediction module further includes at least one activation function, each activation function is connected behind the corresponding convolutional block attention module, and each activation function is used to perform non-linear mapping on the corresponding downsampling result or upsampling result.

7. The method according to any one of claims 1 to 5, characterized in that After determining the area occupied by the calcified plaque based on the calcified plaque in the segmentation image, the method further includes: For each calcified plaque in the segmentation image, determine two tangents of the area occupied by the calcified plaque passing through the center point of the segmentation image, and determine the included angle between the two tangents to obtain the calcification angle corresponding to the calcified plaque. According to the calcification angles corresponding to all the calcified plaques in the segmentation image, determine the total angle of the calcified plaques in the segmentation image.

8. The method according to claim 7, characterized in that After determining the total angle of the calcified plaques in the segmentation image, the method further includes: Determine the calcification grade according to the corresponding relationship between the total angle and the calcification grade.

9. The method according to claim 7, characterized in that The method further includes: Display the segmentation image and the geometric markers of the calcified plaques in the segmentation image, where the geometric markers include at least one of the tangents, the calcification angles corresponding to at least one calcified plaque in the segmentation image, the outer contour of the image corresponding to the included angle between the two tangents, and the total angle.

10. An ultrasonic image processing device, characterized in that: Includes: An image acquisition module, used for acquiring intravascular ultrasound images; an image segmentation module, configured to input the intravascular ultrasound image into an image segmentation model to obtain a segmented image, wherein the image segmentation model is configured to perform image segmentation on calcified plaques in the intravascular ultrasound image, the image segmentation model comprising an interconnected prediction module and a residual refinement module, the prediction module comprising at least one convolutional block attention module; The first determining module is configured to determine an area occupied by the calcified plaque based on the calcified plaque in the segmented image.

11. An electronic device comprising a processor and a memory, characterized in that: The memory stores computer program instructions, which are used by the processor to execute the ultrasound image processing method according to any one of claims 1 to 9 when the processor runs the computer program instructions.

12. A storage medium having computer program instructions stored thereon, characterized in that: The computer program instructions are used to execute the ultrasound image processing method according to any one of claims 1 to 9 when executed.

13. A computer program product comprising computer program instructions, characterized in that The computer program instructions are used to execute the ultrasound image processing method according to any one of claims 1 to 9 when executed.

Images