An image segmentation method and system

Abstract

The application discloses an image segmentation method and system, and relates to the technical field of medical image segmentation. The method comprises the following steps: acquiring a target medical ultrasonic image; inputting the target medical ultrasonic image into a trained image semantic segmentation model to obtain a final image segmentation map; the trained image semantic segmentation model comprises a trained first segmentation network and a trained second segmentation network; the trained first segmentation network is used for performing semantic segmentation on the target medical ultrasonic image to obtain an initial image segmentation map; the trained second segmentation network is used for performing feature extraction on the initial image segmentation map and the target medical ultrasonic image at multiple extraction scales to obtain multiple scale features, fusing all the scale features to obtain fused features, and obtaining the final image segmentation map according to the fused features. The initial image segmentation map output by the first segmentation network is input into the second segmentation network as prior knowledge for further segmentation, and the segmentation result of the tissue is improved.

Description

An image segmentation method and system Technical Field

[0001] This invention relates to the field of medical image segmentation technology, and in particular to an image segmentation method and system. Background Technology

[0002] Semantic segmentation is widely used in medical image analysis. Semantic segmentation typically refers to using computers to process images, identify objects within them, and assign a class label to each pixel of a given image. In recent years, variants based on the Unet architecture have achieved good results in ultrasound segmentation, such as for breast cancer, IVUS (intravascular ultrasound), and kidney ultrasound. However, some tasks, such as neck ultrasound and thyroid ultrasound, suffer from misclassification of similar tissues. Figure 1(a) shows a clinical thyroid ultrasound image. Figure 1(b) shows the corresponding labels, where the gray area represents the thyroid gland, the circles without line segments next to the thyroid gland are blood vessels, and the circles with horizontal line segments inside the thyroid gland are thyroid nodules. Figure 1(c) shows the prediction result from Unet. Because one tissue structure in Figure 1(a) is very similar to a blood vessel, two yellow blood vessels appear in Figure 1(c). This is because Unet failed to learn the location information of the blood vessels, leading to confusion between the two similar tissues.

[0003] Current methods to address this issue primarily involve inserting a context information module into UNet to extract useful information and improve segmentation performance. However, most existing methods mainly extract feature-based context information, such as spatial distance extraction; using Atrous Spatial Pyramid Pooling (ASPP) to extract near-range multi-scale context information; employing SE-Net and EncNet to capture global context information to estimate channel features for weighted convolutional feature layers; and using Dynamic Multi-Scale Filtering (DMNet) to extract context features through convolutional kernels. However, these existing feature-based context information extraction methods do not consider category context information, leading to confusion of similar structures and low segmentation performance. Summary of the Invention

[0004] The purpose of this invention is to provide an image segmentation method and system that can improve the segmentation effect of tissues in images.

[0005] To achieve the above objectives, the present invention provides the following solution:

[0006] An image segmentation method, the method comprising:

[0007] Acquire target medical ultrasound images;

[0008] The target medical ultrasound image is input into a trained image semantic segmentation model to obtain a final image segmentation map; the trained image semantic segmentation model includes a trained first segmentation network and a trained second segmentation network; the final image segmentation map includes the final type information and final location information of the tissue in the target medical ultrasound image;

[0009] The trained first segmentation network is used to perform semantic segmentation on the target medical ultrasound image to obtain an initial image segmentation map; the initial image segmentation map includes initial type information and initial location information of the tissue in the target medical ultrasound image;

[0010] The trained second segmentation network is used to extract features from the initial image segmentation map and the target medical ultrasound image at multiple extraction scales to obtain multiple scale features, and then fuses all the scale features to obtain fused features. The final image segmentation map is obtained based on the fused features.

[0011] Optionally, the trained second segmentation network includes an HLCS feature extraction module, a multiplication module, an encoding module, a DCPF module, and a decoding module;

[0012] The HLCS feature extraction module is used to extract features from the initial image segmentation map to obtain deep convolutional features.

[0013] The multiplication module is used to perform a multiplication operation on the target medical ultrasound image and the initial image segmentation map to obtain a product feature map;

[0014] The encoding module is used to encode the product feature map to obtain encoded features;

[0015] The DCPF module is used to extract features from the encoded features and the deep convolutional features at multiple extraction scales to obtain multiple scale features, and to fuse all the scale features to obtain fused scale features. Then, a first convolutional layer is used to extract features from the fused scale features to obtain fused features.

[0016] The decoding module is used to decode the fused features to obtain the final image segmentation map.

[0017] Optionally, the HLCS feature extraction module includes several deep convolutional sub-modules connected in sequence and a deep convolutional unit; the deep convolutional sub-module includes a deep convolutional unit and a first max pooling layer connected in sequence; the deep convolutional unit includes several deep convolutional layers connected in sequence.

[0018] Optionally, the encoding module includes a plurality of encoding sub-modules connected in sequence and an encoding convolutional unit; the encoding sub-module includes an encoding convolutional unit and a second max pooling layer connected in sequence; the encoding convolutional unit includes an encoding convolutional sub-unit and an encoding fusion sub-unit, the encoding convolutional sub-unit includes a plurality of second convolutional layers connected in sequence, and the encoding fusion sub-unit is used to perform an addition operation on the input and output of the encoding convolutional sub-unit.

[0019] Optionally, the DCPF module includes several scale feature extraction sub-modules and fusion sub-modules;

[0020] The scale feature extraction submodule includes a feature extraction unit, a category context convolution unit, and an ASPP convolution unit; each scale feature extraction submodule corresponds to an extraction scale.

[0021] The feature extraction unit is used to extract features from the encoded features at the extraction scale corresponding to the scale feature extraction submodule to obtain global fusion features;

[0022] The category context convolutional unit is used to fuse the deep convolutional features and the global fusion features to obtain a category dynamic convolutional kernel;

[0023] The ASPP convolutional unit is used to extract features from the encoded features using the category dynamic convolutional kernel to obtain scale features;

[0024] The fusion submodule is used to fuse the scale features obtained by all the scale feature extraction submodules to obtain fused scale features, and to use the first convolutional layer to extract features from the fused scale features to obtain fused features.

[0025] Optionally, the feature extraction unit includes a global feature extraction subunit, a paired convolution subunit, and a global-local fusion subunit;

[0026] The global feature extraction subunit is used to extract features from the encoded features through multiple global convolutional layers to obtain multiple global sub-features, and to fuse all the global sub-features to obtain global features; one global convolutional layer corresponds to one global sub-feature.

[0027] The paired convolutional subunit is used to extract features from the encoded features at the extraction scale corresponding to the scale feature extraction submodule to obtain local features;

[0028] The global-local fusion subunit is used to fuse the global features and the local features to obtain global fused features.

[0029] Optionally, fusing the deep convolutional features and the global fusion features to obtain the category dynamic convolutional kernel specifically includes:

[0030] The deep convolutional features and the global fusion features are multiplied together to obtain the product features.

[0031] The encoded features are extracted using a third convolutional layer to obtain the encoded convolutional features.

[0032] Multiply the depthwise convolutional features and the encoded convolutional features to obtain the channel category features;

[0033] The product feature and the channel category feature are multiplied together to obtain the category convolution feature;

[0034] The category convolutional features are multiplied by a fixed convolutional kernel corresponding to the extraction scale to obtain the category convolutional kernel;

[0035] The dimensionality of the category convolution kernel is adjusted using an adaptive pooling layer to obtain a fixed-size dynamic category convolution kernel.

[0036] Optionally, the step of extracting scale features from the encoded features using the category-based dynamic convolution kernel specifically includes:

[0037] The dynamic convolutional kernels of the category are dilated using multiple dilation rates of different sizes to obtain ASPP convolutional layers corresponding to each dilation rate. The encoded features are then extracted using each ASPP convolutional layer to obtain multiple local scale features. The encoded features and all the local scale features are fused to obtain ASPP convolutional features. Finally, a fourth convolutional layer is used to extract features from the ASPP convolutional features to obtain scale features.

[0038] Optionally, before inputting the target medical ultrasound image into the trained image semantic segmentation model, the method further includes: training the image semantic segmentation model, the training process of which is as follows:

[0039] Obtain a sample set; the sample set includes several sample medical ultrasound images and the final image segmentation map corresponding to each sample medical ultrasound image;

[0040] The second model parameters of the second segmentation network are fixed. The sample medical ultrasound image is used as input, and the final image segmentation map corresponding to the sample medical ultrasound image is used as the label to train the image semantic segmentation model, thereby obtaining the trained first segmentation network.

[0041] The trained first segmentation network and the trained second segmentation network are trained using the sample set to obtain a trained second segmentation network; the trained first segmentation network and the trained second segmentation network constitute the trained image semantic segmentation model.

[0042] The present invention also provides an image segmentation system, the system comprising:

[0043] The image acquisition module is used to acquire target medical ultrasound images;

[0044] An image segmentation module is used to input the target medical ultrasound image into a trained image semantic segmentation model to obtain a final image segmentation map. The trained image semantic segmentation model includes a trained first segmentation network and a trained second segmentation network. The final image segmentation map includes the final type information and final location information of the tissue in the target medical ultrasound image. The trained first segmentation network is used to perform semantic segmentation on the target medical ultrasound image to obtain an initial image segmentation map. The initial image segmentation map includes the initial type information and initial location information of the tissue in the target medical ultrasound image. The trained second segmentation network is used to extract features from the initial image segmentation map and the target medical ultrasound image at multiple extraction scales to obtain multiple scale features, and then fuses all the scale features to obtain fused features. The final image segmentation map is obtained based on the fused features.

[0045] According to specific embodiments provided by the present invention, the following technical effects are disclosed: The present invention provides an image segmentation method and system. The method includes: acquiring a target medical ultrasound image; inputting the target medical ultrasound image into a trained image semantic segmentation model to obtain a final image segmentation map; the trained image semantic segmentation model includes a trained first segmentation network and a trained second segmentation network; the final image segmentation map includes the final type information and final location information of the tissue in the target medical ultrasound image; wherein, the trained first segmentation network is used to perform semantic segmentation on the target medical ultrasound image to obtain an initial image segmentation map; the initial image segmentation map includes the initial type information and initial location information of the tissue in the target medical ultrasound image; the trained second segmentation network is used to extract features from the initial image segmentation map and the target medical ultrasound image at multiple extraction scales to obtain multiple scale features, and fuse all scale features to obtain fused features, and obtain the final image segmentation map based on the fused features. The present invention obtains the final image segmentation map through two-layer segmentation networks, making the obtained tissue segmentation results more accurate. Compared with the prior art, which considers category context information (inputting the initial image segmentation map as a category prior into the second segmentation network for further segmentation processing to obtain the final image segmentation map), the segmentation effect is better. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 is a schematic diagram of segmenting thyroid ultrasound images using Unet according to the present invention;

[0048] Figure 2 is a flowchart illustrating the image segmentation method provided in an embodiment of the present invention;

[0049] Figure 3 is a schematic diagram of the image semantic segmentation model structure provided in an embodiment of the present invention;

[0050] Figure 4 is a schematic diagram of the DCBF module structure provided in an embodiment of the present invention;

[0051] Figure 5 is a schematic diagram of the paired convolution process for extracting local features according to an embodiment of the present invention;

[0052] Figure 6 is a schematic diagram comparing the segmentation performance of the image semantic segmentation model trained by the present invention and an existing segmentation model, according to an embodiment of the present invention. Detailed Implementation

[0053] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] The purpose of this invention is to provide an image segmentation method and system that obtains a final image segmentation map through a two-layer segmentation network, making the segmentation results of the tissue more accurate. Compared with the prior art, it considers category context information (the initial image segmentation map is input as a category prior label into the second segmentation network for further segmentation processing to obtain the final image segmentation map), thus improving the segmentation effect of tissues in medical ultrasound images.

[0055] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0056] As shown in Figure 2, this invention provides an image segmentation method, the method comprising:

[0057] S1: Acquire the target medical ultrasound image.

[0058] S2: The target medical ultrasound image is input into the trained image semantic segmentation model SCCNet to obtain the final image segmentation map. The trained image semantic segmentation model includes a trained first segmentation network and a trained second segmentation network. The final image segmentation map includes the final type information and final location information of the tissue in the target medical ultrasound image. If the target medical ultrasound image is a thyroid ultrasound image, the tissue includes the thyroid gland, thyroid nodules, and blood vessels in the thyroid gland. If the target medical ultrasound image is a kidney ultrasound image, the tissue includes the kidney and blood vessels in the kidney.

[0059] As shown in Figure 3, this embodiment divides the segmentation of the target medical ultrasound image into two stages: the first stage is the trained first segmentation network, and the second stage is the trained second segmentation network.

[0060] The trained first segmentation network is used to perform semantic segmentation on the target medical ultrasound image to obtain an initial image segmentation map; the initial image segmentation map includes the initial type information and initial location information of the tissue in the target medical ultrasound image.

[0061] The trained second segmentation network is used to extract features from the initial image segmentation map and the target medical ultrasound image at multiple extraction scales to obtain multiple scale features, and then fuses all the scale features to obtain fused features. The final image segmentation map is obtained based on the fused features.

[0062] Before inputting the target medical ultrasound image into the trained image semantic segmentation model, the method further includes: training the image semantic segmentation model, the training process of which is as follows:

[0063] Obtain a sample set; the sample set includes several sample medical ultrasound images and the final image segmentation map corresponding to each sample medical ultrasound image.

[0064] The second model parameters of the second segmentation network are fixed. The sample medical ultrasound image is used as input, and the final image segmentation map corresponding to the sample medical ultrasound image is used as the label to train the image semantic segmentation model, thereby obtaining the trained first segmentation network.

[0065] The trained first segmentation network and the trained second segmentation network are trained using the sample set to obtain a trained second segmentation network; the trained first segmentation network and the trained second segmentation network constitute the trained image semantic segmentation model.

[0066] The DBPB module in the first segmentation network is the DBPB module in existing patent CN115456985A, and the total loss function in the first segmentation network is Loss. Total1 for:

[0067]

[0068] Where i represents the replacement of the i-th convolutional block by the DBPB module in the first stage (the convolutional block is any one of the convolutional blocks in the encoding and decoding modules of the first segmentation network), l represents the total number of convolutional blocks in the encoding and decoding modules of the first segmentation network, and Dice loss is... seg Used to predict segmentation map (The second stage is S) pr2 ) and the true segmentation map S gt To approximate even more closely, where ε = 0.0001, to ensure that the denominator in the following formula is not 0.

[0069]

[0070] Through the loss function To determine which convolutional blocks in the Unet model the DBPB module replaces, this embodiment uses the aforementioned loss function to determine the location of the DBPB module, which replaces the first two convolutional blocks of the Unet model encoding module as shown in Figure 2.

[0071] It should be noted that the first segmentation network can be the Unet model or any existing improved Unet model.

[0072] Then, keeping the model parameters in the first segmentation network unchanged, we begin training the second-stage network (i.e., the second segmentation network). Initial sample image segmentation diagram. And weighted input images (the weighted input images are sample medical ultrasound images and The multiplied image is used as input to a two-stage network (i.e., the second segmentation network), where The Class Context Module (DCPF) serves as the category prior input in the second stage, and utilizes the loss function Loss. Total2 The final segmentation result S of the second stage is obtained. pr2 (i.e., the final sample image segmentation map), where Loss Total2 =loss seg This allows us to obtain the trained second segmentation network.

[0073] As shown in Figure 3, the trained second segmentation network includes an HLCS (High-level class semantic map) feature extraction module, a multiplication module, an encoding module, a DCPF (Dynamic class pyramid filter) module, and a decoding module.

[0074] The HLCS feature extraction module is used to extract features from the initial image segmentation map to obtain deep convolutional features.

[0075] The multiplication module is used to multiply the target medical ultrasound image and the initial image segmentation map to obtain a product feature map.

[0076] The encoding module is used to encode the product feature map to obtain encoded features.

[0077] The DCPF module is used to extract features from the encoded features and the deep convolutional features at multiple extraction scales to obtain multiple scale features, and to fuse all the scale features to obtain fused scale features. Then, a first convolutional layer is used to extract features from the fused scale features to obtain fused features.

[0078] The decoding module is used to decode the fused features to obtain the final image segmentation map.

[0079] The HLCS feature extraction module includes several deep convolutional sub-modules connected in sequence and a deep convolutional unit; the deep convolutional sub-module includes a deep convolutional unit and a first max pooling layer connected in sequence; the deep convolutional unit includes several deep convolutional layers connected in sequence. As shown in Figure 3, in this embodiment, the HLCS feature extraction module consists of three deep convolutional sub-modules and one deep convolutional unit, and the deep convolutional unit includes two deep-wise convolutional layers connected in sequence.

[0080] First, a first-stage segmentation map (i.e., the initial image segmentation map) is obtained through a one-stage Unet (i.e., the first segmentation network trained). Then, HLCS features (HLCS features, i.e., deep convolution features) of size N×32×32 are obtained through Depwise convolution.

[0081] The encoding module includes several encoding sub-modules connected in sequence and an encoding convolutional unit; the encoding sub-module includes encoding convolutional units and second max pooling layers connected in sequence; the encoding convolutional unit includes encoding convolutional sub-units and encoding fusion sub-units, the encoding convolutional sub-units include several second convolutional layers connected in sequence, and the encoding fusion sub-units are used to add the input and output of the encoding convolutional sub-units.

[0082] The decoding module includes several decoding sub-modules connected in sequence and a decoding convolutional unit; the decoding sub-module includes a decoding convolutional unit and a third max pooling layer connected in sequence; the decoding convolutional unit includes a decoding convolutional sub-unit and a decoding fusion sub-unit, the decoding convolutional sub-unit includes several third convolutional layers connected in sequence, and the decoding fusion sub-unit is used to add the input and output of the decoding convolutional sub-unit.

[0083] As shown in Figure 3, in this embodiment, the encoding module and the decoding module can constitute a Unet model. The encoding module includes three encoding sub-modules, and the encoding convolutional sub-unit includes two sequentially connected second convolutional layers. The decoding module includes three decoding sub-modules, and the decoding convolutional sub-unit includes two sequentially connected second convolutional layers.

[0084] As shown in Figure 4, the DCPF module includes several scale feature extraction sub-modules and fusion sub-modules.

[0085] The scale feature extraction submodule includes a feature extraction unit, a category context convolution unit, and an ASPP convolution unit; each scale feature extraction submodule corresponds to an extraction scale.

[0086] The feature extraction unit is used to extract features from the encoded features at the extraction scale corresponding to the scale feature extraction submodule to obtain global fusion features.

[0087] The category context convolutional unit is used to fuse the deep convolutional features and the global fusion features to obtain a category dynamic convolutional kernel.

[0088] The ASPP convolutional unit is used to extract features from the encoded features using the category dynamic convolutional kernel to obtain scale features.

[0089] The fusion submodule is used to fuse the scale features obtained by all the scale feature extraction submodules to obtain fused scale features, and to use the first convolutional layer to extract features from the fused scale features to obtain fused features.

[0090] The feature extraction unit includes a global feature extraction subunit, a paired convolution subunit, and a global-local fusion subunit.

[0091] The global feature extraction subunit is used to extract features from the encoded features through multiple global convolutional layers to obtain multiple global sub-features, and to fuse all the global sub-features to obtain global features; each global convolutional layer corresponds to one global sub-feature.

[0092] The paired convolutional subunit is used to extract features from the encoded features at the extraction scale corresponding to the scale feature extraction submodule to obtain local features.

[0093] The global-local fusion subunit is used to fuse the global features and the local features to obtain global fused features.

[0094] As shown in Figure 4(a), long-range and local features of the image are extracted to obtain a convolutional kernel (global fusion feature) based on feature information. The encoded features... The correlation map L (i.e., global features) is obtained by multiplying two 1x1 convolutions. The correlation map L contains long-range features. Furthermore, for... Local information is extracted using paired convolution to obtain local features. Will Multiplying by L yields features containing both long-range and local information at a scale of k×k. (i.e., global fusion features).

[0095] Specifically, as shown in Figure 5, this embodiment proposes a dynamic multi-scale filtering method (DCPF) based on class space correlation priors. When constructing the dynamic convolution kernel, paired convolutions are used to extract local contextual information for DCPF. Let the input features of DCPF be... The dimensions h×w×c are 32×32×512, as shown by the gray rectangle in Figure 5. Here, paired convolutions use a 15×15×512 context-aware window as a sliding window (when k=15) to obtain a coarse context information extraction range corresponding to the pixel at position (i, j), which is shown as the black dot in Figure 5, for example, pixel A. In each context-aware window, m and n represent the pixel coordinates within the context-aware window. This embodiment uses a 3×3 local convolution to learn the local semantic information of the middle pixel (pixel A), and similarly uses a 3×3 local convolution at position (im, jn) to extract local semantic information, for example, pixels B, C, D, and E as the local semantic information surrounding the middle pixel A.

[0096]

[0097] in, This represents the difference in convolution output between the middle pixel position (i,j) and its surrounding pixel positions (im,jn); * indicates a local convolution operator. Represents the Gaussian function; Θ i,j and Θ i-m,j-n There are two convolution kernels; F i,j and F i -m,j-n These are the local features at positions (i,j) and (im,jn), respectively. It should be noted that the sliding window size is 15×15; the sliding window at the middle pixel is not encoded as a feature. When inside, the sliding window extends beyond the encoded feature. The surrounding pixels of a portion are set to 0. For example, if pixel A in Figure 5 is a coded feature... If the first pixel in the top left corner is pixel A, then pixel B in the sliding window of pixel A is not an encoded feature. For pixels in the middle, the difference values ​​of the surrounding pixels in this case are set to 0. Therefore, each middle pixel can obtain 15×15 convolution output differences, which results in a shape mask of size 15×15 for the middle pixel A.

[0098] First, a context-aware window is used as a sliding window to obtain the range corresponding to the coarse context information extraction. Within the context-aware window, two paired 3×3 convolutions are used to obtain the difference between the local convolutions of the center pixel and the surrounding pixels. For example, the difference between the 3×3 convolution of pixel A and the 3×3 convolution of pixel B can be represented as... Based on this calculation formula, the local convolutional information surrounding pixel A can be obtained: for example, pixels B, C, D, and E. Then, the shape mask (Shape mask1) of pixel A is obtained. Next, a context-aware window acts as a sliding window to traverse the entire image (encoding features). ), that is, encoding features Each pixel in the image is considered an intermediate pixel. Within the sliding window corresponding to each intermediate pixel, the context information map, i.e., the shape mask (Shape mask1), is calculated for each center pixel using the previously used paired convolution method. Finally, all pixels in the medical ultrasound image obtain their corresponding shape masks, resulting in a local context information feature (k=15) with dimensions hw×15×15. Here, h and w represent the encoded features. The length and width represent the encoded features. It has h×w pixels.

[0099] To avoid the difference between the local convolution of the middle pixel and its surrounding pixels being negative, this embodiment uses a Gaussian function to map the result to a positive value, as shown in the following formula:

[0100]

[0101] in, This represents the difference value after processing with a Gaussian function, where σ is a parameter of the Gaussian function, set to 3 in paired convolution operations, and 'a' is a specific difference value.

[0102] In this embodiment, the default is... A small difference value will result in a large semantic correlation value. This is because the values ​​of two pixels belonging to the same object and their contextual information are considered to exhibit a strong correlation, as they often coexist in training images. Based on this difference calculation method, the contextual result is obtained. The shape mask is hw×k×k, where the shape mask is k×k (k=15). The total number of pixels is hw, and each pixel corresponds to a specific shape mask, representing the context information of that pixel and its surrounding pixels. The context information of all pixels constitutes the local features.

[0103] As shown in Figure 4(b), the segmentation map of the first stage is introduced as category prior information. In the above category context convolutional unit, the deep convolutional features and the global fusion features are fused to obtain the category dynamic convolutional kernel, which specifically includes:

[0104] The deep convolutional features and the global fusion features are multiplied together to obtain the product features.

[0105] The encoded features are extracted using a third convolutional layer to obtain the encoded convolutional features.

[0106] The channel category features are obtained by multiplying the depthwise convolutional features and the encoded convolutional features.

[0107] The product feature and the channel category feature are multiplied together to obtain the category convolution feature.

[0108] The category convolutional features are multiplied by a fixed convolutional kernel corresponding to the extraction scale to obtain the category convolutional kernel.

[0109] The dimensionality of the category convolution kernel is adjusted using an adaptive pooling layer to obtain a fixed-size dynamic category convolution kernel.

[0110] Specifically: the class label map (deep convolutional features) predicted by Unet in the first stage is used as the class prior. and integrate it with global fusion features Multiplication yields the product characteristics Prior information about the category Perform a weighted increase. Then apply the category prior. With coding features Multiplying these yields the channel category information h1 (i.e., the channel category feature). Finally, ... Multiplying by h1 yields the final categorical convolutional feature C. This categorical convolutional feature is then used as a weight to weight a fixed convolutional kernel S, resulting in a dynamic convolutional kernel (category convolutional kernel) containing category information. The dimension of this kernel is adjusted to a fixed-size dynamic convolutional kernel, Class Context, through an adaptive pooling layer. In this embodiment, the fixed dimension can be 3x3x512.

[0111] It should be noted that in this embodiment, category labels can also be used to weight any information in the context module to finally obtain the dynamic convolution kernel Class Context.

[0112] The step of extracting scale features from the encoded features using the category-based dynamic convolution kernel specifically includes:

[0113] The dynamic convolutional kernels of the category are dilated using multiple dilation rates of different sizes to obtain ASPP convolutional layers corresponding to each dilation rate. The encoded features are then extracted using each ASPP convolutional layer to obtain multiple local scale features. The encoded features and all the local scale features are fused to obtain ASPP convolutional features. Finally, a fourth convolutional layer is used to extract features from the ASPP convolutional features to obtain scale features.

[0114] As shown in Figure 4(c), rich multi-scale features are obtained through variable convolutional kernels containing categories, and scale features DCPF(k) corresponding to the extraction scale are obtained through ASPP convolutional units. That is, the category dynamic convolutional kernel is dilated with three different dilation rates to obtain three ASPP convolutional layers, which are then used to extract encoded features. The local features are extracted and all the extracted local features are compared with the encoded features. The scale features DCPF(k) are obtained by fusing and then performing a 1×1 convolution. k is the paired convolution in Figure 4(a). The parameters used in this context are those used to adjust the scale of local information extraction. Different parameters result in different extracted local information; that is, one parameter corresponds to one extraction scale. The parameters of paired convolutions can control the extraction scale of local information, and changing k yields different local multi-scale information. Combining multi-scale information can further enhance the ability to extract contextual information.

[0115] As shown in Figure 4(d), by adjusting the size of k, scale features corresponding to multiple extraction scales are obtained. All scale features are fused and a fused feature DCPF (k = k1, k2, k3) is obtained through a 1×1 convolution. The fused feature contains global multi-scale information and local multi-scale information extracted by the ASPP convolutional unit.

[0116] This embodiment uses the segmentation results of a thyroid ultrasound image as an example, employing the model of this invention and existing models to segment the same thyroid ultrasound image. Figure 6(a) shows the thyroid ultrasound image, where it can be seen that the boundaries in the original thyroid ultrasound image are difficult to identify the thyroid tissue structure. Figure 6(b) is the ground truth label image. Figure 6(c) shows the prediction results of the existing UNet model, Figure 6(d) shows the prediction results of the existing BPBnet model, Figure 6(e) shows the prediction results of the existing CE-Net model, Figure 6(f) shows the prediction results of the existing DSNnet model, Figure 6(g) shows the prediction results of the existing RAGCnet model, Figure 6(h) shows the prediction results of the existing TransUNet model, and Figure 6(i) shows the prediction results of the two-level cascaded UNet model (the trained image semantic segmentation model) containing the DCPF context module in this invention. It is evident that some similar tissues appear in severe misclassifications, such as the results of the CE-Net and DSNnet models. The misclassification problem of the image semantic segmentation model trained in this invention is effectively improved, indicating that the scheme of introducing category prior information into the context structure is effective. This invention improves the segmentation effect of tissues in medical ultrasound images.

[0117] This invention utilizes category label maps as category prior information, introducing category priors into the context information module to significantly improve the ability to identify similar tissues in ultrasound segmentation. A dynamic convolutional kernel containing category information and varying with the input image is constructed. Furthermore, this invention considers local and global multi-scale information in the category context module. The image segmentation method provided by this invention outperforms existing models that incorporate context modules for segmenting ultrasound images with numerous similar tissues and poor contrast.

[0118] The present invention also provides an image segmentation system, the system comprising:

[0119] The image acquisition module is used to acquire target medical ultrasound images.

[0120] An image segmentation module is used to input the target medical ultrasound image into a trained image semantic segmentation model to obtain a final image segmentation map. The trained image semantic segmentation model includes a trained first segmentation network and a trained second segmentation network. The final image segmentation map includes the final type information and final location information of the tissue in the target medical ultrasound image. The trained first segmentation network is used to perform semantic segmentation on the target medical ultrasound image to obtain an initial image segmentation map. The initial image segmentation map includes the initial type information and initial location information of the tissue in the target medical ultrasound image. The trained second segmentation network is used to extract features from the initial image segmentation map and the target medical ultrasound image at multiple extraction scales to obtain multiple scale features, and then fuses all the scale features to obtain fused features. The final image segmentation map is obtained based on the fused features.

[0121] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0122] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. An image segmentation method, characterized in that, The method includes: acquiring a target medical ultrasound image; inputting the target medical ultrasound image into a trained image semantic segmentation model to obtain a final image segmentation map; the trained image semantic segmentation model includes a trained first segmentation network and a trained second segmentation network; the final image segmentation map includes the final type information and final location information of the tissue in the target medical ultrasound image; wherein, the trained first segmentation network is used to perform semantic segmentation on the target medical ultrasound image to obtain an initial image segmentation map; the initial image segmentation map includes the initial type information and initial location information of the tissue in the target medical ultrasound image; the trained second segmentation network is used to perform semantic segmentation on the target medical ultrasound image at multiple extraction scales. The initial image segmentation map and the target medical ultrasound image are used to extract features at multiple scales. All scale features are then fused to obtain fused features, and the final image segmentation map is obtained based on these fused features. The trained second segmentation network includes an HLCS feature extraction module, a multiplication module, an encoding module, a DCPF module, and a decoding module. The HLCS feature extraction module extracts features from the initial image segmentation map to obtain deep convolutional features. The multiplication module performs a multiplication operation on the target medical ultrasound image and the initial image segmentation map to obtain a product feature map. The encoding module encodes the product feature map to obtain encoded features. The DCPF module... The DCPF module extracts features from the encoded features and the deep convolutional features at multiple extraction scales to obtain multiple scale features. All scale features are then fused to obtain fused scale features. A first convolutional layer is used to extract features from the fused scale features to obtain fused features. The decoding module decodes the fused features to obtain the final image segmentation map. The DCPF module includes several scale feature extraction sub-modules and a fusion sub-module. Each scale feature extraction sub-module includes a feature extraction unit, a category context convolutional unit, and an ASPP convolutional unit. Each scale feature extraction sub-module corresponds to an extraction scale. The feature extraction unit is used to extract features from the scale feature extraction sub-modules at multiple extraction scales. The module extracts features from the encoded features at the extraction scale corresponding to the module to obtain global fusion features. The category context convolutional unit is used to fuse the deep convolutional features and the global fusion features to obtain a category dynamic convolutional kernel. Specifically, this includes: multiplying the deep convolutional features and the global fusion features to obtain product features; extracting features from the encoded features using a third convolutional layer to obtain encoded convolutional features; multiplying the deep convolutional features and the encoded convolutional features to obtain channel category features; multiplying the product features and the channel category features to obtain category convolutional features; and multiplying the category convolutional features with a fixed convolutional kernel corresponding to the extraction scale to obtain the category convolutional kernel.The dimensionality of the class convolutional kernel is adjusted using an adaptive pooling layer to obtain a fixed-size class dynamic convolutional kernel. The ASPP convolutional unit is used to extract features from the encoded features using the class dynamic convolutional kernel to obtain scale features. Specifically, this includes: performing dilation processing on the class dynamic convolutional kernel using multiple dilation rates of different sizes to obtain an ASPP convolutional layer corresponding to each dilation rate; extracting features from the encoded features using each ASPP convolutional layer to obtain multiple local scale features; fusing the encoded features and all the local scale features to obtain ASPP convolutional features; and extracting features from the ASPP convolutional features using a fourth convolutional layer to obtain scale features. The fusion submodule is used to fuse the scale features obtained by all the scale feature extraction submodules to obtain fused scale features; and extracting features from the fused scale features using the first convolutional layer to obtain fused features.

2. The image segmentation method according to claim 1, characterized in that, The HLCS feature extraction module includes several deep convolutional sub-modules connected in sequence and a deep convolutional unit; the deep convolutional sub-module includes a deep convolutional unit and a first max pooling layer connected in sequence; the deep convolutional unit includes several deep convolutional layers connected in sequence.

3. The image segmentation method according to claim 1, characterized in that, The encoding module includes several encoding sub-modules connected in sequence and an encoding convolutional unit; the encoding sub-module includes encoding convolutional units and second max pooling layers connected in sequence; the encoding convolutional unit includes encoding convolutional sub-units and encoding fusion sub-units, the encoding convolutional sub-units include several second convolutional layers connected in sequence, and the encoding fusion sub-units are used to add the input and output of the encoding convolutional sub-units.

4. The image segmentation method according to claim 1, characterized in that, The feature extraction unit includes a global feature extraction subunit, a paired convolution subunit, and a global-local fusion subunit. The global feature extraction subunit extracts features from the encoded features using multiple global convolutional layers to obtain multiple global sub-features, and then fuses all the global sub-features to obtain a global feature; each global convolutional layer corresponds to one global sub-feature. The paired convolution subunit extracts features from the encoded features at the extraction scale corresponding to the scale feature extraction subunit to obtain local features. The global-local fusion subunit fuses the global features and the local features to obtain a global fused feature.

5. The image segmentation method according to claim 1, characterized in that, Before inputting the target medical ultrasound image into the trained image semantic segmentation model, the method further includes: training the image semantic segmentation model, the training process of which is as follows: obtaining a sample set; the sample set includes several sample medical ultrasound images and the sample final image segmentation map corresponding to each sample medical ultrasound image; fixing the second model parameters of the second segmentation network, using the sample medical ultrasound image as input and the sample final image segmentation map corresponding to the sample medical ultrasound image as a label, training the image semantic segmentation model to obtain a trained first segmentation network; using the sample set to train the trained first segmentation network and the second segmentation network to obtain a trained second segmentation network; the trained first segmentation network and the trained second segmentation network constitute the trained image semantic segmentation model.

6. An image segmentation system, characterized in that, The system includes: an image acquisition module for acquiring a target medical ultrasound image; and an image segmentation module for inputting the target medical ultrasound image into a trained image semantic segmentation model to obtain a final image segmentation map. The trained image semantic segmentation model includes a trained first segmentation network and a trained second segmentation network. The final image segmentation map includes the final type information and final location information of the tissue in the target medical ultrasound image. The trained first segmentation network performs semantic segmentation on the target medical ultrasound image to obtain an initial image segmentation map. The initial image segmentation map includes the initial type information and initial location information of the tissue in the target medical ultrasound image. The trained second segmentation network... The second segmentation network is used to extract features from the initial image segmentation map and the target medical ultrasound image at multiple extraction scales to obtain multi-scale features. All scale features are then fused to obtain fused features, and the final image segmentation map is obtained based on the fused features. The trained second segmentation network includes an HLCS feature extraction module, a multiplication module, an encoding module, a DCPF module, and a decoding module. The HLCS feature extraction module extracts features from the initial image segmentation map to obtain deep convolutional features. The multiplication module performs a multiplication operation on the target medical ultrasound image and the initial image segmentation map to obtain a product feature map. The encoding module encodes the product feature map to obtain encoded features. The DCPF module is used to extract features from the encoded features and the depthwise convolutional features at multiple extraction scales to obtain multiple scale features, and to fuse all the scale features to obtain fused scale features. A first convolutional layer is then used to extract features from the fused scale features to obtain fused features. The decoding module is used to decode the fused features to obtain the final image segmentation map. The DCPF module includes several scale feature extraction sub-modules and a fusion sub-module. The scale feature extraction sub-module includes a feature extraction unit, a category context convolutional unit, and an ASPP convolutional unit. Each scale feature extraction sub-module corresponds to an extraction scale. The feature extraction unit is used to extract features at the specified scales. The feature extraction submodule extracts features from the encoded features at the extraction scale corresponding to the feature extraction submodule to obtain global fusion features. The category context convolutional unit is used to fuse the deep convolutional features and the global fusion features to obtain a category dynamic convolutional kernel. Specifically, this includes: multiplying the deep convolutional features and the global fusion features to obtain product features; extracting features from the encoded features using a third convolutional layer to obtain encoded convolutional features; multiplying the deep convolutional features and the encoded convolutional features to obtain channel category features; multiplying the product features and the channel category features to obtain category convolutional features; and multiplying the category convolutional features with a fixed convolutional kernel corresponding to the extraction scale to obtain the category convolutional kernel.The dimensionality of the class convolutional kernel is adjusted using an adaptive pooling layer to obtain a fixed-size class dynamic convolutional kernel. The ASPP convolutional unit is used to extract features from the encoded features using the class dynamic convolutional kernel to obtain scale features. Specifically, this includes: performing dilation processing on the class dynamic convolutional kernel using multiple dilation rates of different sizes to obtain an ASPP convolutional layer corresponding to each dilation rate; extracting features from the encoded features using each ASPP convolutional layer to obtain multiple local scale features; fusing the encoded features and all the local scale features to obtain ASPP convolutional features; and extracting features from the ASPP convolutional features using a fourth convolutional layer to obtain scale features. The fusion submodule is used to fuse the scale features obtained by all the scale feature extraction submodules to obtain fused scale features; and extracting features from the fused scale features using the first convolutional layer to obtain fused features.

Images