Prostate ultrasonic segmentation method and device based on multi-scale feature fusion

Abstract

The invention relates to a prostate ultrasonic segmentation method and device based on multi-scale feature fusion, a multi-scale jump connection mode is introduced, and a model can extract multi-scale feature information of a target area in a prostate image more effectively. Meanwhile, a channel and space attention mechanism MSA is added, unimportant feature information in the space or the channel is screened out through the attention mechanism, so that more key feature information is reserved, and an MSA module is added to each upsampling operation, so that the model can pay attention to useful feature information in each scale slice, and the segmentation precision is improved. According to the method, on the basis of the U-Net model, the convolution of the U-Net model is completely replaced by the mixed convolution, so that the calculation cost is reduced while the good segmentation precision is realized, and the precise segmentation of the prostate image is realized.

Description

Technical Field

[0001] The present application relates to the technical field of image segmentation, and in particular, to a prostate ultrasound segmentation method and device based on multi-scale feature fusion. Background Art

[0002] In recent years, some deep learning methods have also been introduced into the segmentation of prostate ultrasound images. For example, U-Net is a representative 2D convolutional neural network. Although U-Net has excellent performance in image segmentation tasks, it is limited by its single jump connection mode, which makes the model unable to fuse feature information from multiple different levels, thus affecting the segmentation effect. In addition, the 2D segmentation network can only extract features from the two-dimensional plane and cannot obtain the three-dimensional structural information in the image data. This is because the 2D segmentation network has certain limitations in processing the relationship between adjacent slices, and it often cannot identify and construct the contour representation information of the target area. Compared with the 2D model, the 3D model can pay more attention to the detailed features and spatial contours in the image data. In addition, the 3D model also has the ability to process a series of adjacent image slice data at the same time, which are usually described as "stacks" or "volumes". In this way, the 3D model can better capture the spatial relationship between objects and structures. Compared with the 2D model, the 3D model can more comprehensively capture the depth, height, volume and other features in the continuous image slices, thereby providing more complete and coherent image information. However, training 3D networks often requires higher hardware configuration and more computing costs, so in practical applications, the limitations of computing power and storage resources need to be fully considered.

[0003] In order to bridge the gap between 2D and 3D convolutional neural networks, 2.5D segmentation methods have emerged. Many 2.5D segmentation methods in the prior art integrate and incorporate volume information into two-dimensional convolution by improving or designing new architectures to achieve efficient volumetric medical image segmentation. However, although the 2.5D segmentation method can improve the performance of 2D convolutional neural networks and obtain spatial contour information at a low computational cost, it is not suitable for thick-layer data such as transrectal ultrasound. Since a large amount of information in this type of data exists on the section and there is less stereoscopic information, it is not easy to obtain three orthogonal planes with rich information like CT images. In addition, these methods do not effectively integrate the information between slices. They only input three planes (sagittal plane, coronal plane, and axial plane) into the 2D network for training, and finally superimpose and fuse them into the segmentation result by position. Summary of the invention

[0004] In order to overcome at least one deficiency in the prior art, the present application provides a prostate ultrasound segmentation method and device based on multi-scale feature fusion.

[0005] In a first aspect, a prostate ultrasound segmentation method based on multi-scale feature fusion is provided, comprising:

[0006] Construct a prostate ultrasound segmentation model; the prostate ultrasound segmentation model adopts a U-Net network, the U-Net network includes an encoder and a decoder, the encoder includes multiple encoding layers, the decoder includes multiple decoding layers, and the convolution modules in each encoding layer and each decoding layer are replaced with hybrid convolution modules; a channel and spatial attention module is also set between the encoder and the decoder, the spatial attention module is used to retain useful feature information in the features output by the encoding layer, the useful feature information is input into the decoder for multi-scale feature fusion, and the segmentation result is output;

[0007] Training the prostate ultrasound segmentation model to obtain a trained prostate ultrasound segmentation model;

[0008] The prostate ultrasound image to be segmented is input into the trained prostate ultrasound segmentation model to obtain the segmentation result.

[0009] In one embodiment, the hybrid convolution module includes: a first 3D convolution block, a first 2D convolution block, a second 2D convolution block, a second 3D convolution block, and a 2.5D convolution block;

[0010] The input of the hybrid convolution module passes through the first 3D convolution block, the first 2D convolution block, and then is input into the 2.5D convolution block;

[0011] The input of the hybrid convolution module passes through the second 2D convolution block and the second 3D convolution block and then is input into the 2.5D convolution block;

[0012] The 2.5D convolution block performs 2.5D convolution operations on two inputs respectively and fuses the convolution results into an output.

[0013] In one embodiment, the encoder is provided with 5 encoding layers, which are respectively recorded as the first encoding layer, the second encoding layer, the third encoding layer, the fourth encoding layer, and the fifth encoding layer; the decoder is provided with 4 decoding layers, which are respectively recorded as the first decoding layer, the second decoding layer, the third decoding layer, and the fourth decoding layer;

[0014] The channel and spatial attention module is equipped with five channel and spatial attention mechanisms, which are respectively recorded as the first channel and spatial attention mechanism, the second channel and spatial attention mechanism, the third channel and spatial attention mechanism, the fourth channel and spatial attention mechanism, and the fifth channel and spatial attention mechanism;

[0015] The outputs of the first channel and the spatial attention mechanism, the second channel and the spatial attention mechanism, the third channel and the spatial attention mechanism, the fourth channel and the spatial attention mechanism, and the fifth channel and the spatial attention mechanism are all input into the first decoding layer for multi-scale fusion operation to obtain a first fusion result;

[0016] The outputs of the first channel and the spatial attention mechanism, the second channel and the spatial attention mechanism, the third channel and the spatial attention mechanism, and the first fusion result are all input into the second decoding layer for multi-scale fusion operation to obtain a second fusion result;

[0017] The output of the first channel and the spatial attention mechanism, the output of the second channel and the spatial attention mechanism, and the second fusion result are input into the third decoding layer for multi-scale fusion operation to obtain a third fusion result;

[0018] The output of the first channel and the spatial attention mechanism and the third fusion result are input into the fourth decoding layer for multi-scale fusion operation to obtain the segmentation result.

[0019] In one embodiment, the output of the first channel and the spatial attention mechanism is first subjected to a maximum pooling operation after being input to the decoding layer, and then fused with other inputs of the decoding layer;

[0020] The output of the second channel and the spatial attention mechanism is first subjected to a maximum pooling operation after being input to the decoding layer, and then fused with other inputs of the decoding layer;

[0021] The output of the third channel and the spatial attention mechanism is first subjected to a mixed convolution operation after being input to the decoding layer, and then fused with other inputs of the decoding layer;

[0022] The output of the fourth channel and the spatial attention mechanism is first bilinearly interpolated after being input to the decoding layer, and then fused with other inputs of the decoding layer;

[0023] The output of the fifth channel and the spatial attention mechanism is first bilinearly interpolated after being input to the decoding layer, and then fused with other inputs of the decoding layer;

[0024] The outputs of the first, second, and third decoding layers are first subjected to bilinear interpolation before being input to the next decoding layer, and then fused with other inputs of the decoding layer.

[0025] In a second aspect, a prostate ultrasound segmentation device based on multi-scale feature fusion is provided, comprising:

[0026] A model building module is used to build a prostate ultrasound segmentation model; the prostate ultrasound segmentation model adopts a U-Net network, the U-Net network includes an encoder and a decoder, the encoder includes multiple encoding layers, the decoder includes multiple decoding layers, and the convolution modules in each encoding layer and each decoding layer are replaced with hybrid convolution modules; a channel and spatial attention module is also provided between the encoder and the decoder, the spatial attention module is used to retain useful feature information in the features output by the encoding layer, the useful feature information is input into the decoder for multi-scale feature fusion, and the segmentation result is output;

[0027] A model training module, used for training the prostate ultrasound segmentation model to obtain a trained prostate ultrasound segmentation model;

[0028] The segmentation module is used to input the prostate ultrasound image to be segmented into the trained prostate ultrasound segmentation model to obtain the segmentation result.

[0029] In one embodiment, the hybrid convolution module includes: a first 3D convolution block, a first 2D convolution block, a second 2D convolution block, a second 3D convolution block, and a 2.5D convolution block;

[0030] The input of the hybrid convolution module passes through the first 3D convolution block, the first 2D convolution block, and then is input into the 2.5D convolution block;

[0031] The input of the hybrid convolution module passes through the second 2D convolution block and the second 3D convolution block and then is input into the 2.5D convolution block;

[0032] The 2.5D convolution block performs 2.5D convolution operations on two inputs respectively and fuses the convolution results into an output.

[0033] In one embodiment, the encoder is provided with 5 encoding layers, which are respectively recorded as the first encoding layer, the second encoding layer, the third encoding layer, the fourth encoding layer, and the fifth encoding layer; the decoder is provided with 4 decoding layers, which are respectively recorded as the first decoding layer, the second decoding layer, the third decoding layer, and the fourth decoding layer;

[0034] The channel and spatial attention module is equipped with five channel and spatial attention mechanisms, which are respectively recorded as the first channel and spatial attention mechanism, the second channel and spatial attention mechanism, the third channel and spatial attention mechanism, the fourth channel and spatial attention mechanism, and the fifth channel and spatial attention mechanism;

[0035] The outputs of the first channel and the spatial attention mechanism, the second channel and the spatial attention mechanism, the third channel and the spatial attention mechanism, the fourth channel and the spatial attention mechanism, and the fifth channel and the spatial attention mechanism are all input into the first decoding layer for multi-scale fusion operation to obtain a first fusion result;

[0036] The outputs of the first channel and the spatial attention mechanism, the second channel and the spatial attention mechanism, the third channel and the spatial attention mechanism, and the first fusion result are all input into the second decoding layer for multi-scale fusion operation to obtain a second fusion result;

[0037] The output of the first channel and the spatial attention mechanism, the output of the second channel and the spatial attention mechanism, and the second fusion result are input into the third decoding layer for multi-scale fusion operation to obtain a third fusion result;

[0038] The output of the first channel and the spatial attention mechanism and the third fusion result are input into the fourth decoding layer for multi-scale fusion operation to obtain the segmentation result.

[0039] In one embodiment, the output of the first channel and the spatial attention mechanism is first subjected to a maximum pooling operation after being input to the decoding layer, and then fused with other inputs of the decoding layer;

[0040] The output of the second channel and the spatial attention mechanism is first subjected to a maximum pooling operation after being input to the decoding layer, and then fused with other inputs of the decoding layer;

[0041] The output of the third channel and the spatial attention mechanism is first subjected to a mixed convolution operation after being input to the decoding layer, and then fused with other inputs of the decoding layer;

[0042] The output of the fourth channel and the spatial attention mechanism is first bilinearly interpolated after being input to the decoding layer, and then fused with other inputs of the decoding layer;

[0043] The output of the fifth channel and the spatial attention mechanism is first bilinearly interpolated after being input to the decoding layer, and then fused with other inputs of the decoding layer;

[0044] The outputs of the first, second, and third decoding layers are first subjected to bilinear interpolation before being input to the next decoding layer, and then fused with other inputs of the decoding layer.

[0045] In a third aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the prostate ultrasound segmentation method based on multi-scale feature fusion is implemented.

[0046] In a fourth aspect, a computer program product is provided, characterized in that it includes a computer program / instructions, which, when executed by a processor, implement the above-mentioned prostate ultrasound segmentation method based on multi-scale feature fusion.

[0047] Compared with the prior art, this application has the following beneficial effects:

[0048] 1. This application introduces a multi-scale skip connection method to address the semantic gap caused by the codec in the U-Net model that can only transfer information at the same scale. The model can more effectively extract multi-scale feature information of the target area in the prostate image. At the same time, the channel and space attention mechanism MSA (Multi-Scale-Attention) is added to filter out unimportant feature information in space or channels through the attention mechanism, thereby retaining more critical feature information, and adding the MSA module to each upsampling operation, so that the model can pay attention to the useful feature information in each scale slice, thereby improving the segmentation accuracy.

[0049] 2. Based on the U-Net model, this application replaces all its convolutions with hybrid convolutions. The 2D and 3D CNNs contained in the hybrid convolution can capture local and spatial information at the same time, and each dimensional branch has a feature fusion process. The purpose is to merge these two-dimensional and three-dimensional features so that the entire 2.5D model can capture the representation of each dimension without requiring a large number of parameters and computational costs. It aims to achieve good segmentation accuracy while reducing computational costs, and realize accurate segmentation of prostate images. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The present application may be better understood by referring to the following description given in conjunction with the accompanying drawings, which together with the following detailed description are included in this specification and form a part of this specification. In the drawings:

[0051] Figure 1 A schematic diagram of a prostate ultrasound segmentation model is shown;

[0052] Figure 2 shows a schematic diagram of a hybrid convolutional module FM;

[0053] Figure 3 The structural block diagram of the prostate ultrasound segmentation device based on multi-scale feature fusion is shown. DETAILED DESCRIPTION

[0054] The exemplary embodiments of the present application will be described below in conjunction with the accompanying drawings. For the sake of clarity and conciseness, not all features of the actual embodiments are described in the specification. However, it should be understood that many implementation-specific decisions can be made in the process of developing any such actual embodiments in order to achieve the specific goals of the developer, and these decisions may vary from embodiment to embodiment.

[0055] It is also necessary to explain here that, in order to avoid obscuring the present application due to unnecessary details, only the device structure closely related to the scheme according to the present application is shown in the drawings, while other details that are not closely related to the present application are omitted.

[0056] It should be understood that the present application is not limited to the described implementation forms due to the following description with reference to the accompanying drawings. In this article, where feasible, the embodiments can be combined with each other, features between different embodiments can be replaced or borrowed, and one or more features can be omitted in one embodiment.

[0057] The present application embodiment provides a prostate ultrasound segmentation method based on multi-scale feature fusion, including:

[0058] Step S1, constructing a prostate ultrasound segmentation model; Figure 1 A schematic diagram of the prostate ultrasound segmentation model is shown in Figure 1 The prostate ultrasound segmentation model adopts the U-Net network. The U-Net network includes an encoder and a decoder. The encoder includes multiple encoding layers, and the decoder includes multiple decoding layers. The convolution modules in each encoding layer and each decoding layer are replaced by hybrid convolution modules. A channel and spatial attention module is also set between the encoder and the decoder. The spatial attention module is used to retain the useful feature information in the features output by the encoding layer. The useful feature information is input into the decoder for multi-scale feature fusion, and the segmentation result is output.

[0059] Figure 2 The figure shows a schematic diagram of the hybrid convolution module FM, which is composed of two paths; the purple square contains 2D and 3D convolution blocks, and the blue square contains only a single 3D block (with two 3×3×3 convolution layers inside the 3D block), which is responsible for extracting feature information in the three-dimensional space of the prostate ultrasound image. The green square contains a single 2D convolution block, which contains two 3×3 convolution layers. The green square focuses on extracting local detail features from the two-dimensional slices of the prostate ultrasound image. After each convolution layer, instance normalization and ReLU activation function are followed to enhance the nonlinear representation ability and stability of the model.

[0060] Specifically, the hybrid convolution module includes: a first 3D convolution block, a first 2D convolution block, a second 2D convolution block, a second 3D convolution block and a 2.5D convolution block;

[0061] The input of the hybrid convolution module passes through the first 3D convolution block, the first 2D convolution block, and then is input into the 2.5D convolution block;

[0062] The input of the hybrid convolution module passes through the second 2D convolution block and the second 3D convolution block and then is input into the 2.5D convolution block;

[0063] The 2.5D convolution block performs 2.5D convolution operations on two inputs respectively and fuses the convolution results into an output.

[0064] Here, a hybrid convolution module is used to allow CNNs of different dimensions to extract feature information at their respective levels. In this process, feature maps of different dimensions can complement and enhance information, thereby forming a richer and more comprehensive feature representation. Such a feature representation not only includes local detail information from 2D CNN, but also incorporates spatial contour information from 3D CNN, and combines inter-layer correlation information. Secondly, due to the shallow fusion of multiple CNNs, a large number of new paths are derived, thereby constructing a more complex multi-level structural network. In order to integrate the multi-level structural network, this embodiment adopts a strategy of fusing shallow and deep layers. On each hybrid convolution block (including 2D / 2.5D / 3D CNN), if sparse inter-slice information is over-represented, it means that there is a lack of local feature information in the slice at this time; then the model will choose to pass through the two-dimensional convolution branch to focus on the extraction of intra-slice information. In addition, when it is necessary to obtain relevant spatial information, the three-dimensional convolution in the branch will play its role to perform effective feature extraction and integrate spatial three-dimensional information. Through this hybrid convolution method, the processing strategy can be flexibly adjusted according to different task requirements, thereby improving the overall performance and achieving a comprehensive understanding of the input image.

[0065] Step S2: training the prostate ultrasound segmentation model to obtain a trained prostate ultrasound segmentation model.

[0066] First, data collection is carried out, and cooperation is carried out with medical institutions to obtain the original data of prostate ultrasound images of several patients. The format of prostate TRUS image data is DICOM format. The data of each patient will be placed in a separate folder, and these folders will contain all the prostate TRUS images of the patient, that is, continuous ultrasound image slices. These data are collected by transrectal ultrasound probes at different angles and different times. The prostate TRUS images of patients in DICOM format can be viewed through specific software. Since the data provided by each medical institution are different, the obtained data must be screened as follows: First, those images with clear imaging, obvious prostate contours, and complete prostate shapes can provide accurate and detailed prostate structure and pathological information. Secondly, there are images with unclear prostate TRUS image contours and incomplete glands. Although these images may not provide sufficiently detailed prostate information, they may still contain valuable diagnostic clues, so they are also included in the data set. Finally, some prostate images taken by ultrasound probes from different angles are also included. These images cannot directly reflect the true morphology and pathological characteristics of the prostate. In clinical practice, doctors cannot make corresponding judgments on such images, so these data are discarded.

[0067] Then, the data was preprocessed. Since the acquired data was a prostate transrectal ultrasound image file in DICOM format, it contained a wealth of information, including instrument settings, image acquisition parameters, and patient personal information, etc. However, the purpose of this study was to focus on the image information of the prostate TRUS image itself, and other information was considered redundant. Therefore, the algorithm batch processing method was used to make all data focus only on the entire glandular part of the prostate TRUS image and adjust it to a uniform pixel size. After completing this processing, this article saved the captured prostate TRUS image in JPG format, and a total of 1,800 prostate TRUS continuous slice images were included. It is worth noting that one patient corresponds to multiple continuous TRUS slices and the corresponding.

[0068] Then, the prostate ultrasound segmentation model is trained based on the preprocessed data. The specific training process belongs to the conventional technology in this field and will not be described in detail here.

[0069] Step S3, inputting the prostate ultrasound image to be segmented into the trained prostate ultrasound segmentation model to obtain a segmentation result.

[0070] In one embodiment, see Figure 1 , the encoder is provided with 5 encoding layers, which are respectively recorded as the first encoding layer, the second encoding layer, the third encoding layer, the fourth encoding layer, and the fifth encoding layer; the decoder is provided with 4 decoding layers, which are respectively recorded as the first decoding layer, the second decoding layer, the third decoding layer, and the fourth decoding layer;

[0071] The channel and spatial attention module is equipped with five channel and spatial attention mechanisms MSA (Multi-Scale-Attention), which are respectively recorded as the first channel and spatial attention mechanism, the second channel and spatial attention mechanism, the third channel and spatial attention mechanism, the fourth channel and spatial attention mechanism, and the fifth channel and spatial attention mechanism;

[0072] The outputs of the first channel and the spatial attention mechanism, the second channel and the spatial attention mechanism, the third channel and the spatial attention mechanism, the fourth channel and the spatial attention mechanism, and the fifth channel and the spatial attention mechanism are all input into the first decoding layer for multi-scale fusion operation to obtain a first fusion result;

[0073] The outputs of the first channel and the spatial attention mechanism, the second channel and the spatial attention mechanism, the third channel and the spatial attention mechanism, and the first fusion result are all input into the second decoding layer for multi-scale fusion operation to obtain a second fusion result;

[0074] The output of the first channel and the spatial attention mechanism, the output of the second channel and the spatial attention mechanism, and the second fusion result are input into the third decoding layer for multi-scale fusion operation to obtain a third fusion result;

[0075] The output of the first channel and the spatial attention mechanism and the third fusion result are input into the fourth decoding layer for multi-scale fusion operation to obtain the segmentation result.

[0076] In this embodiment, in order to solve the semantic gap caused by the codec in the U-Net model being able to only transfer information at the same scale, a multi-scale skip connection method is introduced, and the model can more effectively extract multi-scale feature information of the target area in the prostate image. At the same time, a channel and space attention mechanism MSA is added, and the unimportant feature information in the space or channel is filtered out through the attention mechanism, thereby retaining more critical feature information, and the MSA module is added to each upsampling operation, so that the model can pay attention to the useful feature information in each scale slice, thereby improving the segmentation accuracy.

[0077] A channel and spatial cross-attention mechanism is added to each upsampling operation. First, multi-scale features of the input data are extracted through different convolutional layers or pooling layers. For each scale of features, MSA calculates an attention weight. These weights are usually calculated through a lightweight attention network (such as self-attention or channel attention network). This network evaluates the importance of each feature and outputs a weight vector. Next, MSA uses the calculated attention weights to weightedly fuse features of different scales. In order to retain the information of the original input features and prevent the gradient from disappearing, MSA adds the weighted fused features to the original input features through a residual connection. Finally, MSA outputs a feature representation that combines multi-scale information and attention weights.

[0078] In one embodiment, the output of the first channel and the spatial attention mechanism is first subjected to a max pooling operation after being input to the decoding layer, and then fused with other inputs of the decoding layer;

[0079] The output of the second channel and the spatial attention mechanism is first subjected to a maxpooling operation after being input to the decoding layer, and then fused with other inputs of the decoding layer;

[0080] The output of the third channel and the spatial attention mechanism is first subjected to a mixed convolution operation after being input to the decoding layer, and then fused with other inputs of the decoding layer;

[0081] The output of the fourth channel and the spatial attention mechanism is first bilinearly interpolated after being input to the decoding layer, and then fused with other inputs of the decoding layer;

[0082] The output of the fifth channel and the spatial attention mechanism is first bilinearly interpolated after being input to the decoding layer, and then fused with other inputs of the decoding layer;

[0083] The outputs of the first, second, and third decoding layers are first subjected to bilinear interpolation before being input to the next decoding layer, and then fused with other inputs of the decoding layer.

[0084] In order to further verify the effectiveness of the method of this application, this application uses three evaluation indicators: similarity coefficient Dice, intersection over union ratio IoU, and cross entropy loss function Loss to measure and evaluate the method of this application. The method of this application is experimentally compared with multiple segmentation networks, including traditional segmentation models such as U-Net, UNet++, and nnUNet. The experimental results are shown in Table 1 below.

[0085] Table 1 Performance comparison of multiple segmentation networks on the prostate TRUS dataset

[0086]

[0087]

[0088] It can be observed from Table 1 that the method of this application obtains the highest Dice index, which is 4.92% higher than the traditional U-Net. This is mainly due to the effective extraction of feature information of each scale by the multi-scale jump connection in the method of this application, and the keen attention of MSA to important information. Compared with U-Net++, the network structure of this application adopts a simpler and more effective jump connection structure, which improves the Dice index by 2.79%. Compared with the latest segmentation methods, the method of this application also shows advantages.

[0089] Using the same inventive concept as the prostate ultrasound segmentation method based on multi-scale feature fusion, this embodiment also provides a corresponding prostate ultrasound segmentation device based on multi-scale feature fusion. Figure 3 The structure block diagram of the prostate ultrasound segmentation device based on multi-scale feature fusion is shown, including:

[0090] The model building module 31 is used to build a prostate ultrasound segmentation model; the prostate ultrasound segmentation model adopts a U-Net network, the U-Net network includes an encoder and a decoder, the encoder includes multiple encoding layers, the decoder includes multiple decoding layers, and the convolution modules in each encoding layer and each decoding layer are replaced with hybrid convolution modules; a channel and spatial attention module is also provided between the encoder and the decoder, the spatial attention module is used to retain useful feature information in the features output by the encoding layer, the useful feature information is input into the decoder for multi-scale feature fusion, and the segmentation result is output;

[0091] A model training module 32 is used to train the prostate ultrasound segmentation model to obtain a trained prostate ultrasound segmentation model;

[0092] The segmentation module 33 is used to input the prostate ultrasound image to be segmented into the trained prostate ultrasound segmentation model to obtain a segmentation result.

[0093] The prostate ultrasound segmentation device based on multi-scale feature fusion of this embodiment has the same inventive concept as the prostate ultrasound segmentation method based on multi-scale feature fusion mentioned above. Therefore, the specific implementation method of the device can be seen in the implementation example part of the prostate ultrasound segmentation method based on multi-scale feature fusion mentioned above, and its technical effect corresponds to the technical effect of the above method, which will not be repeated here.

[0094] An embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the prostate ultrasound segmentation method based on multi-scale feature fusion is implemented.

[0095] An embodiment of the present application provides a computer program product, including a computer program / instruction. When the computer program / instruction is executed by a processor, the prostate ultrasound segmentation method based on multi-scale feature fusion is implemented.

[0096] In summary, this application has the following technical effects:

[0097] 1. This application introduces a multi-scale skip connection method to address the semantic gap caused by the codec in the U-Net model that can only transfer information at the same scale. The model can more effectively extract multi-scale feature information of the target area in the prostate image. At the same time, the channel and space attention mechanism MSA (Multi-Scale-Attention) is added to filter out unimportant feature information in space or channels through the attention mechanism, thereby retaining more critical feature information, and adding the MSA module to each upsampling operation, so that the model can pay attention to the useful feature information in each scale slice, thereby improving the segmentation accuracy.

[0098] 2. Based on the U-Net model, this application replaces all its convolutions with hybrid convolutions. The 2D and 3D CNNs contained in the hybrid convolution can capture local and spatial information at the same time, and each dimensional branch has a feature fusion process. The purpose is to merge these two-dimensional and three-dimensional features so that the entire 2.5D model can capture the representation of each dimension without requiring a large number of parameters and computational costs. It aims to achieve good segmentation accuracy while reducing computational costs, and realize accurate segmentation of prostate images.

[0099] The above are only various implementations of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A prostate ultrasound segmentation method based on multi-scale feature fusion, characterized in that: include: Constructing a prostate ultrasound segmentation model; the prostate ultrasound segmentation model adopts a U-Net network, the U-Net network includes an encoder and a decoder, the encoder includes multiple encoding layers, the decoder includes multiple decoding layers, and the convolution modules in each encoding layer and each decoding layer are replaced with hybrid convolution modules; a channel and spatial attention module is also provided between the encoder and the decoder, the spatial attention module is used to retain useful feature information in the features output by the encoding layer, the useful feature information is input into the decoder for multi-scale feature fusion, and the segmentation result is output; Training the prostate ultrasound segmentation model to obtain a trained prostate ultrasound segmentation model; The prostate ultrasound image to be segmented is input into the trained prostate ultrasound segmentation model to obtain a segmentation result.

2. The method according to claim 1, characterized in that The hybrid convolution module includes: a first 3D convolution block, a first 2D convolution block, a second 2D convolution block, a second 3D convolution block and a 2.5D convolution block; The input of the hybrid convolution module passes through the first 3D convolution block and the first 2D convolution block and then is input into the 2.5D convolution block; The input of the hybrid convolution module passes through the second 2D convolution block and the second 3D convolution block and then is input into the 2.5D convolution block; The 2.5D convolution block performs 2.5D convolution operations on two inputs respectively, and fuses the convolution results for output.

3. The method according to claim 1, characterized in that The encoder is provided with 5 encoding layers, which are respectively recorded as the first encoding layer, the second encoding layer, the third encoding layer, the fourth encoding layer, and the fifth encoding layer; the decoder is provided with 4 decoding layers, which are respectively recorded as the first decoding layer, the second decoding layer, the third decoding layer, and the fourth decoding layer; The channel and space attention module is provided with 5 channel and space attention mechanisms, which are respectively recorded as the first channel and space attention mechanism, the second channel and space attention mechanism, the third channel and space attention mechanism, the fourth channel and space attention mechanism, and the fifth channel and space attention mechanism; The outputs of the first channel and the spatial attention mechanism, the second channel and the spatial attention mechanism, the third channel and the spatial attention mechanism, the fourth channel and the spatial attention mechanism, and the fifth channel and the spatial attention mechanism are all input into the first decoding layer for multi-scale fusion operation to obtain a first fusion result; The outputs of the first channel and the spatial attention mechanism, the second channel and the spatial attention mechanism, the third channel and the spatial attention mechanism, and the first fusion result are input into the second decoding layer for multi-scale fusion operation to obtain a second fusion result; The outputs of the first channel and the spatial attention mechanism, the second channel and the spatial attention mechanism, and the second fusion result are all input into the third decoding layer for multi-scale fusion operation to obtain a third fusion result; The output of the first channel and the spatial attention mechanism and the third fusion result are input into the fourth decoding layer for multi-scale fusion operation to obtain a segmentation result.

4. The method according to claim 3, characterized in that The output of the first channel and the spatial attention mechanism is first subjected to a maximum pooling operation after being input to the decoding layer, and then fused with other inputs of the decoding layer; The output of the second channel and the spatial attention mechanism is first subjected to a maximum pooling operation after being input to the decoding layer, and then fused with other inputs of the decoding layer; The output of the third channel and the spatial attention mechanism is first subjected to a mixed convolution operation after being input to the decoding layer, and then fused with other inputs of the decoding layer; The output of the fourth channel and the spatial attention mechanism is first subjected to a bilinear interpolation operation after being input into the decoding layer, and then fused with other inputs of the decoding layer; The output of the fifth channel and the spatial attention mechanism is first subjected to a bilinear interpolation operation after being input into the decoding layer, and then fused with other inputs of the decoding layer; After being input into the next decoding layer, the outputs of the first decoding layer, the second decoding layer and the third decoding layer are first subjected to a bilinear interpolation operation and then fused with other inputs of the decoding layer.

5. A prostate ultrasound segmentation device based on multi-scale feature fusion, characterized in that: include: A model building module is used to build a prostate ultrasound segmentation model; the prostate ultrasound segmentation model adopts a U-Net network, the U-Net network includes an encoder and a decoder, the encoder includes multiple encoding layers, the decoder includes multiple decoding layers, and the convolution modules in each encoding layer and each decoding layer are replaced with a hybrid convolution module; a channel and spatial attention module is also provided between the encoder and the decoder, the spatial attention module is used to retain useful feature information in the features output by the encoding layer, the useful feature information is input into the decoder for multi-scale feature fusion, and the segmentation result is output; A model training module, used for training the prostate ultrasound segmentation model to obtain a trained prostate ultrasound segmentation model; The segmentation module is used to input the prostate ultrasound image to be segmented into the trained prostate ultrasound segmentation model to obtain a segmentation result.

6. The device according to claim 5, characterized in that The hybrid convolution module includes: a first 3D convolution block, a first 2D convolution block, a second 2D convolution block, a second 3D convolution block and a 2.5D convolution block; The input of the hybrid convolution module passes through the first 3D convolution block and the first 2D convolution block and then is input into the 2.5D convolution block; The input of the hybrid convolution module passes through the second 2D convolution block and the second 3D convolution block and then is input into the 2.5D convolution block; The 2.5D convolution block performs 2.5D convolution operations on two inputs respectively, and fuses the convolution results for output.

7. The device according to claim 5, characterized in that The encoder is provided with 5 encoding layers, which are respectively recorded as the first encoding layer, the second encoding layer, the third encoding layer, the fourth encoding layer, and the fifth encoding layer; the decoder is provided with 4 decoding layers, which are respectively recorded as the first decoding layer, the second decoding layer, the third decoding layer, and the fourth decoding layer; The channel and space attention module is provided with 5 channel and space attention mechanisms, which are respectively recorded as the first channel and space attention mechanism, the second channel and space attention mechanism, the third channel and space attention mechanism, the fourth channel and space attention mechanism, and the fifth channel and space attention mechanism; The outputs of the first channel and the spatial attention mechanism, the second channel and the spatial attention mechanism, the third channel and the spatial attention mechanism, the fourth channel and the spatial attention mechanism, and the fifth channel and the spatial attention mechanism are all input into the first decoding layer for multi-scale fusion operation to obtain a first fusion result; The outputs of the first channel and the spatial attention mechanism, the second channel and the spatial attention mechanism, the third channel and the spatial attention mechanism, and the first fusion result are input into the second decoding layer for multi-scale fusion operation to obtain a second fusion result; The outputs of the first channel and the spatial attention mechanism, the second channel and the spatial attention mechanism, and the second fusion result are all input into the third decoding layer for multi-scale fusion operation to obtain a third fusion result; The output of the first channel and the spatial attention mechanism and the third fusion result are input into the fourth decoding layer for multi-scale fusion operation to obtain a segmentation result.

8. The device according to claim 7, characterized in that The output of the first channel and the spatial attention mechanism is first subjected to a maximum pooling operation after being input to the decoding layer, and then fused with other inputs of the decoding layer; The output of the second channel and the spatial attention mechanism is first subjected to a maximum pooling operation after being input to the decoding layer, and then fused with other inputs of the decoding layer; The output of the third channel and the spatial attention mechanism is first subjected to a mixed convolution operation after being input to the decoding layer, and then fused with other inputs of the decoding layer; The output of the fourth channel and the spatial attention mechanism is first subjected to a bilinear interpolation operation after being input into the decoding layer, and then fused with other inputs of the decoding layer; The output of the fifth channel and the spatial attention mechanism is first subjected to a bilinear interpolation operation after being input into the decoding layer, and then fused with other inputs of the decoding layer; After being input into the next decoding layer, the outputs of the first decoding layer, the second decoding layer and the third decoding layer are first subjected to a bilinear interpolation operation and then fused with other inputs of the decoding layer.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the prostate ultrasound segmentation method based on multi-scale feature fusion as described in any one of claims 1-4.

10. A computer program product, characterized in that It includes a computer program / instruction, which, when executed by a processor, implements the prostate ultrasound segmentation method based on multi-scale feature fusion as described in any one of claims 1 to 4.

Images