Image segmentation method and device, electronic equipment and storage medium

Abstract

The invention provides an image segmentation method and device, electronic equipment and a storage medium, and relates to the technical field of image processing, and the method comprises the steps: determining a to-be-segmented image; inputting the to-be-segmented image into the segmentation model to obtain a segmentation mask image corresponding to the to-be-segmented image output by the segmentation model; wherein the segmentation model is used for extracting different levels of feature maps in the to-be-segmented image, and generating the segmentation mask map based on the different levels of feature maps; the segmentation mask graph is used for representing different types of areas in the to-be-segmented image; and superposing the segmentation mask graph on the to-be-segmented image in a color coding form to obtain different types of regions in the to-be-segmented image. According to the invention, the segmentation model is utilized to extract the feature patterns of different levels, the segmentation mask pattern is generated based on the feature patterns of different levels, and the to-be-segmented image is segmented according to the segmentation mask pattern, so that the accuracy of complex image segmentation is improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image segmentation method, device, electronic equipment and storage medium. Background Art

[0002] With the rapid development of deep learning technology, deep learning-based image segmentation methods have made significant progress in natural image and medical image segmentation. However, with the increasing complexity of the image to be segmented and the interference of noise and artifacts, the difficulty of segmentation increases. Without a large amount of high-quality labeled data, the segmentation model based on deep learning will have poor segmentation effect and low accuracy. Summary of the invention

[0003] The present invention provides an image segmentation method, device, electronic device and storage medium, which are used to solve the defects in the prior art that due to the increased complexity of the image to be segmented and the interference of noise and artifacts, the deep learning has poor segmentation effect and low accuracy in segmenting the image in the absence of high-quality labeled data. The method realizes the use of a segmentation model to extract feature maps of different levels, and generates a segmentation mask map based on the feature maps of different levels. The image to be segmented is segmented according to the segmentation mask map, thereby improving the accuracy of complex image segmentation.

[0004] The present invention provides an image segmentation method, comprising: Determine the image to be segmented; Inputting the image to be segmented into a segmentation model, obtaining a segmentation mask map corresponding to the image to be segmented output by the segmentation model; wherein the segmentation model is used to extract feature maps of different levels in the image to be segmented, and generate the segmentation mask map based on the feature maps of different levels; the segmentation mask map is used to characterize different categories of regions in the image to be segmented; The segmentation mask image is superimposed on the image to be segmented in a color-coded form to obtain regions of different categories in the image to be segmented.

[0005] According to an image segmentation method provided by the present invention, the segmentation model includes an encoder and a decoder, and the image to be segmented is input into the segmentation model to obtain a segmentation mask map corresponding to the image to be segmented output by the segmentation model, including: inputting the image to be segmented into the encoder to obtain a first feature map output by the encoder; the encoder is used to perform a depth-separable convolution operation and downsampling on the image to be segmented; copying and cropping the first feature map to obtain a second feature map; inputting the two feature maps into the decoder for sampling processing to obtain a third feature map; splicing the second feature map and the third feature map to obtain a fourth feature map; inputting the fourth feature map into the decoder to obtain a segmentation mask map output by the decoder; the decoder is used to perform a multi-scale convolution attention operation and upsampling on the fourth feature map.

[0006] According to an image segmentation method provided by the present invention, the encoder includes a sub-network block, a convolution block and a downsampling block, and the step of inputting the image to be segmented into the encoder to obtain a first feature map output by the encoder includes: inputting the image to be segmented into the sub-network block to obtain a fifth feature map output by the sub-network block; the sub-network block is used to extract features of the image to be segmented using a deep convolution layer and a point-by-point convolution layer; the fifth feature map is input into the convolution block to obtain a sixth feature map output by the convolution block; the convolution block is used to extract features of the fifth feature map using standard convolution; the sixth feature map is input into the downsampling block to obtain a first feature map output by the downsampling block.

[0007] According to an image segmentation method provided by the present invention, the decoder includes: a multi-scale convolutional attention module and an upsampling convolution block, and the fourth feature map is input into the decoder to obtain the segmentation mask map output by the decoder, including: inputting the fourth feature map into the multi-scale convolutional attention module to obtain the seventh feature map output by the multi-scale convolutional attention module; the multi-scale convolutional attention module is used to perform multi-scale feature extraction on the fourth feature map; the seventh feature map is input into the upsampling convolution block to obtain the segmentation mask map output by the upsampling convolution block.

[0008] According to an image segmentation method provided by the present invention, the multi-scale convolutional attention module includes: a channel attention block, a spatial attention block and a multi-scale convolutional block; the fourth feature map is input into the multi-scale convolutional attention module to obtain the seventh feature map output by the multi-scale convolutional attention module, including: inputting the fourth feature map into the channel attention block to obtain the channel attention map output by the channel attention block; the channel attention map is a feature map after the channels of the fourth feature map are weighted; the channel attention is input into the spatial attention block to obtain the spatial attention map output by the spatial attention block; the spatial attention map is a feature map after each position in the channel attention map is weighted; the spatial attention map is input into the multi-scale convolutional block to obtain the seventh feature map output by the multi-scale convolutional block.

[0009] According to an image segmentation method provided by the present invention, the multi-scale convolution block includes: a first point-by-point convolution layer, a first batch normalization layer, an activation layer, a second point-by-point convolution layer and a second batch normalization layer, and the spatial attention map is input into the multi-scale convolution block to obtain the seventh feature map output by the multi-scale convolution block, including: inputting the spatial attention map into the first point-by-point convolution layer to obtain an eighth feature map; the first point-by-point convolution layer is used to increase the dimension of the number of channels of the spatial attention map; the eighth feature map is sequentially input into the first batch normalization layer and the activation layer to obtain a ninth feature map; the channels in the ninth feature map are shuffled to obtain a ninth feature map after merging the channels; the ninth feature map after merging the channels is input into the second point-by-point convolution layer to obtain a tenth feature map, and the second point-by-point convolution layer is used to reduce the dimension of the number of channels after the ninth feature map after merging the channels; the tenth feature is input into the second batch normalization layer to obtain the seventh feature map.

[0010] According to an image segmentation method provided by the present invention, determining the image to be segmented includes: obtaining at least one frame of initial image to be segmented from a video to be segmented; performing edge detection on the initial image to be segmented to obtain a first image to be segmented; performing edge connection on the first image to be segmented to obtain a second image to be segmented; and performing edge repair on the second image to be segmented to obtain the image to be segmented.

[0011] The present invention also provides an image segmentation device, comprising the following modules: A determination module, used for determining an image to be segmented; A segmentation module, used for inputting the image to be segmented into a segmentation model, and obtaining a segmentation mask map corresponding to the image to be segmented output by the segmentation model; wherein the segmentation model is used for extracting feature maps of different levels in the image to be segmented, and generating the segmentation mask map based on the feature maps of different levels; the segmentation mask map is used for characterizing regions of different categories in the image to be segmented; The encoding module is used to superimpose the segmentation mask image on the image to be segmented in the form of color coding to obtain regions of different categories in the image to be segmented.

[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, any of the above-mentioned image segmentation methods is implemented.

[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the image segmentation method described in any one of the above is implemented.

[0014] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the image segmentation method described above is implemented.

[0015] The image segmentation method, device, electronic device and storage medium provided by the present invention extract feature maps of different levels in the image to be segmented by using a segmentation model to obtain a multi-scale feature map, then generate a segmentation mask map based on the multi-scale feature map, and finally, superimpose the segmentation mask map on the image to be segmented in the form of color coding to obtain different categories of regions in the image to be segmented. In this way, by extracting feature maps of different levels, more high-level semantic features and detail features of the segmented image can be fully and deeply understood without the need for a large amount of high-quality annotated data, so as to accurately generate a segmentation mask map based on more high-level semantic features and detail features, further improving the accuracy of the segmentation of the image to be segmented, and at the same time, color coding enhances the visualization effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 It is a flowchart of the image segmentation method provided by the present invention.

[0018] Figure 2 It is a structural schematic diagram of the image segmentation device provided by the present invention.

[0019] Figure 3 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] It should be noted that the image segmentation method provided by the present invention is applicable to various fields, such as medical images, natural images, etc. Among them, musculoskeletal ultrasound imaging has been widely used in modern medical diagnosis, and can provide high-resolution images of muscles, bones and soft tissues in real time, with the significant advantages of non-invasiveness and convenience. Musculoskeletal ultrasound imaging plays a vital role in the diagnosis of musculoskeletal diseases, evaluation of treatment effects, and surgical guidance.

[0022] However, the field of ultrasound examination in my country faces many challenges. First of all, it is difficult to train qualified ultrasound doctors. Doctors need to go through a lot of training and learning before they can master the skills to correctly segment ultrasound image areas. This is not only difficult to learn, but also has a long learning cycle. In addition, there is a serious shortage of ultrasound doctors in my country. According to the Statistical Yearbook of the National Health and Family Planning Commission, there is a shortage of at least 100,000 registered ultrasound doctors in my country. Faced with such a high demand for musculoskeletal diagnosis, how to help ultrasound doctors realize the automated segmentation of musculoskeletal ultrasound images has become the key to solving this contradiction. This will help to quickly train a large number of professionals who can identify musculoskeletal ultrasound images in a short period of time, greatly alleviating the current shortage of ultrasound doctors.

[0023] With the rapid development of deep learning technology, deep learning-based image segmentation methods have made significant progress in natural image and medical image segmentation. These methods can assist doctors in automatically segmenting various tissues in musculoskeletal ultrasound images by automatically learning and extracting features of musculoskeletal ultrasound images, effectively solving the difficulties faced by ultrasound doctors in image recognition. However, there are two main problems with the segmentation of musculoskeletal ultrasound images: (1) The lack of high-quality annotated data. The annotation of these data must be completed by professional ultrasound doctors, which is not only labor-intensive but also extremely difficult. (2) The complexity of medical image segmentation. Medical images are often interfered by noise and artifacts, and there are large differences between individuals, which increases the difficulty of segmentation, resulting in poor segmentation effect and low accuracy of deep learning-based segmentation models. In addition, accurate deep learning segmentation algorithms are usually highly complex, and the training and inference processes require a lot of computing resources to support them.

[0024] Based on the above problems, the present invention proposes an image segmentation method, which can comprehensively and deeply understand more high-level semantic features and detail features of the segmented image by extracting feature maps at different levels without the need for a large amount of high-quality annotated data, thereby accurately generating a segmentation mask map based on more high-level semantic features and detail features, further improving the accuracy of segmentation of the image to be segmented, and at the same time, color coding enhances the visualization effect.

[0025] Combine the following Figure 1 The image segmentation method of the present invention is described. The execution subject of the method can be an electronic device, or it can be an image segmentation method set in the electronic device. The image segmentation method can be implemented by software, hardware or a combination of the two.

[0026] Figure 1 It is a flow chart of the image segmentation method provided by the present invention, such as Figure 1 As shown, the method includes the following: Step 101: Determine an image to be segmented.

[0027] Here, the image to be segmented can be any suitable image, such as muscle and bone ultrasound images, animal images, flower and grass images, etc., wherein the image to be segmented can be a frame image obtained from a video; it can also be an image downloaded from a search engine; it can also be a partial area image captured from a large image, etc.

[0028] Here, the image to be segmented may be an unprocessed image or an image that has been preprocessed. The preprocessing may include edge detection, edge connection, edge repair, and the like.

[0029] Exemplarily, determining the image to be segmented includes: At least one frame of initial image to be segmented is obtained from the video to be segmented; edge detection is performed on the initial image to be segmented to obtain a first image to be segmented; edge connection is performed on the first image to be segmented to obtain a second image to be segmented; and edge repair is performed on the second image to be segmented to obtain the image to be segmented.

[0030] Here, the video to be segmented may be a musculoskeletal ultrasound video.

[0031] Edge detection can extract important structural information from an image. Here, edge detection methods include but are not limited to Laplacian operator, Canny algorithm, Sobel algorithm, etc.

[0032] Exemplarily, the Laplacian operator is used to perform edge detection on an ultrasound medical image. The Laplacian operator is an edge detection method based on second-order derivatives, which is suitable for detecting fine edges in ultrasound videos. It can enhance high-frequency components in ultrasound images, thereby highlighting the edge portion of the image and effectively detecting fine edge details. The steps of edge detection using the Laplacian operator include: 1) Convert the ultrasound image from a color or multi-channel image to a single-channel grayscale image. The conversion method can be to use a weighted average method to convert the red, green, and blue (RGB) channels of the color image into a grayscale image. The weighted conversion coefficients of the RGB three channels are 0.299, 0.587, and 0.114, respectively.

[0033] Exemplarily, the grayscale processing is as follows: (1) in, Represents the gray value, Indicates red, Indicates green, Indicates blue.

[0034] 2) Calculate the Laplacian of the image and perform threshold processing to obtain a binary edge image. Use the Laplacian operator to calculate the second-order derivative of the grayscale image.

[0035] Exemplarily, the Laplacian is calculated as follows: (2) in, represents the second-order mentor, (x, y) represents the pixel coordinates, Indicates the derivation of the pixel X value. Indicates the derivation of the y value of the pixel.

[0036] Exemplarily, the threshold processing is as follows formula (3): (3) Among them, Edge(x,y) represents the edge pixel point. If the second-order derivative is greater than T, the pixel value is 255, otherwise, the pixel value is 0.

[0037] 3) Perform non-maximum suppression on the edge image to remove non-edge points, wherein the suppression method can be to traverse the image along the gradient direction, and for each pixel point, if its gradient amplitude is not the local maximum in this direction, set it to 0 to suppress the non-edge points.

[0038] It should be noted that grayscale processing helps to simplify calculations and makes subsequent edge detection of various tissues more accurate; binary edge images can enhance edge information; and non-maximum suppression can retain the most significant edges.

[0039] Edge connection is to connect the detected edge points into continuous edge lines. Here, the edge connection method includes but is not limited to Hough transform, hysteresis threshold method, boundary tracking algorithm, etc.

[0040] Exemplarily, Hough transform is used for edge connection. Hough transform is a method of converting points in image space to parameter space, and detecting straight lines or curves by finding the maximum value in parameter space. In ultrasound image processing, Hough transform can effectively connect scattered edge points and generate complete edge contours, providing more continuous and clear edge information for subsequent processing. The specific steps are as follows: 1) Perform Hough transform on the binary image after edge detection to extract the straight lines and curves in the image, map each edge point in the image space to the parameter space, and detect the straight lines and curves by finding the local maximum in the parameter space. When using the standard Hough transform, the parameter ρ is set to 1 pixel, θ is set to np.pi / 180 radians, and the threshold is set to the minimum number of votes for detecting a straight line; 2) According to the results of the Hough transform, the scattered edge points are connected into continuous edge lines to generate a complete edge contour. According to the results of the Hough transform, the detected straight line segments are connected into continuous edge lines. For complex edge situations in ultrasound images, the probabilistic Hough transform is used to improve detection accuracy and efficiency. When using the probabilistic Hough transform, the parameter ρ is set to 1 pixel, θ is set to np.π / 180 radians, the threshold is set to 50, the minimum line length (minLineLength) is set to 30 pixels, and the maximum line length (maxLineGap) is set to 10 pixels.

[0041] Edge repair is to repair the edges of the segmented image. Edge repair methods include but are not limited to: edge connection (Edge-Connection), sharpening filter, etc. The Edge-Connection algorithm is used to repair the edges in the ultrasound image. The Edge-Connection algorithm can maintain the continuity of the edge during the image repair and filling process, thereby improving the visual quality of the repaired image. This is particularly important for artifacts and noise in ultrasound images, and helps to ensure the accuracy and stability of edge detection. The specific steps include: 1) Using the edge contour generated by Hough transform as a reference, the broken edges in the ultrasound image are connected; 2) Linear interpolation is performed on the connected edges to fill the missing edge parts and ensure the continuity of the edges. During the interpolation process, the interpolation step size and algorithm parameters are set according to the distance and distribution of the edge points. 3) The repaired edges are smoothed to reduce the impact of artifacts and noise, maintain the overall texture and structure of the image, and the repaired edges are smoothed using a Gaussian filter. The kernel size of the Gaussian filter is set to (5, 5) and the standard deviation is set to 0.

[0042] In the embodiment of the present invention, a series of operations such as edge detection, edge connection and edge repair can effectively eliminate artifacts and noise in ultrasound images, maintain the texture and structure of the image, and provide a clearer and more accurate image basis for subsequent feature extraction and segmentation.

[0043] Step 102: input the image to be segmented into a segmentation model to obtain a segmentation mask image corresponding to the image to be segmented output by the segmentation model; The segmentation model is used to extract feature maps of different levels in the image to be segmented, and generate the segmentation mask map based on the feature maps of different levels; The segmentation mask image is used to represent regions of different categories in the image to be segmented.

[0044] Here, the segmentation model may be a Unet model including an encoder and a decoder, or other models. Here, the segmentation model may be obtained by training with multiple images to be segmented; or it may be a model downloaded from a server.

[0045] Here, the segmentation mask image is a binary image of the same size as the image to be segmented. Different categories in the segmentation mask image are represented by different values. For example, if the image to be segmented includes cats, dogs, and background, the segmentation mask image uses 0 to represent the background area, 1 to represent the cat area, and 2 to represent the dog area.

[0046] It should be noted that features at different levels represent features at different depths. As the depth increases, high-level semantic features and fine-grained features can be obtained. The high-level semantic features are then mapped back to the high resolution of the image to be segmented to generate a pixel-by-pixel segmentation mask. The boundary information and corresponding category information of a specific area in the image to be segmented can be determined based on the segmentation mask.

[0047] Step 103: superimpose the segmentation mask image on the image to be segmented in a color-coded form to obtain regions of different categories in the image to be segmented.

[0048] It should be noted that different categories are represented by different colors.

[0049] Here, the color coding method can be RGB color space or Hue Saturation Lightness (HSL) encoding, and different categories of organizations use predefined colors.

[0050] Exemplarily, the image to be segmented is a muscle-bone ultrasound image, and the segmentation result is superimposed on the original ultrasound image in a color-coded manner, different categories are marked with different colors, and the name corresponding to each tissue is displayed in the center of each tissue area.

[0051] In an embodiment of the present invention, a segmentation model is used to extract feature maps of different levels in the image to be segmented, a multi-scale feature map is obtained, and then a segmentation mask map is generated according to the multi-scale feature map. Finally, the segmentation mask map is superimposed on the image to be segmented in the form of color coding to obtain regions of different categories in the image to be segmented. In this way, by extracting feature maps of different levels, more high-level semantic features and detail features of the segmented image can be fully and deeply understood without the need for a large amount of high-quality annotated data, so that the segmentation mask map accurately generated according to more high-level semantic features and detail features can further improve the accuracy of the segmentation of the image to be segmented. At the same time, color coding enhances the visualization effect.

[0052] Furthermore, the segmentation model includes an encoder and a decoder, and the image to be segmented is input into the segmentation model to obtain a segmentation mask map corresponding to the image to be segmented output by the segmentation model, including: inputting the image to be segmented into the encoder to obtain a first feature map output by the encoder; the encoder is used to perform a depth-separable convolution operation and downsampling on the image to be segmented; copying and cropping the first feature map to obtain a second feature map; inputting the two feature maps into the decoder for sampling processing to obtain a third feature map; splicing the second feature map and the third feature map to obtain a fourth feature map; inputting the fourth feature map into the decoder to obtain a segmentation mask map output by the decoder; the decoder is used to perform a multi-scale convolution attention operation and upsampling on the fourth feature map.

[0053] Here, the description of the encoder and decoder is based on the processing process that the encoder and decoder each include one level. In the present invention, the number of levels in the encoder and decoder can be determined according to actual conditions, for example, the encoder and decoder for musculoskeletal ultrasound images can be set to 4 levels respectively.

[0054] It should be noted that the networks in the encoder and decoder are corresponding. For example, the encoder contains 4 encoding levels, and each encoding level extracts and downsamples the feature map output by the previous encoding level in turn. The decoder also has 4 corresponding decoding levels. Each decoding level includes feature extraction operations and upsampling operations. The first decoding level extracts and upsampling operations on the fused features in the encoder, and other decoding levels extract and upsample the feature maps output by the previous decoding level in turn, and finally obtains the segmentation mask map.

[0055] Here, the separable convolution operation includes depth-wise convolution and point-wise convolution.

[0056] In an embodiment of the present invention, in the encoder network, a deep convolution with a large kernel is introduced to replace the ordinary convolution to fully mix the distant space and position information. In the decoder network, by introducing a new efficient multi-scale convolution attention module and upsampling convolution block, multi-scale features are fused, which can fully and deeply understand more high-level semantic features and detail features of the segmented image, thereby accurately generating a segmentation mask map based on more high-level semantic features and detail features, further improving the accuracy of the segmentation of the image to be segmented.

[0057] Further, the encoder includes a sub-network block, a convolution block and a downsampling block, The step of inputting the image to be segmented into the encoder to obtain a first feature map output by the encoder includes: Inputting the image to be segmented into the sub-network block to obtain a fifth feature map output by the sub-network block; the sub-network block is used to extract features of the image to be segmented using a deep convolution layer and a point-by-point convolution layer; Inputting the fifth feature map into the convolution block to obtain a sixth feature map output by the convolution block; the convolution block is used to extract features from the fifth feature map using standard convolution; The sixth feature map is input into the down-sampling block to obtain a first feature map output by the down-sampling block.

[0058] Exemplarily, the encoder is composed of at least one level, each level is composed of a sub-network block, a standard convolution block and a downsampling operation, and the sub-network block includes a depth-separable convolution layer, an activation layer, a point-by-point convolution layer and a batch normalization (BN) layer. Among them, the activation layer can use any suitable activation function, such as GELU activation function, RELU activation function, Sigmd activation function, etc.

[0059] Here, the downsampling block is used to reduce the resolution of the image, hence, the encoder outputs low-resolution feature maps.

[0060] In an embodiment of the present invention, by performing deep convolution layers and point-by-point convolution layers, standard convolutions and downsampling on the image to be segmented, high-level semantic features and fine-grained features are obtained, thereby improving the accuracy of image segmentation. At the same time, consistency with the skip connection of the top layer can be maintained, and it is ensured that important features are not lost during the encoding process.

[0061] Furthermore, the decoder comprises: a multi-scale convolutional attention module and an upsampling convolutional block, The step of inputting the fourth feature map into the decoder to obtain a segmentation mask map output by the decoder comprises: Inputting the fourth feature map into the multi-scale convolutional attention module to obtain a seventh feature map output by the multi-scale convolutional attention module; the multi-scale convolutional attention module is used to perform multi-scale feature extraction on the fourth feature map; The seventh feature map is input into the upsampling convolution block to obtain a segmentation mask map output by the upsampling convolution block.

[0062] Here, multi-scale convolutional attention modules are used to refine the features, where the number of multi-scale convolutional attention modules is the same as the number of levels in the encoder.

[0063] Here, the upsampling convolution block can be understood as a deconvolution operation, which is used to enlarge the size of the image, so the decoder outputs a high-resolution image.

[0064] In an embodiment of the present invention, a multi-scale convolutional attention module is used to refine the feature map, and an upsampling convolution block is used to upgrade the refined feature map to generate a final segmentation output. In this way, the boundaries of each tissue corresponding to the image to be segmented and the categories to which the boundary areas belong can be accurately and clearly segmented.

[0065] Further, the multi-scale convolutional attention module includes: a channel attention block, a spatial attention block and a multi-scale convolution block; The step of inputting the fourth feature map into the multi-scale convolutional attention module to obtain a seventh feature map output by the multi-scale convolutional attention module comprises: Inputting the fourth feature map into the channel attention block to obtain a channel attention map output by the channel attention block; the channel attention map is a feature map after the channels of the fourth feature map are weighted; Input the channel attention into the spatial attention block to obtain a spatial attention map output by the spatial attention block; the spatial attention map is a feature map after weighting each position in the channel attention map; The spatial attention map is input into the multi-scale convolution block to obtain a seventh feature map output by the multi-scale convolution block.

[0066] Here, the channel attention block is used to assign a weight to each channel. The specific steps include: feature conversion; calculating the channel attention weight; multiplying the fourth feature map by the attention weight to obtain the channel attention map.

[0067] Here, the spatial attention block is used to assign a weight to each spatial position. The specific steps include: feature extraction, calculating the spatial attention weight, multiplying the channel attention map with the spatial attention weight to obtain the spatial attention map.

[0068] Here, the multi-scale convolution block is used to perform convolution operations on the same input feature map using multiple convolution kernels of different sizes. The specific steps include: receiving the feature map from the previous layer as input; using multiple convolution kernels of different sizes to perform convolution operations on the input feature map. These convolution kernels can be applied to the input feature map in parallel to generate multiple feature maps of different scales; multiple feature maps of different scales are fused to integrate information of multiple scales; and a fused feature map is generated.

[0069] Exemplarily, the operation process of the Multi-scale Convolutional Attention Module (MSCAM) module may include the following formula (4): (4) Where X is the input tensor, represents the channel attention block (CAB) calculation, Represents the spatial attention block (SAB) calculation, Represents the multi-scale convolution block (MSCB) calculation.

[0070] For example, the Efficient Upsampling Convolutional Block (EUCB) operation includes formula (5): EUCB(x) = C1×1(ReLU (BN (DWC(Up(x))))) (5) Among them, Up(x) represents upsampling, DWC represents depthwise convolution (DWC), ReLU activation function, C 1×1 Indicates convolution using a 1 × 1 convolution kernel.

[0071] In an embodiment of the present invention, MSCAM can perform deep convolution at multiple scales to refine the feature map generated by the visual encoder, while emphasizing relevant channels and capturing local context information, and suppressing irrelevant areas to capture multi-scale salient features.

[0072] Furthermore, the multi-scale convolution block includes: a first point-by-point convolution layer, a first batch normalization layer, an activation layer, a second point-by-point convolution layer, and a second batch normalization layer. Inputting the spatial attention map into the multi-scale convolution block to obtain a seventh feature map output by the multi-scale convolution block includes: Inputting the spatial attention map into the first point-by-point convolutional layer to obtain an eighth feature map; the first point-by-point convolutional layer is used to increase the dimension of the number of channels of the spatial attention map; Inputting the eighth feature map into the first batch normalization layer and the activation layer in sequence to obtain a ninth feature map; Performing a shuffling operation on the channels in the ninth feature map to obtain a ninth feature map after merging the channels; Inputting the ninth feature map after the merged channels into the second point-by-point convolutional layer to obtain a tenth feature map, and the second point-by-point convolutional layer is used to reduce the number of channels after the ninth feature map after the merged channels; The tenth feature is input into the second batch normalization to obtain the seventh feature map.

[0073] Here, a batch normalization layer is used to standardize the data to mitigate internal covariate shift.

[0074] For example, the operation process of the multi-scale convolution block is as follows: (6) in, Represents the first point-by-point convolution, the size of the convolution kernel is 1×1, Indicates batch normalization. Represents an activation layer whose activation function is ReLU6. Represents multi-scale depth convolution for merging channels, CS Represents a channel shuffle operation, represents the second point-wise convolution.

[0075] further, The operation process is as follows formula (7): (7) in, Indicates the deep convolution operation on the feature map. represents the size of the convolution kernel. The convolution operation includes the following formula (8): (8) in, Represents a depthwise convolution with kernel size ks.

[0076] It should be noted that MSCB is an efficient multi-scale convolution block used to enhance the features generated by the cascade expansion path. In MSCB, the design of the Inverted Residual Block (IRB) of MobileNetV2 is followed. However, unlike IRB, MSCB performs deep convolutions at multiple scales and uses channel shuffling to reorganize channels across groups. More specifically, in MSCB, point-by-point convolution is first used. Expand the number of channels (i.e. expansion factor = 2); then use a batch normalization layer and ReLU6 activation layer R6 ; Then, use multi-scale deep convolution MSDC To capture multi-scale and multi-resolution context; Since the deep convolution ignores the relationship between channels, the present invention uses a channel shuffle operation to merge the relationship between channels; Use another point-by-point convolution and To transform back to the original channels, the dependencies between channels are encoded.

[0077] In the embodiment of the present invention, the order of channels is rearranged to promote cross-channel information exchange, thereby improving the representation ability and performance of the model and encoding the dependency relationship between channels.

[0078] It should be noted that after the model architecture is designed, the labeled data is used for training and evaluation. Finally, the model parameters are solidified for image segmentation prediction.

[0079] The image segmentation method proposed in the present invention also has the following advantages: high-precision segmentation: through the combination of a full convolutional network and a multi-scale convolutional network, the boundaries of each tissue corresponding to each frame of ultrasound image in the musculoskeletal ultrasound video and the category to which the boundary area belongs can be accurately and clearly segmented; enhanced visualization effect: the segmentation results are superimposed on the original ultrasound image in a color-coded manner, and the name corresponding to each tissue is displayed in the center of each tissue area, so that the segmentation results are intuitive and easy to read, and easy to understand quickly; fast processing: fast and accurate segmentation of images (such as musculoskeletal ultrasound images), to assist doctors in diagnosis and treatment, improve diagnostic efficiency, and reduce the workload of doctors.

[0080] 4) Wide application: The present invention can be applied to a variety of musculoskeletal ultrasound examinations, especially those scenarios that require high-precision image segmentation, such as muscle, bone and soft tissue diagnosis, treatment effect evaluation and surgical guidance.

[0081] The image segmentation device provided by the present invention is described below. The image segmentation device described below and the image segmentation method described above can be referred to each other.

[0082] Figure 2 : is a schematic diagram of the structure of the image segmentation device provided by the present invention, such as Figure 2 As shown, the image segmentation device 200 includes: A determination module 201 is used to determine an image to be segmented; The segmentation module 202 is used to input the image to be segmented into a segmentation model to obtain a segmentation mask map corresponding to the image to be segmented output by the segmentation model; wherein the segmentation model is used to extract feature maps of different levels in the image to be segmented, and generate the segmentation mask map based on the feature maps of different levels; the segmentation mask map is used to characterize different categories of regions in the image to be segmented; The encoding module 203 is used to superimpose the segmentation mask image on the image to be segmented in the form of color coding to obtain regions of different categories in the image to be segmented.

[0083] In some embodiments, the segmentation model includes an encoder and a decoder, and the segmentation module 202 is also specifically used to: input the image to be segmented into the encoder to obtain a first feature map output by the encoder; the encoder is used to perform a depth-separable convolution operation and downsampling on the image to be segmented; copy and crop the first feature map to obtain a second feature map; input the two feature maps into the decoder for sampling processing to obtain a third feature map; splice the second feature map and the third feature map to obtain a fourth feature map; input the fourth feature map into the decoder to obtain a segmentation mask map output by the decoder; the decoder is used to perform a multi-scale convolution attention operation and upsampling on the fourth feature map.

[0084] In some embodiments, the encoder includes a sub-network block, a convolution block and a downsampling block, and the segmentation module 202 is further specifically used to: input the image to be segmented into the sub-network block to obtain a fifth feature map output by the sub-network block; the sub-network block is used to extract features of the image to be segmented using a deep convolution layer and a point-by-point convolution layer; input the fifth feature map into the convolution block to obtain a sixth feature map output by the convolution block; the convolution block is used to extract features of the fifth feature map using standard convolution; and input the sixth feature map into the downsampling block to obtain a first feature map output by the downsampling block.

[0085] In some embodiments, the decoder includes: a multi-scale convolutional attention module and an upsampling convolution block, a segmentation module 202, and is also specifically used to: input the fourth feature map into the multi-scale convolutional attention module to obtain a seventh feature map output by the multi-scale convolutional attention module; the multi-scale convolutional attention module is used to perform multi-scale feature extraction on the fourth feature map; input the seventh feature map into the upsampling convolution block to obtain a segmentation mask map output by the upsampling convolution block.

[0086] In some embodiments, the multi-scale convolutional attention module includes: a channel attention block, a spatial attention block and a multi-scale convolution block; the segmentation module 202 is also specifically used to: input the fourth feature map into the channel attention block to obtain a channel attention map output by the channel attention block; the channel attention map is a feature map after the channels of the fourth feature map are weighted; the channel attention is input into the spatial attention block to obtain a spatial attention map output by the spatial attention block; the spatial attention map is a feature map after each position in the channel attention map is weighted; the spatial attention map is input into the multi-scale convolution block to obtain a seventh feature map output by the multi-scale convolution block.

[0087] In some embodiments, the multi-scale convolution block includes: a first point-by-point convolution layer, a first batch normalization layer, an activation layer, a second point-by-point convolution layer and a second batch normalization layer. The segmentation module 202 is also specifically used to: input the spatial attention map into the multi-scale convolution block to obtain the seventh feature map output by the multi-scale convolution block, including: inputting the spatial attention map into the first point-by-point convolution layer to obtain an eighth feature map; the first point-by-point convolution layer is used to increase the number of channels of the spatial attention map; inputting the eighth feature map into the first batch normalization layer and the activation layer in sequence to obtain a ninth feature map; shuffling the channels in the ninth feature map to obtain a ninth feature map after merging the channels; inputting the ninth feature map after merging the channels into the second point-by-point convolution layer to obtain a tenth feature map, and the second point-by-point convolution layer is used to reduce the number of channels after the ninth feature map after merging the channels; inputting the tenth feature into the second batch normalization layer to obtain the seventh feature map.

[0088] In some embodiments, the determination module 201 is further specifically used to obtain at least one frame of initial image to be segmented from the video to be segmented; perform edge detection on the initial image to be segmented to obtain a first image to be segmented; perform edge connection on the first image to be segmented to obtain a second image to be segmented; perform edge repair on the second image to be segmented to obtain the image to be segmented.

[0089] Figure 3 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 3 As shown, the electronic device may include: a processor 310, a communication interface 320, a memory 330 and a communication bus 340, wherein the processor 310, the communication interface 320 and the memory 330 communicate with each other through the communication bus 340. The processor 310 may call the logic instructions in the memory 330 to execute the image segmentation method, which includes: determining an image to be segmented; inputting the image to be segmented into a segmentation model to obtain a segmentation mask map corresponding to the image to be segmented output by the segmentation model; wherein the segmentation model is used to extract feature maps of different levels in the image to be segmented, and generate the segmentation mask map based on the feature maps of different levels; the segmentation mask map is used to characterize different categories of regions in the image to be segmented; and the segmentation mask map is superimposed on the image to be segmented in the form of color coding to obtain different categories of regions in the image to be segmented.

[0090] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0091] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the image segmentation method provided by the above-mentioned methods, and the method includes: determining an image to be segmented; inputting the image to be segmented into a segmentation model to obtain a segmentation mask map corresponding to the image to be segmented output by the segmentation model; wherein the segmentation model is used to extract feature maps of different levels in the image to be segmented, and generate the segmentation mask map based on the feature maps of different levels; the segmentation mask map is used to characterize different categories of areas in the image to be segmented; and superimposing the segmentation mask map on the image to be segmented in the form of color coding to obtain different categories of areas in the image to be segmented.

[0092] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the image segmentation method provided by the above-mentioned methods, the method comprising: determining an image to be segmented; inputting the image to be segmented into a segmentation model to obtain a segmentation mask map corresponding to the image to be segmented output by the segmentation model; wherein the segmentation model is used to extract feature maps of different levels in the image to be segmented, and generate the segmentation mask map based on the feature maps of different levels; the segmentation mask map is used to characterize different categories of regions in the image to be segmented; and superimposing the segmentation mask map on the image to be segmented in the form of color coding to obtain different categories of regions in the image to be segmented.

[0093] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0094] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0095] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image segmentation method, characterized in that: include: Determine the image to be segmented; Inputting the image to be segmented into a segmentation model, obtaining a segmentation mask map corresponding to the image to be segmented output by the segmentation model; wherein the segmentation model is used to extract feature maps of different levels in the image to be segmented, and generate the segmentation mask map based on the feature maps of different levels; the segmentation mask map is used to characterize different categories of regions in the image to be segmented; The segmentation mask image is superimposed on the image to be segmented in a color-coded form to obtain regions of different categories in the image to be segmented.

2. The image segmentation method according to claim 1, characterized in that: The segmentation model includes an encoder and a decoder, and the step of inputting the image to be segmented into the segmentation model to obtain a segmentation mask image corresponding to the image to be segmented output by the segmentation model includes: Inputting the image to be segmented into the encoder to obtain a first feature map output by the encoder; the encoder is used to perform a depth-separable convolution operation and downsampling on the image to be segmented; Copying and cropping the first feature map to obtain a second feature map; Inputting the two feature maps into the decoder for sampling processing to obtain a third feature map; Concatenate the second feature map and the third feature map to obtain a fourth feature map; The fourth feature map is input into the decoder to obtain a segmentation mask map output by the decoder; the decoder is used to perform multi-scale convolution attention operation and upsampling on the fourth feature map.

3. The image segmentation method according to claim 2, characterized in that: The encoder includes a sub-network block, a convolution block and a down-sampling block, The step of inputting the image to be segmented into the encoder to obtain a first feature map output by the encoder includes: Inputting the image to be segmented into the sub-network block to obtain a fifth feature map output by the sub-network block; the sub-network block is used to extract features of the image to be segmented using a deep convolution layer and a point-by-point convolution layer; Inputting the fifth feature map into the convolution block to obtain a sixth feature map output by the convolution block; the convolution block is used to extract features from the fifth feature map using standard convolution; The sixth feature map is input into the down-sampling block to obtain a first feature map output by the down-sampling block.

4. The image segmentation method according to claim 2, characterized in that: The decoder includes: a multi-scale convolutional attention module and an upsampling convolution block, The step of inputting the fourth feature map into the decoder to obtain a segmentation mask map output by the decoder comprises: Inputting the fourth feature map into the multi-scale convolutional attention module to obtain a seventh feature map output by the multi-scale convolutional attention module; the multi-scale convolutional attention module is used to perform multi-scale feature extraction on the fourth feature map; The seventh feature map is input into the upsampling convolution block to obtain a segmentation mask map output by the upsampling convolution block.

5. The image segmentation method according to claim 4, characterized in that: The multi-scale convolutional attention module includes: a channel attention block, a spatial attention block and a multi-scale convolution block; The step of inputting the fourth feature map into the multi-scale convolutional attention module to obtain a seventh feature map output by the multi-scale convolutional attention module comprises: Inputting the fourth feature map into the channel attention block to obtain a channel attention map output by the channel attention block; the channel attention map is a feature map after the channels of the fourth feature map are weighted; Input the channel attention into the spatial attention block to obtain a spatial attention map output by the spatial attention block; the spatial attention map is a feature map after weighting each position in the channel attention map; The spatial attention map is input into the multi-scale convolution block to obtain a seventh feature map output by the multi-scale convolution block.

6. The image segmentation method according to claim 5, characterized in that: The multi-scale convolution block includes: a first point-by-point convolution layer, a first batch normalization layer, an activation layer, a second point-by-point convolution layer, and a second batch normalization layer. Inputting the spatial attention map into the multi-scale convolution block to obtain a seventh feature map output by the multi-scale convolution block includes: Inputting the spatial attention map into the first point-by-point convolutional layer to obtain an eighth feature map; the first point-by-point convolutional layer is used to increase the dimension of the number of channels of the spatial attention map; Inputting the eighth feature map into the first batch normalization layer and the activation layer in sequence to obtain a ninth feature map; Performing a shuffling operation on the channels in the ninth feature map to obtain a ninth feature map after merging the channels; Inputting the ninth feature map after merging the channels into the second point-by-point convolutional layer to obtain a tenth feature map, wherein the second point-by-point convolutional layer is used to reduce the number of channels after the ninth feature map after merging the channels; The tenth feature is input into the second batch normalization to obtain the seventh feature map.

7. The image segmentation method according to any one of claims 1 to 6, characterized in that: The step of determining the image to be segmented comprises: Obtain at least one frame of initial image to be segmented from the video to be segmented; Performing edge detection on the initial image to be segmented to obtain a first image to be segmented; Performing edge connection on the first image to be segmented to obtain a second image to be segmented; Perform edge repair on the second image to be segmented to obtain the image to be segmented.

8. An image segmentation device, characterized in that: include: A determination module, used for determining an image to be segmented; A segmentation module, used for inputting the image to be segmented into a segmentation model, and obtaining a segmentation mask map corresponding to the image to be segmented output by the segmentation model; wherein the segmentation model is used for extracting feature maps of different levels in the image to be segmented, and generating the segmentation mask map based on the feature maps of different levels; the segmentation mask map is used for characterizing regions of different categories in the image to be segmented; The encoding module is used to superimpose the segmentation mask image on the image to be segmented in the form of color coding to obtain regions of different categories in the image to be segmented.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the image segmentation method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image segmentation method according to any one of claims 1 to 7 is implemented.

Images