Image segmentation method, device, equipment, storage medium and program product

Abstract

The embodiments of the present application provide an image segmentation method, apparatus, device, storage medium and program product. The method includes: obtaining an image to be segmented and a constraint feature, extracting a first feature of the image to be segmented through an encoder of an image segmentation model, generating a second feature based on the first feature and the constraint feature, and then processing the second feature through a decoder to obtain a segmentation result. By adding a constraint feature for constraining the contour of the segmentation result during the model processing, the shape of the segmentation result can be limited, so that the image segmentation result is more in line with expectations and the accuracy of image segmentation is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to an image segmentation method, apparatus, device, storage medium and program product. Background Art

[0002] Image segmentation, as a kind of image processing method, is widely used in industrial detection, monitoring and medical fields. For example, in the medical field, it is often necessary to segment some medical images to obtain the region of interest for further analysis.

[0003] At present, image segmentation algorithms place too much emphasis on the local perception characteristics of the relationship between adjacent pixels. However, some images contain irrelevant parts with grayscale close to the region of interest. When segmenting such images, irrelevant parts are often mistakenly segmented into the region of interest, resulting in insufficient accuracy, which requires re-segmentation or manual error correction. Summary of the invention

[0004] The embodiments of the present application provide an image segmentation method, apparatus, device, storage medium and program product to solve the problem of insufficient image segmentation accuracy.

[0005] In a first aspect, an embodiment of the present application provides an image segmentation method, comprising:

[0006] Acquire an image to be segmented and a constraint feature, wherein the constraint feature is used to constrain the shape of a segmentation result of the image to be segmented;

[0007] Extracting features of the image to be segmented by an encoder of an image segmentation model, and generating a second feature according to the extracted first feature and the constraint feature, wherein the image segmentation model includes the encoder and the decoder;

[0008] The second feature is processed by the decoder to obtain a segmentation result of the image to be segmented.

[0009] In a possible implementation, the constraint feature includes a constraint matrix, and the constraint matrix is ​​obtained in the following manner:

[0010] Obtaining mask images corresponding to a plurality of sample images, wherein pixel values ​​of the mask images represent segmentation results of the sample images;

[0011] Performing a first spatial transformation on the mask image, where the first spatial transformation is used to transform different mask images into the same coordinate system;

[0012] For the mask images in the same coordinate system, the pixel average values ​​of the multiple mask images at each pixel point are calculated, and a constraint matrix is ​​generated, wherein each element in the constraint matrix represents the pixel average value of a pixel point.

[0013] In a possible implementation, performing a first spatial transformation on the mask image includes:

[0014] Selecting a sample image from the multiple sample images as an initial reference image;

[0015] For sample images other than the initial reference image among the multiple sample images, sequentially registering the sample images with the initial reference image to transform the sample images into a coordinate system corresponding to the initial reference image;

[0016] Based on the parameters of the registration between the sample image and the initial reference image, a spatial transformation is performed on the mask image corresponding to the sample image.

[0017] In a possible implementation, for the mask images in the same coordinate system, the pixel average values ​​of the multiple mask images at each pixel point are calculated, and a constraint matrix is ​​generated, including:

[0018] If there are more than two types of pixel values ​​in the mask image, one type of pixel value is selected as the target pixel value based on the segmentation result represented by the pixel value, and other types of pixel values ​​are set as default pixel values;

[0019] The average pixel values ​​of the plurality of mask images are calculated based on the target pixel value and the default pixel value, and a constraint matrix is ​​generated, where the constraint matrix is ​​used to constrain the segmentation result corresponding to the target pixel value.

[0020] In a possible implementation manner, before generating the second feature according to the extracted first feature and the constraint feature, the method further includes:

[0021] A second spatial transformation is performed on at least one of the first feature and the constraint feature, where the second spatial transformation is used to convert the first feature and the constraint feature into the same coordinate system.

[0022] In a possible implementation, performing a second spatial transformation on at least one of the first feature and the constraint feature includes:

[0023] For the mask image in the same coordinate system, calculating the similarity between the mask image and the constraint feature, and selecting the mask image with the greatest similarity as the registration reference image;

[0024] Registering the image to be segmented with the registration reference image to obtain registration parameters;

[0025] The first feature or the constraint feature is spatially transformed based on the registration parameter.

[0026] In a possible implementation, the first feature extracted by the encoder includes a spatial feature and a semantic feature, and generating the second feature according to the extracted first feature and the constraint feature includes:

[0027] Determine whether the resolution of the spatial feature matches that of the constraint feature, and if so, fuse the spatial feature with the constraint feature by performing a Hadamard product operation, and combine the fused feature with the semantic feature to obtain a second feature;

[0028] If there is no match, the constraint feature is downsampled based on the resolution of the spatial feature, the spatial feature is fused with the downsampling result through a Hadamard product operation, and the fused feature is combined with the semantic feature to obtain a second feature.

[0029] In a second aspect, an embodiment of the present application provides an image segmentation device, comprising:

[0030] An acquisition module, used for acquiring an image to be segmented and a constraint feature, wherein the constraint feature is used for constraining the shape of a segmentation result of the image to be segmented;

[0031] A feature processing module, used for extracting features of the image to be segmented through an encoder of an image segmentation model, and generating a second feature according to the extracted first feature and the constraint feature, wherein the image segmentation model includes the encoder and the decoder;

[0032] A segmentation module is used to process the second feature through the decoder to obtain a segmentation result of the image to be segmented.

[0033] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor;

[0034] The memory stores computer-executable instructions;

[0035] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.

[0036] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementations of the first aspect.

[0037] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementation methods of the first aspect.

[0038] The image segmentation method, device, equipment, storage medium and program product provided by the embodiments of the present application relate to the field of image processing. The image segmentation method comprises: obtaining an image to be segmented, extracting a first feature of the image to be segmented through an encoder of an image segmentation model, generating a second feature based on the first feature and a constraint feature, and then processing the second feature through a decoder to obtain a segmentation result. By adding a constraint feature for constraining the contour of the segmentation result during the model processing, the shape of the segmentation result can be limited, so that the image segmentation result is more in line with expectations and the accuracy of image segmentation is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0040] Figure 1 It is a schematic diagram of image segmentation provided in an embodiment of the present application;

[0041] Figure 2 It is a flowchart of an image segmentation method provided in an embodiment of the present application;

[0042] Figure 3 It is a schematic diagram of a model processing process provided by an embodiment of the present application;

[0043] Figure 4 It is a schematic diagram of a process for obtaining a constraint matrix provided in an embodiment of the present application;

[0044] Figure 5 is a working schematic diagram of an image segmentation model provided in an embodiment of the present application;

[0045] Figure 6 is another image segmentation schematic diagram provided in an embodiment of the present application;

[0046] Figure 7 is a structural schematic diagram of an image segmentation device provided in an embodiment of the present application;

[0047] Figure 8 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application.

[0048] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0049] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0050] The terms "comprises", "includes" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, product or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, product or device. In the absence of further restrictions, it is not excluded that there are other identical or equivalent elements in the process, method, product or device including the elements. For example, if the words first, second, etc. are used to indicate names, they do not indicate any particular order.

[0051] Image segmentation has a wide range of applications in the medical field and can be used to extract meaningful information from medical images. For example, some studies need to obtain image information of specific bones, and a specific vertebra can be obtained by performing image segmentation on the CT (Computed Tomography) image of the spine.

[0052] At present, image segmentation algorithms can use neural networks to replace manual segmentation of batch images, but the segmentation effect is not ideal. The segmentation results often contain redundant and irrelevant parts, and the accuracy needs to be improved.

[0053] The inventors have found through research that the reason why image segmentation includes irrelevant parts is that the neural network used in the segmentation algorithm pays too much attention to the local perception characteristics of the relationship between adjacent pixels, while some irrelevant parts have grayscale features similar to the expected segmentation results in the image, and these irrelevant parts are easily mistakenly segmented into normal results. For example, bones such as the spine often have bone hyperplasia, bone spurs, etc. These irrelevant parts such as bone hyperplasia and bone spurs are sometimes not part of the expected segmentation results, but these parts are close to the grayscale of normal bones in the image, which easily leads to these parts being regarded as normal bones during segmentation, resulting in an extra part of the segmentation result, forming over-segmentation.

[0054] Figure 1 is a schematic diagram of image segmentation provided by an embodiment of the present application. Figure 1 , Figure 1 The left side is the image before segmentation. The image contains multiple bones, and there is a bone hyperplasia in the upper left corner of the second bone. Figure 1The right side is the image after segmentation, where the part covered by the diagonal lines is the segmentation result. It can be seen that due to the close grayscale of the bone hyperplasia area and the second bone, the bone hyperplasia area was mistakenly segmented as the second bone during segmentation.

[0055] In response to the above problems, the inventor proposed a technical concept. On the basis of grayscale-based image segmentation, the shape features of the parts involved in image segmentation are analyzed, the shape features are used as constraints, and a new image segmentation method is implemented in combination with grayscale features to alleviate the over-segmentation problem in image segmentation and improve the accuracy of image segmentation.

[0056] Based on this, the inventors also found that although there may be parts in the image that are unrelated to the expected segmentation results, such as bone hyperplasia and bone spurs, the sizes and shapes of these parts are different, making it difficult to conduct a unified analysis. The inventors therefore thought that when analyzing shape features, the expected segmentation results should be used as the analysis target, and several accurate segmentation results should be collected to establish constraints. Taking the segmentation of spinal bones as an example, the bones obtained by normal segmentation are collected, and constraints are determined based on the shapes of these bones. The constraints are then applied to the image segmentation process to improve segmentation accuracy.

[0057] The application scenarios involved above are only partial examples. Those skilled in the art can expand the application according to the specific scenarios, and the embodiments of the present application do not impose specific restrictions on this. In other scenarios, image segmentation can also be used in the industrial field. For example, some industrial processes need to use image segmentation to obtain specific parts of the equipment. Due to process defects and other reasons, some parts of the equipment may have deformations and protruding parts. These parts may also cause over-segmentation, and the image segmentation method provided by this application can also be applied. It should be noted that the above application scenarios are only shown to facilitate the understanding of the spirit and principles of this application, and the implementation methods of this application are not subject to any restrictions in this regard. On the contrary, the implementation methods of this application can be applied to any applicable scenario.

[0058] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0059] Figure 2 FIG. 1 is a flow chart of an image segmentation method provided in an embodiment of the present application. Figure 2 As shown, the method includes:

[0060] Step S201, obtaining an image to be segmented and constraint features.

[0061] The constraint feature is used to constrain the shape of the segmentation result of the image to be segmented.

[0062] Exemplarily, a data set may be pre-established, and the data set may include several sample images of the same type as the image to be segmented. For example, if the image to be segmented is a spinal image, the sample image may also be a spinal image. The data set may also include image segmentation results of the sample images. By analyzing the shapes of these image segmentation results, the shape information of the correct segmentation results may be determined as a constraint feature. For example, the sample image may be a spinal image that does not include irrelevant parts such as bone hyperplasia and bone spurs. For these images, relatively standard bones may be segmented, and the shape features of these segmented bones may be summarized. The constraint features may be determined by calculating the average value of the pixels corresponding to the bones in the image.

[0063] It should be noted that the sample image may also contain irrelevant parts such as bone hyperplasia and bone spurs. The irrelevant parts present in the sample image are inversely proportional to the accuracy of the constraint feature, that is, the fewer irrelevant parts there are, the higher the accuracy of the constraint feature, and thus the higher the accuracy of the image segmentation after applying the constraint feature.

[0064] Step S202: extract features from the image to be segmented through an encoder of the image segmentation model, and generate second features based on the extracted first features and constraint features.

[0065] The image segmentation model includes an encoder and a decoder. The encoder may include multiple encoding layers, each of which may be used to perform operations such as convolution, activation, and / or pooling.

[0066] Exemplarily, the image to be segmented can be input into the first encoding layer of the encoder to obtain a feature map processed by the first encoding layer, and then the feature map is input into the second encoding layer. In this way, each encoding layer processes the feature map output by the previous encoding layer to obtain a new feature map after processing.

[0067] like Figure 3 As shown, the encoder can extract features of the input image, and perform feature fusion on the extracted features and the pre-acquired constraint features, and after fusion, the features are input into the decoder to output the segmentation result through the decoder.

[0068] For example, the image segmentation model can be trained based on a neural network structure such as U-Net or V-Net. The encoder and decoder can be trained in advance using a number of sample images. The training method can be the same as Figure 3 The processing process is the same. After training iterations, when the loss value is less than the threshold, the trained encoder and decoder can be obtained.

[0069] The encoder can gradually extract the features of the input image and compress the spatial size of the features. For example, the multiple encoding layers contained in the encoder can be used to extract features at different levels. The front encoding layers can mainly extract shape features such as texture and contour in the image, and the back encoding layers can extract deep features such as semantics in the image. Optionally, when performing feature fusion, only the constraint features can be fused with the shape features extracted by the front encoding layer, and deep features such as semantic features can be retained.

[0070] Step S203: Process the second feature through the decoder to obtain a segmentation result of the image to be segmented.

[0071] The decoder may include multiple decoding layers for decoding the features extracted by the encoder and restoring image details.

[0072] Exemplarily, the feature map output by the last encoding layer can be input to the first decoding layer, and the feature map output by the first decoding layer can be input to the second decoding layer, and so on. The decoding layer can gradually amplify the feature map by upsampling and use operations such as convolution to extract details. When the decoding layer processes shape-related features, the feature map output by the previous decoding layer can be spliced ​​with the part of the second feature that incorporates the constraint features, so that the decoding layer can determine the shape of the segmentation result under the constraints, and finally output the image segmentation result.

[0073] In the above embodiment, the encoder of the image segmentation model can extract the first feature of the image to be segmented, and the second feature can be generated according to the first feature and the constraint feature, and the decoder can process the second feature to obtain the segmentation result. By adding the constraint feature for constraining the contour of the segmentation result in the model processing process, the shape of the segmentation result can be limited, so that the image segmentation result is more in line with expectations and the accuracy of image segmentation is improved.

[0074] In one embodiment, the constraint feature may include a constraint matrix such as Figure 4 As shown, the constraint matrix is ​​obtained as follows:

[0075] Step S401, obtaining mask images corresponding to multiple sample images.

[0076] The pixel values ​​of the mask image represent the segmentation result of the sample image.

[0077] Exemplarily, several CT images can be obtained as sample images, and mask images corresponding to these CT images can be obtained. The CT image can be a grayscale image. The mask image can be a binary image (the pixel value of the segmented part is 1, and the pixel value of the remaining part is 0). For example, if the sample image is a spinal CT image containing multiple vertebrae, and the desired segmentation part is the second vertebra in the image, then in the mask image corresponding to the sample image, the pixel value of the pixel point where the second vertebra is located is 1, and the pixel points of the remaining parts of the mask image are 0.

[0078] Exemplarily, the mask image may also contain more than two pixel values. For example, taking a spinal CT image containing multiple vertebrae, if the second vertebrae and the third vertebrae are both desired segmentation sites, then in the mask image corresponding to the sample image, the pixel value of the pixel where the second vertebrae is located is 1, the pixel value of the pixel where the third vertebrae is located is 2, and the pixel values ​​of the remaining parts of the mask image are 0. Accordingly, the mask image may also contain three, four or even more pixel values.

[0079] Step S402: performing a first spatial transformation on the mask image.

[0080] The first spatial transformation is used to transform different mask images into the same coordinate system.

[0081] In some possible implementations, performing a first spatial transformation on the mask image may include: selecting a sample image from multiple sample images as an initial reference image; for sample images other than the initial reference image among the multiple sample images, aligning the sample images with the initial reference image in turn to convert the sample images to a coordinate system corresponding to the initial reference image; and performing a spatial transformation on the mask image corresponding to the sample image based on the parameters of the alignment of the sample image with the initial reference image.

[0082] For example, N sample images (such as CT1, CT2, ..., CT N ) selects an initial reference image that is easy to align with other images and can be recorded as CT(Ref0). The other sample images are sequentially aligned with Ref0 and the alignment parameters of each alignment are recorded. For example, the alignment parameters of image CT1 after alignment with CT(Ref0) are Regp1:(r x ,r y ,r z ,t x ,t y ,t z ), which represents the rotation angle around its X-axis, Y-axis, and Z-axis and the translation distance along its X-axis, Y-axis, and Z-axis; if it is a two-dimensional image, the registration parameters can be expressed as (r, t x ,t y), r can represent the angle of rotation of the image around the X axis. After obtaining the registration parameters, the registration parameters of the mask image mask1 corresponding to CT1 can also be expressed as Regp1, and Regp1 is used to transform mask1 to obtain the transformed mask 1m , mask 1m The mask image corresponding to Ref0 is in the same coordinate system. Accordingly, by transforming other mask images, we can get mask 2m 、mask 3m 、…。

[0083] Step S403 , for the mask images in the same coordinate system, calculate the pixel average value of the multiple mask images at each pixel point, and generate a constraint matrix.

[0084] Each element in the constraint matrix represents the average value of a pixel.

[0085] For example, after the mask images are aligned to the same coordinate system through image registration, the mask images can be 1m 、mask 2m 、mask 3m etc.) average by pixel position and calculate the average mask mean . Mask mean The constraint matrix can be obtained by mapping the values ​​of to the elements of the matrix.

[0086]

[0087] If the mask image is a binary image, then in mask mean In the mask, the value of each pixel is in the interval [0,1]. im The pixel value is 0 or 1, mask mean The value of is a floating point number (decimal) in the interval [0,1], with a maximum of 1 and a minimum of 0.

[0088] In some possible implementations, for mask images in the same coordinate system, calculating the pixel average of multiple mask images at each pixel point and generating a constraint matrix may also include:

[0089] If there are more than two types of pixel values ​​in the mask image, one type of pixel value is selected as the target pixel value based on the segmentation result represented by the pixel value, and the pixel values ​​of other types are set as the default pixel values; the average pixel value of multiple mask images is calculated based on the target pixel value and the default pixel value, and a constraint matrix is ​​generated. The constraint matrix is ​​used to constrain the segmentation result corresponding to the target pixel value.

[0090] Exemplarily, the mask image contains three pixel values ​​of 0, 1, and 2, where the pixel point with a pixel value of 1 represents the second vertebra, and the pixel point with a pixel value of 2 represents the third vertebra. If you want to obtain the constraint matrix for segmenting the third vertebra, you can first set all the pixel points with a pixel value of 1 to 0, and then set all the pixel points with a pixel value of 2 to 1, and then calculate the constraint matrix according to the above method.

[0091] In some other possible implementations, after obtaining mask images of multiple sample images, a constraint matrix can be formed according to the pixel value distribution rules of different mask images. For example, the value of the pixel point can be determined according to the number of masks with a value of 1 at the same pixel point, and the constraint matrix can be generated based on the final determined values ​​of all the pixels. For example, there are CT1, CT2, ..., CT 10 There are 10 sample images in total. The resolutions of these 10 sample images are aligned (i.e., the pixels correspond one to one). The pixel values ​​of CT1 to CT9 at these pixels are all 1. 10 When the pixel value of this pixel is 0, and the distribution of these sample images at this pixel meets certain conditions (such as the number of mask images with a pixel value of 1 at this pixel is greater than or equal to 9, or the number of mask images with a pixel value of 1 at this pixel accounts for 90% or more of the total number of mask images), then the value of the element corresponding to this pixel in the finally generated constraint matrix is ​​1.

[0092] Figure 5 A schematic diagram of the working of an image segmentation model provided in an embodiment of the present application. Figure 5 The model processing part in this application is described with specific embodiments.

[0093] like Figure 5 As shown, the image segmentation model includes an encoder part on the left and a decoder part on the right, and there is a jump connection between the encoder and the decoder.

[0094] The encoder can be used to gradually extract features of the input image and compress the spatial size of the features. The encoder can contain multiple encoding layers (i.e., the cuboid on the left side of the figure), which can be composed of a series of convolution, activation and / or pooling operations. After multiple operations, the encoder can extract deep features of the image and reduce the spatial resolution of the feature map.

[0095] The decoder can contain multiple decoding layers to gradually decode the features extracted by the encoding layer and restore more detailed information. It can be composed of a series of upsampling and convolution operations. The decoding layer gradually enlarges the feature map by upsampling and uses convolution operations to extract details.

[0096] The jump connection can concatenate the original feature maps of the same size in the encoding layer ( Figure 5 The C in the middle circle represents feature map concatenation) to the decoding layer, which can retain the original high-resolution information and help the decoding layer better restore image details.

[0097] In one embodiment, the first feature extracted by the encoder includes a spatial feature and a semantic feature, and generating a second feature according to the extracted first feature and the constraint feature includes:

[0098] Determine whether the resolutions of the spatial feature and the constraint feature match. If they match, fuse the spatial feature and the constraint feature by performing a Hadamard product operation, and combine the fused feature with the semantic feature to obtain a second feature.

[0099] If there is no match, the constraint feature is downsampled based on the resolution of the spatial feature, and the spatial feature is fused with the downsampling result through a Hadamard product operation, and the fused feature is combined with the semantic feature to obtain a second feature.

[0100] Reference Figure 5 In the image segmentation model shown, the image to be segmented can be input into the first encoding layer of the encoder. During the feature extraction process of multiple encoding layers, the first two encoding layers can extract more spatial features related to shape and contour, and the subsequent encoding layers can extract deeper semantic features.

[0101] For example, after the first encoding layer extracts the spatial features, the jump connection can pass the spatial features to the right to the last decoding layer. In this process, the constraint features can be fused with the transferred spatial features through the Hadamard product operation, and then concatenated with the features output by the penultimate decoding layer and the concatenated features are input into the last decoding layer.

[0102] For example, after the second encoding layer extracts the spatial feature, the jump connection can pass the spatial feature to the right to the second-to-last decoding layer, but the resolution of the spatial feature is smaller than the resolution of the constraint feature. The constraint feature can be downsampled to match its resolution with the spatial feature, and then fused through the Hadamard product operation. After the fusion, it is spliced ​​with the feature output by the third-to-last decoding layer and the spliced ​​features are input into the second-to-last decoding layer.

[0103] Exemplarily, the features extracted by the third coding layer and subsequent coding layers can be considered to be weakly associated with spatial features such as shape, and these features can be directly transmitted to the corresponding decoding layer without being fused with constraint features.

[0104] It should be noted that the above-mentioned division of spatial features is only a possible example. According to the differences in image segmentation models, the method of fusing constraint features can be adaptively selected (such as which features of the encoding layer outputs can be fused with constraint features, and which features of the encoding layer outputs can not be fused with constraint features). This application does not limit this.

[0105] In one embodiment, before generating the second feature according to the extracted first feature and the constraint feature, the method further includes:

[0106] A second spatial transformation is performed on at least one of the first feature and the constraint feature.

[0107] The second spatial transformation is used to transform the first feature and the constraint feature into the same coordinate system.

[0108] Optionally, for the mask image in the same coordinate system, the similarity between the mask image and the constraint feature is calculated, and the sample image corresponding to the mask image with the largest similarity is selected as the registration reference image; the image to be segmented is registered with the registration reference image to obtain the registration parameters; and the first feature or the constraint feature is spatially transformed based on the registration parameters.

[0109] For example, for the registered mask image (such as mask 1m 、mask 2m 、mask 3m etc.), each mask image mask can be calculated im With the average mask mean The similarity between them is calculated, and the sample image corresponding to the mask image with the largest similarity is taken as the registration reference image, which can be recorded as CT(Ref1).

[0110] For example, when calculating the similarity, a generalized Dice coefficient between the mask image and the constraint matrix may be selected as the similarity.

[0111] by Figure 5 Taking the image segmentation model shown as an example, in addition to inputting the image to be segmented into the encoder, the image to be segmented can also be registered with the registration reference image Ref1, and the registration parameters are recorded. The constraint features are spatially transformed using the registration parameters, so that the constraint features can be aligned with the image to be segmented to the same coordinate system, and better integrated with the spatial features extracted by the encoding layer of the image to be segmented, thereby improving the accuracy of image segmentation.

[0112] It should be noted that before using the image segmentation model to process the image to be segmented, the image segmentation model can be trained in advance using the sample image, the mask image corresponding to the sample image, and the constraint features. The training process can refer to the above-mentioned processing process of the image to be segmented. The loss function can use sieve coefficients and / or cross entropy, etc. The optimization method can use gradient descent or Adam and other methods to set certain learning rate adjustment rules and batch size. Based on the sample image and the mask image, after a certain number of iterations, when the loss function is less than a certain value, a trained image segmentation model can be obtained.

[0113] Figure 6 Another image segmentation schematic diagram provided in the embodiment of the present application. Figure 6 As shown, for Figure 1 The spine image in the figure is segmented using the above-mentioned method of adding constraint features for shape constraint. In the obtained image, only the pixel value of the second bone (the part covered by the diagonal line indicates that the pixel value is 1) is 1, and the pixel value of the rest (including the original bone hyperplasia area) is 0, thereby obtaining the correct segmentation result and improving the accuracy of image segmentation.

[0114] Figure 7 Schematic diagram of the structure of an image segmentation device provided in an embodiment of the present application. Figure 7 As shown, the image segmentation device 700 may include:

[0115] The acquisition module 701 is used to acquire the image to be segmented and constraint features, where the constraint features are used to constrain the shape of the segmentation result of the image to be segmented.

[0116] The feature processing module 702 is used to extract features of the image to be segmented through an encoder of an image segmentation model, and generate a second feature based on the extracted first feature and the constraint feature, and the image segmentation model includes the encoder and the decoder.

[0117] The segmentation module 703 is used to process the second feature through the decoder to obtain a segmentation result of the image to be segmented.

[0118] In one embodiment, the acquisition module 701 further includes:

[0119] The first acquisition unit may be used to acquire mask images corresponding to a plurality of sample images, wherein the pixel values ​​of the mask images represent the segmentation results of the sample images.

[0120] The first transformation unit may be used to perform a first spatial transformation on the mask image, where the first spatial transformation is used to convert different mask images into the same coordinate system.

[0121] The generating unit may be used to calculate the pixel average value of multiple mask images at each pixel point for the mask images in the same coordinate system, and generate a constraint matrix, wherein each element in the constraint matrix represents the pixel average value of a pixel point.

[0122] In one embodiment, the first transformation unit is also used to: select a sample image from the multiple sample images as an initial reference image; for sample images other than the initial reference image among the multiple sample images, align the sample image with the initial reference image in turn to convert the sample image to a coordinate system corresponding to the initial reference image; and perform spatial transformation on the mask image corresponding to the sample image based on the parameters of the alignment between the sample image and the initial reference image.

[0123] In one embodiment, the generating unit is further used for: if there are more than two types of pixel values ​​in the mask image, selecting one pixel value as the target pixel value based on the segmentation result represented by the pixel value, and setting the pixel values ​​of other types as default pixel values; calculating the average pixel value of multiple mask images based on the target pixel value and the default pixel value, and generating a constraint matrix, wherein the constraint matrix is ​​used to constrain the segmentation result corresponding to the target pixel value.

[0124] In one embodiment, the feature processing module 702 is further used to: perform a second spatial transformation on at least one of the first feature and the constraint feature, wherein the second spatial transformation is used to convert the first feature and the constraint feature into the same coordinate system.

[0125] In one embodiment, the feature processing module 702 is also used to: calculate the similarity between the mask image and the constraint feature in the same coordinate system, and select the mask image with the largest similarity as the registration reference image; align the image to be segmented with the registration reference image to obtain registration parameters; and perform spatial transformation on the first feature or the constraint feature based on the registration parameters.

[0126] In one embodiment, the feature processing module 702 is also used to: determine whether the resolutions of the spatial feature and the constraint feature match; if they match, the spatial feature and the constraint feature are fused by performing a Hadamard product operation, and the fused feature is combined with the semantic feature to obtain a second feature; if they do not match, the constraint feature is downsampled based on the resolution of the spatial feature, and the spatial feature is fused with the downsampling result by performing a Hadamard product operation, and the fused feature is combined with the semantic feature to obtain a second feature.

[0127] The image segmentation device provided in this embodiment can be used to execute the technical solution in any of the aforementioned method embodiments. Its implementation principle and technical effects are similar and will not be repeated here.

[0128] It should be understood that the above-mentioned device embodiments are only illustrative, and the device of the present application can also be implemented in other ways. For example, the division of units / modules in the above-mentioned embodiments is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.

[0129] In addition, unless otherwise specified, each functional unit / module in each embodiment of the present application may be integrated into one unit / module, each unit / module may exist physically separately, or two or more units / modules may be integrated together. The above-mentioned integrated unit / module may be implemented in the form of hardware or in the form of a software program module.

[0130] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 8 As shown, the electronic device 80 may include:

[0131] Processor 81, memory 82, and communication interface 83;

[0132] The memory 82 is used to store executable instructions of the processor 81; the executable instructions may be computer-executable instructions;

[0133] The processor 81 is configured to execute the technical solution in any of the aforementioned method embodiments by executing the executable instructions.

[0134] Optionally, the memory 82 can be independent or integrated with the processor 81.

[0135] Optionally, when the memory 82 is a device independent of the processor 81, the electronic device 80 may further include:

[0136] The bus 84 , the memory 82 and the communication interface 83 are connected to the processor 81 via the bus 84 and communicate with each other. The communication interface 83 is used to communicate with other devices.

[0137] Optionally, the communication interface 83 may be implemented by a transceiver. The communication interface is used to implement communication between the database access device and other devices (such as a client, a read-write library, and a read-only library). The memory may include a random access memory (RAM) and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage.

[0138] The bus 84 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0139] The above-mentioned processor can be a general-purpose processor, including a central processing unit CPU, a network processor (NP), etc.; it can also be a digital signal processor DSP, an application-specific integrated circuit ASIC, a field programmable gate array FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0140] The electronic device is used to execute the technical solution in any of the aforementioned method embodiments, and its implementation principle and technical effect are similar and will not be repeated here.

[0141] An embodiment of the present application also provides a readable storage medium, which may be a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the technical solution provided by any of the aforementioned method embodiments is implemented.

[0142] An embodiment of the present application also provides a computer program product, including a computer program, which is used to implement the technical solution provided by any of the aforementioned method embodiments when executed by a processor.

[0143] In the above embodiments, the description of each embodiment has its own emphasis. For the part not described in detail in a certain embodiment, please refer to the relevant description of other embodiments. The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0144] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0145] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program codes.

[0146] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0147] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. An image segmentation method, characterized in that: include: Acquire an image to be segmented and a constraint feature, wherein the constraint feature is used to constrain the shape of a segmentation result of the image to be segmented; Extracting features of the image to be segmented by an encoder of an image segmentation model, and generating a second feature according to the extracted first feature and the constraint feature, wherein the image segmentation model includes the encoder and the decoder; The second feature is processed by the decoder to obtain a segmentation result of the image to be segmented.

2. The image segmentation method according to claim 1, characterized in that: The constraint feature includes a constraint matrix, and the constraint matrix is ​​obtained in the following manner: Obtaining mask images corresponding to a plurality of sample images, wherein pixel values ​​of the mask images represent segmentation results of the sample images; Performing a first spatial transformation on the mask image, where the first spatial transformation is used to transform different mask images into the same coordinate system; For the mask images in the same coordinate system, the pixel average values ​​of the multiple mask images at each pixel point are calculated, and a constraint matrix is ​​generated, wherein each element in the constraint matrix represents the pixel average value of a pixel point.

3. The image segmentation method according to claim 2, characterized in that: The performing a first spatial transformation on the mask image comprises: Selecting a sample image from the multiple sample images as an initial reference image; For sample images other than the initial reference image among the multiple sample images, sequentially registering the sample images with the initial reference image to transform the sample images into a coordinate system corresponding to the initial reference image; Based on the parameters of the registration between the sample image and the initial reference image, a spatial transformation is performed on the mask image corresponding to the sample image.

4. The image segmentation method according to claim 2 or 3, characterized in that: The step of calculating the pixel average of each pixel point of the plurality of mask images for the mask images in the same coordinate system and generating a constraint matrix includes: If there are more than two types of pixel values ​​in the mask image, one type of pixel value is selected as the target pixel value based on the segmentation result represented by the pixel value, and other types of pixel values ​​are set as default pixel values; The average pixel values ​​of the plurality of mask images are calculated based on the target pixel value and the default pixel value, and a constraint matrix is ​​generated, where the constraint matrix is ​​used to constrain the segmentation result corresponding to the target pixel value.

5. The image segmentation method according to any one of claims 1 to 3, characterized in that: Before generating the second feature according to the extracted first feature and the constraint feature, the method further includes: A second spatial transformation is performed on at least one of the first feature and the constraint feature, where the second spatial transformation is used to convert the first feature and the constraint feature into the same coordinate system.

6. The image segmentation method according to claim 5, characterized in that: The performing a second spatial transformation on at least one of the first feature and the constraint feature comprises: For the mask image in the same coordinate system, calculating the similarity between the mask image and the constraint feature, and selecting the sample image corresponding to the mask image with the greatest similarity as the registration reference image; Registering the image to be segmented with the registration reference image to obtain registration parameters; The first feature or the constraint feature is spatially transformed based on the registration parameter.

7. The image segmentation method according to any one of claims 1 to 3, characterized in that: The first feature extracted by the encoder includes a spatial feature and a semantic feature, and the generating the second feature according to the extracted first feature and the constraint feature includes: Determine whether the resolution of the spatial feature matches that of the constraint feature, and if so, fuse the spatial feature with the constraint feature by performing a Hadamard product operation, and combine the fused feature with the semantic feature to obtain a second feature; If there is no match, the constraint feature is downsampled based on the resolution of the spatial feature, the spatial feature is fused with the downsampling result through a Hadamard product operation, and the fused feature is combined with the semantic feature to obtain a second feature.

8. An image segmentation device, characterized in that: include: An acquisition module, used for acquiring an image to be segmented and a constraint feature, wherein the constraint feature is used for constraining the shape of a segmentation result of the image to be segmented; A feature processing module, configured to extract features of the image to be segmented through an encoder of an image segmentation model, and generate a second feature according to the extracted first feature and the constraint feature, wherein the image segmentation model includes the encoder and the decoder; A segmentation module is used to process the second feature through the decoder to obtain a segmentation result of the image to be segmented.

9. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

11. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 7 when being executed by a processor.

Images