Image processing method, apparatus, device, and medium

Abstract

The embodiment of the application provides an image processing method, device, equipment and medium, a first feature is obtained by learning local information according to a bone marrow region image, a second feature is obtained by performing three-dimensional voxel-based multi-head self-attention calculation processing on the bone marrow region image, and the first feature and the second feature are fused; a plurality of bone marrow information of different scales is extracted according to the fused feature, and a global feature is obtained from the plurality of bone marrow information; decoding is performed according to the fused feature and the global feature to obtain a decoding feature; a loss function value of an image processing model is calculated according to the decoding feature, and the parameters of the image processing model are adjusted according to the loss function value; the problems of large morphological change, unbalanced category distribution, too small volume size of part of the bone marrow, and fuzzy boundary of functional bone marrow in the strongest region and the second strongest region of the functional bone marrow can be effectively solved, and the functional bone marrow region can be accurately and effectively segmented from the CT image.

Description

TECHNICAL FIELD

[0001] The embodiment of the present application relates to but is not limited to the field of image processing, and particularly relates to an image processing method and device, equipment and a medium. BACKGROUND

[0002] Accurate segmentation of a functional bone marrow distribution area with hematopoietic activity from a CT image can effectively guide the design of precise IMRT, and then guide the optimization of a radiotherapy plan, fully protect the functional bone marrow area, and especially protect the functional bone marrow with active hematopoietic function. The proportion volume and area of the pelvic functional bone marrow image bone marrow differ greatly, and the segmentation belongs to extremely unbalanced categories, and the medical image boundary is fuzzy and the gradient is complex. The current image processing method has poor segmentation effect on the functional bone marrow. SUMMARY

[0003] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0004] The embodiment of the present application provides an image processing method, device, equipment and medium, which can accurately and effectively segment the functional bone marrow area from the CT image.

[0005] The first aspect of the present application is an image processing method, comprising:

[0006] Obtaining a bone marrow area image, and inputting the bone marrow area image into an image processing model;

[0007] Learning local information according to the bone marrow area image to obtain a first feature, and performing three-dimensional voxel-based multi-head self-attention calculation processing on the bone marrow area image to obtain a second feature, and obtaining a fusion feature according to the first feature and the second feature;

[0008] Extracting a plurality of bone marrow information of different scales according to the fusion feature, encoding the plurality of bone marrow information to obtain a global feature;

[0009] Decoding the fusion feature and the global feature to obtain a decoding feature;

[0010] Calculating a loss function value of the image processing model according to the decoding feature, adjusting parameters of the image processing model according to the loss function value, and obtaining a trained image processing model;

[0011] Obtaining an image to be segmented, inputting the image to be segmented into the trained image processing model, and obtaining a bone marrow area segmentation image.

[0012] According to the first aspect of the present application, the three-dimensional voxel-based multi-head self-attention calculation processing on the bone marrow area image to obtain a second feature comprises:

[0013] reform the bone marrow region image according to three-dimensional voxels to obtain a reformatted image;

[0014] According to the query, key, value and relative position coding based on the self-attention mechanism, the attention value of each three-dimensional voxel is obtained according to the reformatted image;

[0015] According to the attention value of each three-dimensional voxel, the second feature is obtained.

[0016] According to the first aspect of the present application, the fusion feature is obtained according to the first feature and the second feature, comprising:

[0017] The bone marrow region image is subjected to convolution processing, normalization processing and activation processing to obtain a convolution feature;

[0018] The first feature and the second feature are spliced to obtain an attention block feature;

[0019] The convolution feature and the attention block feature are fused to obtain a fusion feature.

[0020] According to the first aspect of the present application, the plurality of bone marrow information of different scales is extracted according to the fusion feature, comprising:

[0021] The fusion feature is subjected to a pooling operation corresponding to a first pooling kernel to obtain a first pooling feature;

[0022] The fusion feature is subjected to a pooling operation corresponding to a second pooling kernel to obtain a second pooling feature, and the size of the second pooling kernel is smaller than that of the first pooling kernel;

[0023] The first pooling feature is up-sampled by tri-linear interpolation to obtain a first up-sampled feature;

[0024] The first up-sampled feature and the second pooling feature are taken as a plurality of bone marrow information.

[0025] According to the first aspect of the present application, the plurality of bone marrow information is encoded to obtain a global feature, comprising:

[0026] The plurality of bone marrow information is subjected to linear projection to obtain spatial coding information including a dimension reduction feature and a global mapping projection matrix;

[0027] The dimension reduction feature and the global mapping projection matrix are reshaped, the reshaped dimension reduction feature and the global mapping projection matrix are subjected to matrix multiplication to obtain a node feature mapping, the node feature mapping is input into a Transformer block composed of an MSA layer and an MLP layer to obtain a first sub-global feature;

[0028] According to the plurality of bone marrow information, a reverse projection matrix is created, the first sub-global feature is multiplied by the reverse projection matrix to obtain a second sub-global feature transformed to an original space;

[0029] According to the second sub-global feature and the plurality of bone marrow information, the global feature is obtained.

[0030] According to the first aspect of the present application, the decoding according to the fusion feature and the global feature obtains a decoded feature, including:

[0031] The fusion feature is sequentially subjected to a plurality of up-sampling modules for up-sampling to obtain a plurality of second up-sampling features;

[0032] A plurality of the global features and a plurality of the second up-sampling features are one-to-one correspondingly connected by jumping;

[0033] From a key sampling position of the global feature, a key sampling feature is extracted, and the key sampling feature and the second up-sampling feature are subjected to matrix multiplication to obtain the decoded feature.

[0034] According to the first aspect of the present application, the loss function value includes a Dice coefficient loss function value and a cross-entropy loss function value.

[0035] The second aspect of the present application is an image processing device, including:

[0036] An input module is configured to acquire a bone marrow region image and input the bone marrow region image to an image processing model;

[0037] An encoding module is configured to learn local information according to the bone marrow region image to obtain a first feature, and perform a three-dimensional voxel-based multi-head self-attention calculation process according to the bone marrow region image to obtain a second feature, and obtain a fusion feature according to the first feature and the second feature;

[0038] A global feature acquisition module is configured to extract a plurality of bone marrow information of different scales according to the fusion feature, encode the plurality of bone marrow information, and obtain a global feature;

[0039] A decoding module is configured to decode according to the fusion feature and the global feature to obtain a decoded feature;

[0040] A parameter adjustment module is configured to calculate a loss function value of the image processing model according to the decoded feature, adjust parameters of the image processing model according to the loss function value, and obtain a trained image processing model;

[0041] A segmentation module is configured to acquire an image to be segmented, input the image to be segmented to the trained image processing model, and obtain a bone marrow region segmentation image.

[0042] In a third aspect of the present application, an electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the image processing method as described above when executing the computer program.

[0043] In a fourth aspect of the present application, a computer readable storage medium stores computer executable instructions for executing the image processing method as described above.

[0044] Embodiments of the present application include: obtaining a bone marrow region image, inputting the bone marrow region image into an image processing model; learning local information according to the bone marrow region image to obtain first features, and performing three-dimensional voxel-based multi-head self-attention calculation processing according to the bone marrow region image to obtain second features, and obtaining fusion features according to the first features and the second features; extracting multiple bone marrow information of different scales according to the fusion features, encoding the multiple bone marrow information to obtain global features; decoding according to the fusion features and the global features to obtain decoded features; calculating a loss function value of the image processing model according to the decoded features, adjusting parameters of the image processing model according to the loss function value to obtain a trained image processing model; obtaining a to-be-segmented image, inputting the to-be-segmented image into the trained image processing model to obtain a bone marrow region segmentation image; which can effectively solve the problems of large morphological changes, unbalanced class distribution, and fuzzy functional bone marrow boundary of the strongest and second strongest regions of functional bone marrow, and realize accurate and effective segmentation of functional bone marrow regions from CT images. BRIEF DESCRIPTION OF DRAWINGS

[0045] The accompanying drawings are included to provide a further understanding of the technical solutions of the present application, constitute a part of the specification, and are used together with the embodiments of the present application to explain the technical solutions of the present application, and do not constitute a limitation to the technical solutions of the present application.

[0046] Figure 1 is a step diagram of the image processing method provided by the embodiments of the present application;

[0047] Figure 2 is a sub-step diagram of the step of obtaining second features by performing three-dimensional voxel-based multi-head self-attention calculation processing according to the bone marrow region image;

[0048] Figure 3 is a sub-step diagram of the step of extracting multiple bone marrow information of different scales according to the fusion features;

[0049] Figure 4 is a sub-step diagram of the step of encoding the multiple bone marrow information to obtain global features;

[0050] Figure 5 is a sub-step diagram of the step of decoding according to the fusion features and the global features to obtain decoded features;

[0051] Figure 6 is a schematic diagram of an image processing model;

[0052] Figure 7 is a schematic diagram of an encoder;

[0053] Figure 8 is a schematic diagram of a multi-scale Transformer module;

[0054] Figure 9 is a schematic diagram of a bone marrow information encoding module;

[0055] Figure 10 is a schematic diagram of a deformable attention gate module;

[0056] Figure 11 is a structural diagram of an image processing device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0057] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0058] It should be noted that although the functional modules are divided in the device schematic diagram, and the logical sequence is shown in the flowchart, in some cases, the steps shown or described can be executed in a manner different from the module division in the device or the sequence in the flowchart. The terms "first", "second", etc. in the specification, claims or above-described drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0059] The pelvic cavity is prone to various malignant cancers and tumors, such as rectal cancer, cervical cancer, and endometrial cancer. Radiotherapy is an important treatment for cancer. In adults, the pelvic bone marrow accounts for about 40% of the total body bone marrow. The larger the irradiated volume and dose of the pelvic bone marrow, the higher the risk of hematological toxicity during radiotherapy and chemotherapy. The hematopoiesis of different bones in the pelvic cavity has obvious heterogeneity. The active hematopoietic area is mainly limited to the bones in the central region of the pelvic cavity, i.e. the functional bone marrow. According to the bone markers of CT images, the bony pelvic cavity is mainly divided into 17 sub-regions, and the functional bone marrow is distributed in 7 of them, i.e. the femoral part from the femoral head to the ischial tuberosity level, the coccyx region from the lower edge of the sacroiliac joint to the lower edge of the coccyx, etc.

[0060] Intensity-modulated radiation therapy (IMRT) technology can maximize the guarantee of target area coverage while well protecting adjacent organs such as bone marrow, bladder and the like. IMRT technology can fully protect functional bone marrow without affecting the quality of the target area and the protection of other organs at risk. Therefore, accurately segmenting the functional bone marrow distribution area with hematopoietic activity from the CT image can effectively guide the design of precise IMRT, and then guide the optimization of radiotherapy plan, fully protect the functional bone marrow area, especially the functional bone marrow with active hematopoietic function.

[0061] The proportion of the pelvic functional bone marrow image bone marrow volume and the area is quite different, and the segmentation belongs to extremely unbalanced categories and the medical image boundary is fuzzy and the gradient is complex.

[0062] The present application provides an image processing method, device, equipment and medium, by acquiring a bone marrow area image, inputting the bone marrow area image into an image processing model; learning local information according to the bone marrow area image to obtain a first feature, and performing three-dimensional voxel-based multi-head self-attention calculation processing based on the bone marrow area image to obtain a second feature, and obtaining a fusion feature according to the first feature and the second feature; extracting a plurality of bone marrow information of different scales according to the fusion feature, encoding the plurality of bone marrow information to obtain a global feature; decoding according to the fusion feature and the global feature to obtain a decoding feature; calculating a loss function value of the image processing model according to the decoding feature, adjusting the parameters of the image processing model according to the loss function value to obtain a trained image processing model; acquiring a to-be-segmented image, inputting the to-be-segmented image into the trained image processing model to obtain a bone marrow area segmentation image; can effectively solve the problems of large morphological change, unbalanced category distribution, small volume size of part of the bone marrow, and fuzzy boundary of functional bone marrow of the strongest and second strongest areas of functional bone marrow, and accurately and effectively segment the functional bone marrow area from the CT image.

[0063] The embodiments of the present application will be further described below with reference to the accompanying drawings.

[0064] The embodiments of the present application provide an image processing method.

[0065] Reference Figure 1 , Figure 1 is a step diagram of the image processing method. An image processing method comprises:

[0066] Step S100, acquiring a bone marrow area image, inputting the bone marrow area image into an image processing model;

[0067] Step S200, learning local information according to the bone marrow region image to obtain a first feature, and performing three-dimensional voxel-based multi-head self-attention calculation processing on the bone marrow region image to obtain a second feature, and obtaining a fusion feature according to the first feature and the second feature;

[0068] Step S300, extracting a plurality of bone marrow information of different scales according to the fusion feature, encoding the plurality of bone marrow information to obtain a global feature;

[0069] Step S400, decoding according to the fusion feature and the global feature to obtain a decoding feature;

[0070] Step S500, calculating a loss function value of the image processing model according to the decoding feature, adjusting parameters of the image processing model according to the loss function value, and obtaining a trained image processing model;

[0071] Step S600, obtaining a to-be-segmented image, inputting the to-be-segmented image into the trained image processing model, and obtaining a bone marrow region segmentation image.

[0072] For step S100, the bone marrow region image is obtained, which is a CT image of the pelvic cavity and contains functional bone marrow regions. The bone marrow region image is a training sample, part of which is labeled by professionals for functional bone marrow regions, and the first and second functional bone marrow regions are outlined. The bone marrow cavity outlined when the CT value is 2000-800 is used as a substitute for the functional bone marrow.

[0073] The bone marrow region image is preprocessed, which includes but is not limited to resampling, data enhancement, windowing processing and feature scaling. Data enhancement includes but is not limited to scaling, horizontal offset, vertical offset, rotation and the like. In order to reduce the amount of calculation, resampling operation is performed on all images, and the size of the resampled image is 96*128*128(D*H*W).

[0074] Referring to Figure 6 The bone marrow region image is input into the image processing model, and the image processing model is a U-net structure, including a network encoding path, a multi-scale Transformer module 120 and a network decoding path.

[0075] The input 101 of the image processing model is processed by the image processing model to obtain the output 102 of the image processing model.

[0076] For step S200, step S200 is executed by the network encoding path. The encoding layer 130 of the network encoding path includes two convolutional blocks of the first layer and three encoder layers 112 connected in sequence.

[0077] The two convolutional blocks of the first layer are stride convolutional blocks 111, which obtain convolutional features by performing convolution, normalization and activation on the bone marrow region image.

[0078] For the three-layer encoder 112 connected in sequence, the input of the first encoder 112 is the convolutional feature, the input of the second encoder 112 is the output of the first encoder 112, and the input of the third encoder 112 is the output of the second encoder 112.

[0079] Referring to Figure 7 For each encoder 112, there are two branches, and the convolutional feature X ∈ R D×H×w×C As the input of the two branches, the input of the first branch is X l , and x l =X. The first branch learns local information by using a 3x3x3 convolutional kernel on the image x l , uses InstanceNorm to finely process each image, and uses the activation function LeakyReLU to reduce information loss to obtain the first feature.

[0080] For the second branch, the input of the second branch is X g , and X g =X. The second branch performs multi-head self-attention calculation based on three-dimensional voxels according to the bone marrow region image to obtain the second feature.

[0081] The input 201 of the encoder is processed by the encoder 112 to obtain the output 202 of the encoder.

[0082] Referring to Figure 2 , the step includes but is not limited to the following steps:

[0083] Step S221, remodeling the bone marrow region image according to three-dimensional voxels to obtain a remodeled image;

[0084] Step S222, based on the query, key, value and relative position encoding of the self-attention mechanism, obtaining the attention value of each three-dimensional voxel according to the remodeled image;

[0085] Step S223, obtaining the second feature according to the attention value of each three-dimensional voxel.

[0086] For step S221, in order to meet the input sequence requirement of the attention mechanism, the bone marrow region image X g is remodeled to , and then remodeled to N V is the number of three-dimensional voxels, N T =S H ×S w ×S DPatch tokens representing each three-dimensional voxel, H , w , D} represents the size of each three-dimensional voxel. Specifically, {S H , S w , S D} is set to {4, 4, 4}

[0087] For step S222, based on the query, key, value and relative position encoding of the self-attention mechanism, the attention value of each three-dimensional voxel is obtained according to the reshaped image. The attention value of the three-dimensional voxel can be represented by the following formula: Where Q, K, represent the query, key, value of the matrix with dimension d k . is the relative position encoding.

[0088] For step S223, according to the attention value of each three-dimensional voxel, the second feature is obtained. Where the second feature can be represented by the following formula: MultiHead(Q, K, V) = Concat(head1, …, head h )W o , MultiHead(Q, K, V) represents the second feature, and the mapping parameter matrix h = 8, and d k = d v = d model / h. Of course in other embodiments, h can take other values according to actual needs.

[0089] Then for the encoder 112, the calculation complexity of the three-dimensional voxel-based multi-head self-attention of one h x w x d block size is: Ω(V-MSA) = 4hwdC 2 + 2S H S w S D hwdC. Through the above attention calculation method, the model can learn the context with a wider view to focus on important areas, and reduce the calculation cost.

[0090] In addition, the fusion feature is obtained according to the first feature and the second feature, including but not limited to the following steps:

[0091] The bone marrow region image is subjected to convolution processing, normalization processing and activation processing to obtain a convolution feature;

[0092] The first feature and the second feature are spliced to obtain an attention block feature;

[0093] The convolutional feature and the attention block feature are fused to obtain a fusion feature.

[0094] The attention block feature is represented as: The fusion feature is represented as: y=F(concat(Z i , LeakyReLU(IN(X i ))). Wherein F represents the convolution operation of the Conv3d convolution block through the 1×1×1 convolution kernel.

[0095] Given an input image X with height H, width W, and depth D, it is gradually encoded into a low-resolution feature representation Through the above encoding, the size of the output image can be reduced to 1 / 8 of the input image (overall stride=8). In this way, the rich 3D local context features and the global image interlaced fusion features are effectively embedded into the fusion features.

[0096] Referring to Figure 8 For step S300, step S300 is performed by the multi-scale Transformer module 120. The input 301 of the multi-scale Transformer module is processed by the multi-scale Transformer module 120 to obtain the output 302 of the multi-scale Transformer module.

[0097] Referring to Figure 3 The fusion feature is used to extract multiple bone marrow information of different scales, including but not limited to the following steps:

[0098] Step S311, performing a pooling operation on the fusion feature corresponding to a first pooling kernel to obtain a first pooling feature;

[0099] Step S312, performing a pooling operation on the fusion feature corresponding to a second pooling kernel to obtain a second pooling feature, the size of the second pooling kernel being smaller than that of the first pooling kernel;

[0100] Step S313, upsampling the first pooling feature by tri-linear interpolation to obtain a first upsampled feature;

[0101] Step S314, taking the first upsampled feature and the second pooling feature as the multiple bone marrow information.

[0102] Specifically, the fused features are input to three paths. For the first path, a three-dimensional convolution operation with a kernel size of 3x3x3 is performed on the fused features, and then a pooling operation with a kernel size of 3x3x3 is performed to obtain a first pooled feature. For the second path, a three-dimensional convolution operation with a kernel size of 3x3x3 is performed on the fused features, and then a pooling operation with a kernel size of 5x5x5 is performed to obtain another first pooled feature. For the third path, a three-dimensional convolution operation with a kernel size of 3x3x3 is performed on the fused features, and then a pooling operation with a kernel size of 2x2x2 is performed to obtain a second pooled feature.

[0103] The two first pooled features are upsampled by trilinear interpolation to obtain two first upsampled features, respectively.

[0104] Referring to Figure 9 , the bone marrow information encoding module encodes the plurality of bone marrow information to obtain the global feature. The input 401 of the bone marrow information encoding module is the plurality of bone marrow information, and the output 402 of the bone marrow information encoding module is the global feature.

[0105] Referring to Figure 4 , in addition, the plurality of bone marrow information is encoded to obtain the global feature, including:

[0106] In step S321, the plurality of bone marrow information is linearly projected to obtain spatial encoding information including a reduced dimension feature and a global mapping projection matrix:

[0107] In step S322, the reduced dimension feature and the global mapping projection matrix are reshaped, and the matrix multiplication is performed on the reshaped reduced dimension feature and the global mapping projection matrix to obtain a node feature mapping. The node feature mapping is input into a Transformer block composed of an MSA layer and an MLP layer to obtain a first sub-global feature.

[0108] In step S323, an inverse projection matrix is created according to the plurality of bone marrow information, and the first sub-global feature and the inverse projection matrix are multiplied to obtain a second sub-global feature transformed to the original space.

[0109] In step S324, the global feature is obtained according to the second sub-global feature and the plurality of bone marrow information.

[0110] For step S321, the linear projection of the plurality of bone marrow information to obtain the spatial encoding information can be represented by the following formula: In this way, the specific position information is added to the N blocks X p in the D-dimensional latent space, where is the block embedding mapping, denotes the position encoding, and two mappings are actually generated, which are the reduced dimension feature and the global mapping projection matrix.

[0111] For step S322, the dimensionality reduction features are reshaped as The global mapping projection matrix is reshaped and transposed as The reshaped dimensionality reduction features and the global mapping projection matrix are subjected to matrix multiplication to obtain node feature mapping.

[0112] The node feature mapping is subjected to L-layer Transformer blocks, which are composed of MSA layers and MLP layers. The output of the Ith layer Transformer block can be expressed as: Z' l = MSA(LN(Z l-1 ))+Z l-1 ), Z l = MLP(LN(Z' l ))+Z' l ), I belongs to L. LN represents a layer normalization operator, and Z l represents the output of the Transformer block, i.e., the first sub-global feature is obtained.

[0113] For step S323, the inverse projection matrix X is created according to the plurality of bone marrow information. d The first sub-global feature is multiplied by the inverse projection matrix X d to obtain the second sub-global feature transformed to the original space.

[0114] The two first up-sampling features are connected, and then spliced with the second pooling feature. Then, the second sub-global feature is added element by element, and finally a stride convolution block 111 is used to obtain the global feature The final obtained global feature contains both global feature information and original feature information, which enables the model to have the ability to fully process long-range context information.

[0115] Due to the characteristics of the pelvic functional bone marrow of variable size, large shape change, fuzzy boundary, and symmetrical distribution of functional bone marrow regions in the human body, a multi-scale Transformer module 120 is added at the end of the encoding path. The multi-scale Transformer module 120 can process images of different receptive fields, improve the segmentation accuracy of fuzzy edges and small volume bone marrow, and then fuse and transmit information of different scales to the decoder, while balancing the calculation amount and the robustness of the model. The multi-scale Transformer module 120 has the advantage of capturing long-distance context relationships, and under the trade-off between computing power and model performance, it can effectively model the long-term dependencies of the image, identify the same type of bone marrow information at different positions, and effectively model the positional relationship of different types of bone marrow through multiple receptive fields, thereby improving the segmentation accuracy of small volume bone marrow.

[0116] For step S400, step S400 is performed by the decoding path.

[0117] Referring to Figure 5 , decoding according to the fusion feature and the global feature to obtain a decoding feature, comprising:

[0118] Step S410, sequentially passing the fusion feature through a plurality of up-sampling modules to obtain a plurality of second up-sampled features;

[0119] Step S420, a plurality of global features and a plurality of second up-sampled features are correspondingly connected by jumping;

[0120] Step S430, extracting key sampling features from key sampling positions of the global feature, and performing matrix multiplication on the key sampling features and the second up-sampled features to obtain the decoding feature.

[0121] For step S410, up-sampling is performed by a step of 2x2x2 and a 3x3x3 convolution kernel and a 1x1x1 convolution.

[0122] For step S420, before splicing the global feature and the second up-sampled feature, an attention guide module is executed to refine the global feature. The attention guide module generates a selection signal to control the importance of features at different spatial positions. The plurality of global features and the plurality of second up-sampled features are correspondingly connected by jumping, and the global feature and the second up-sampled feature are combined, so that the decoding feature contains more information.

[0123] For step S430, a deformable attention gate module 130 is constructed, which aims to extract key information of the encoding layer feature map, gradually suppress the feature response of irrelevant background area, and improve the functional bone marrow segmentation accuracy of smaller volume, and reduce the difference of pelvic functional bone marrow image category imbalance segmentation.

[0124] It can be understood that there are three encoders 112 on the encoding path, and there are three deformable attention gate modules 130 as decoders on the decoding path.

[0125] Referring to Figure 10 , Figure 10 is a schematic diagram of the deformable attention gate module 130. For the deformable attention gate module 130, key sampling features are extracted from key sampling positions of the global feature, and matrix multiplication is performed on the key sampling features and the second up-sampled features to obtain the decoding feature.

[0126] wherein, is a feature representation of a query matrix q of the deformable attention gate module 130, is a normalized three-dimensional coordinate of a reference point, is a global feature, K is the number of key sampling positions, and ∧(zq ) iqk ∈[0, 1] is the attention weight of the deformable attention gate module 130, is the sampling offset of the k-th key sampling position, and the calculation formula of the feature representation of the i-th attention head is: ∧(z q ) iqk and Δp iqk are the query feature representation z q obtained through a linear projection layer.

[0127] Then the decoding feature output by the deformable attention gate module 130 is: where H is the number of attention heads, and Φ is a linear projection layer calculation for weighting and aggregating all attention heads, is the feature map of the encoder 112, is the feature map of the previous decoder.

[0128] For step S500, the loss function value includes a Dice coefficient loss function value and a cross-entropy loss function value.

[0129] Then the loss function value is represented by the following formula: g i represents the ground truth, p i represents the predicted probability that the pixel x belongs to the class i, and M is the number of classes of the image processing model.

[0130] The ground truth segmentation map is down-sampled to the prediction map size of the typical stage, so as to match the resolution of the prediction map.

[0131] For different resolutions, the sum of the loss functions at three resolutions is used as the final loss function, that is, where α {1,2,3} represents the loss coefficient at different resolutions. α {1,2,3} is reduced by half after the resolution is reduced by half, so Finally, all the weight factors are normalized to 1.

[0132] Through the image processing of the image to be segmented in step S600, it can be concluded that the local and global information is captured by using the convolution-Attention fusion encoding, the same type of bone marrow is located, and then the multi-scale Transformer module 120 is used for fine segmentation. In the upsampling process, the extraction of redundant information is suppressed to reduce the wrong segmentation and segmentation. The features in the encoding layer and the feature information in the decoding layer are ingeniously fused to generate a new up-sampling feature map, thereby generating a final good segmentation effect. Therefore, for the functional bone marrow with extremely small volume and large shape change, the image processing model can identify and accurately segment the results, which is beneficial to the segmentation of the edge of the foreground region and improves the segmentation accuracy.

[0133] Embodiments of the present application provide an image processing device.

[0134] Reference Figure 11 , Figure 11 is a structural diagram of an image processing device. The image processing device comprises an input module 10, an encoding module 20, a global feature acquisition module 30, a decoding module 40, a parameter adjustment module 50 and a segmentation module 60.

[0135] The input module 10 is configured to obtain a bone marrow region image and input the bone marrow region image into an image processing model. The encoding module 20 is configured to learn local information according to the bone marrow region image to obtain first features, and perform three-dimensional voxel-based multi-head self-attention calculation processing on the bone marrow region image to obtain second features, and obtain fusion features according to the first features and the second features. The global feature acquisition module 30 is configured to extract a plurality of bone marrow information of different scales according to the fusion features, encode the plurality of bone marrow information, and obtain global features. The decoding module 40 is configured to decode the fusion features and the global features to obtain decoding features. The parameter adjustment module 50 is configured to calculate a loss function value of the image processing model according to the decoding features, adjust parameters of the image processing model according to the loss function value, and obtain a trained image processing model. The segmentation module 60 is configured to obtain an image to be segmented, input the image to be segmented into the trained image processing model, and obtain a bone marrow region segmentation image.

[0136] It can be understood that the contents in the image processing method embodiments are all applicable to the present image processing device embodiment, the functions specifically implemented by the image processing device embodiment are the same as those of the image processing method embodiments, and the beneficial effects achieved by the image processing device embodiment are also the same as those achieved by the image processing method embodiments.

[0137] Embodiments of the present application provide an electronic device. The electronic device comprises a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program to implement the image processing method as described above.

[0138] The embodiment of the present application provides a computer readable storage medium. The computer readable storage medium stores computer executable instructions for performing the image processing method as described above.

[0139] The memory, as a non-transitory computer readable storage medium, can be used to store non-transitory software programs and non-transitory computer executable programs, such as the image processing in the above embodiment of the present application. The processor realizes the image processing in the above embodiment of the present application by running the non-transitory software programs and programs stored in the memory.

[0140] The memory can include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required by a function; and the data storage area can store data required for performing the image processing in the above embodiment of the present application. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and these remote memories can be connected to the terminal through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0141] Those of ordinary skill in the art will appreciate that all or certain steps, systems of the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Certain physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software can be distributed on computer readable media, which can include computer storage media (or non-transitory media) and communication media (or transitory media). As is well known to those of ordinary skill in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computer. Further, it is common in the art to refer to communication media, which typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism, and includes any information delivery media. In the foregoing description of the present specification, reference has been made to descriptive terms such as "one embodiment / implementation," "another embodiment / implementation," or "some embodiments / implementations," etc. It is understood that such descriptive terms are not intended to necessarily describe only a single or certain embodiments / implementations. Rather, such descriptive terms are intended to describe some or certain embodiments / implementations, but not necessarily all embodiments / implementations. Moreover, such descriptive terms are not intended to exclude or otherwise preclude the inclusion of certain amounts or aspects of a feature, structure, material, or characteristic in some or even all embodiments / implementations. Rather, such descriptive terms are intended to describe some or certain embodiments / implementations, but not necessarily all embodiments / implementations.

[0142] While the embodiments of the present application have been shown and described, it is to be understood that various modifications, substitutions, replacements and variations can be made to the embodiments without departing from the principles and spirit of the present application.

Claims

1. An image processing method, characterized by, The method comprises the following steps: obtain a bone marrow region image, and input the bone marrow region image into an image processing model; learn local information according to the bone marrow region image to obtain first features, and perform three-dimensional voxel-based multi-head self-attention calculation processing on the bone marrow region image to obtain second features, and obtain fusion features according to the first features and the second features; extract multiple bone marrow information of different scales according to the fusion features, encode the multiple bone marrow information to obtain global features; decode the fusion features and the global features to obtain decoded features; calculate a loss function value of the image processing model according to the decoded features, adjust parameters of the image processing model according to the loss function value, and obtain a trained image processing model; obtain a to-be-segmented image, input the to-be-segmented image into the trained image processing model, and obtain a bone marrow region segmentation image; wherein, the fusion features obtained according to the first features and the second features comprise: perform convolution processing, normalization processing and activation processing on the bone marrow region image to obtain convolution features; concatenate the first features and the second features to obtain attention block features; fuse the convolution features and the attention block features to obtain fusion features; the multiple bone marrow information of different scales extracted according to the fusion features comprises: perform a pooling operation on the fusion features corresponding to a first pooling kernel to obtain first pooling features; perform a pooling operation on the fusion features corresponding to a second pooling kernel to obtain second pooling features, the size of the second pooling kernel being smaller than that of the first pooling kernel; perform upsampling on the first pooling features by trilinear interpolation to obtain first upsampled features; use the first upsampled features and the second pooling features as the multiple bone marrow information.

2. The image processing method of claim 1, wherein, the second features obtained by performing three-dimensional voxel-based multi-head self-attention calculation processing on the bone marrow region image comprise: reformat the bone marrow region image according to three-dimensional voxels to obtain a reformatted image; obtain attention values of each three-dimensional voxel according to query, key, value and relative position encoding of the self-attention mechanism based on the reformatted image; obtain the second features according to the attention values of each three-dimensional voxel.

3. The image processing method of claim 1, wherein, the global features obtained by encoding the multiple bone marrow information comprise: perform linear projection on the multiple bone marrow information to obtain spatial encoding information comprising dimension reduction features and global mapping projection matrices; reformat the dimension reduction features and the global mapping projection matrices, perform matrix multiplication on the reformatted dimension reduction features and the global mapping projection matrices to obtain node feature mapping, input the node feature mapping into a Transformer block composed of an MSA layer and an MLP layer to obtain first sub-global features; create an inverse projection matrix according to the multiple bone marrow information, multiply the first sub-global features and the inverse projection matrix to obtain second sub-global features transformed to an original space; obtain the global features according to the second sub-global features and the multiple bone marrow information.

4. The image processing method of claim 1, wherein, the decoded features obtained by decoding the fusion features and the global features comprise: The fusion features are sequentially subjected to a plurality of up-sampling modules for up-sampling to obtain a plurality of second up-sampling features; The plurality of global features and the plurality of second up-sampling features are one-to-one correspondingly connected by jumping; Key sampling features are extracted from key sampling positions of the global features, and the key sampling features and the second up-sampling features are subjected to matrix multiplication to obtain the decoding features.

5. The image processing method of claim 1, wherein, The loss function value includes a Dice coefficient loss function value and a cross-entropy loss function value.

6. An image processing apparatus characterized by comprising: It comprises: An input module is configured to obtain a bone marrow region image and input the bone marrow region image into an image processing model; An encoding module is configured to learn local information from the bone marrow region image to obtain first features, and perform three-dimensional voxel-based multi-head self-attention calculation processing on the bone marrow region image to obtain second features, and obtain fusion features from the first features and the second features; A global feature acquisition module is configured to extract a plurality of bone marrow information of different scales from the fusion features, encode the plurality of bone marrow information, and obtain global features; A decoding module is configured to decode the fusion features and the global features to obtain decoding features; A parameter adjustment module is configured to calculate a loss function value of the image processing model according to the decoding features, adjust parameters of the image processing model according to the loss function value, and obtain a trained image processing model; A segmentation module is configured to obtain an image to be segmented, input the image to be segmented into the trained image processing model, and obtain a bone marrow region segmentation image; The fusion features are obtained from the first features and the second features, comprising: The bone marrow region image is subjected to convolution processing, normalization processing and activation processing to obtain convolution features; The first features and the second features are spliced to obtain attention block features; The convolution features and the attention block features are fused to obtain fusion features; The fusion features are subjected to a first pooling operation corresponding to a first pooling kernel to obtain a first pooled feature; The fusion features are subjected to a second pooling operation corresponding to a second pooling kernel to obtain a second pooled feature, and the size of the second pooling kernel is smaller than that of the first pooling kernel; The first pooled feature is up-sampled by tri-linear interpolation to obtain a first up-sampled feature; The first up-sampled feature and the second pooled feature are used as a plurality of bone marrow information. A memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the image processing method of any one of claims 1-5 when executing the computer program.

7. An electronic device comprising: Computer executable instructions are stored, and the computer executable instructions are used to execute the image processing method of any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, ​

Images