Medical image segmentation method and device, electronic equipment and storage medium

Abstract

The invention relates to the field of image processing, and provides a medical image segmentation method and device, electronic equipment and a storage medium, and the method comprises the steps: carrying out the feature extraction of a to-be-processed medical image, so as to obtain a first feature map of the to-be-processed medical image; performing graph-based convolution processing on the first feature graph to obtain a second feature graph; performing spatial feature extraction based on an attention mechanism on the first feature map to obtain a third feature map; performing graph reconstruction according to the second feature graph and the third feature graph to obtain a reconstructed feature graph; and performing up-sampling processing on the reconstructed feature map to obtain a segmentation result. According to the medical image segmentation method disclosed by the invention, the segmentation accuracy and precision of the medical image are improved.

Description

Background Art

[0002] Organ segmentation of human medical images can provide an important basis for preoperative evaluation and surgical guidance, and has important practical significance. In related technologies, while retaining the high-resolution detail information of the image, medical image segmentation cannot extract complete voxel semantic features, resulting in low accuracy of the segmentation results; at the same time, due to the low contrast of medical images, overlap between organs, and large differences in organ size, the segmentation process is greatly disturbed, affecting the segmentation accuracy.

[0003] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0004] The purpose of the present disclosure is to provide a medical image segmentation method and device, electronic equipment, and storage medium, thereby improving the accuracy and precision of medical image segmentation at least to a certain extent.

[0005] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0006] According to a first aspect of the present disclosure, a medical image segmentation method is provided, the method comprising: performing feature extraction on a medical image to be processed to obtain a first feature map of the medical image to be processed; performing graph-based convolution processing on the first feature map to obtain a second feature map; performing spatial feature extraction based on an attention mechanism on the first feature map to obtain a third feature map; performing graph reconstruction based on the second feature map and the third feature map to obtain a reconstructed feature map; and performing upsampling processing on the reconstructed feature map to obtain a segmentation result.

[0007] Optionally, feature extraction is performed on the medical image to be processed, including: extracting features from the medical image to be processed through N coding units connected in series in sequence; wherein, the feature extraction corresponding to each coding unit includes: performing depth-separable convolution on the input of the current coding unit; performing first channel convolution and downsampling on the depth-separable convolution result to obtain the output corresponding to the current coding unit; wherein, the input of the first coding unit is the medical image to be processed, the inputs of the second to Nth coding units are the outputs of the previous coding units connected in series therewith, and the output of the Nth coding unit is the first feature map.

[0008] Optionally, a depthwise separable convolution is performed on the input of the current coding unit, including: performing a dilated separable convolution at multiple scales and multiple dilation rates on the input of the current coding unit to obtain multiple dilated convolution results; and fusing the multiple dilated convolution results to obtain a depthwise separable convolution result corresponding to the current coding unit.

[0009] Optionally, the first feature map is subjected to graph-based convolution processing, including: performing graph interaction processing of different feature spaces on the first feature map to construct an adjacency matrix; performing a second channel convolution on the first feature map to obtain a third feature matrix; and performing graph convolution processing on the third feature matrix using the adjacency matrix.

[0010] Optionally, the first feature map is subjected to graph interaction processing of different feature spaces, including: performing channel compression on the first feature map to obtain a first feature matrix corresponding to the first feature space and a second feature matrix corresponding to the second feature space; performing matrix operations on the first feature matrix and the second feature matrix to obtain an interaction matrix; and performing nonlinear processing on the interaction matrix to obtain an adjacency matrix.

[0011] Optionally, the graph convolution processing includes: performing graph convolution processing on the third feature matrix based on the adjacency matrix through M graph convolution units connected in series in sequence; wherein, for the first graph convolution unit, performing a graph convolution processing on the third feature matrix once using the adjacency matrix and the weight parameters of the first graph convolution unit; for the second to Mth graph convolution units, performing a second graph convolution processing on the graph convolution result of the previous graph convolution unit connected in series with the current graph convolution unit using the adjacency matrix and the weight parameters of the current graph convolution unit.

[0012] Optionally, spatial feature extraction based on the attention mechanism is performed on the first feature map, including: performing three-dimensional convolution processing based on the attention mechanism on the first feature map to obtain an attention matrix; and dimensionally adjusting the attention matrix to obtain a third feature map, wherein the dimension of the third feature map is the same as that of the second feature map.

[0013] Optionally, the method also includes: calculating a loss function based on the segmentation result; and adjusting parameters of feature extraction, graph-based convolution processing, attention mechanism-based spatial feature extraction, and upsampling processing with the goal of minimizing the loss function.

[0014] According to a second aspect of the present disclosure, a medical image segmentation device is provided, including an image determination module, a similarity determination module and a transmission module, wherein: the image determination module is configured to determine a corresponding first image from an original image of a current frame according to a user's gaze area; and based on the original image of the current frame, determine a second image corresponding to the current frame, wherein the resolution of the second image is lower than that of the first image; the similarity determination module is configured to determine the similarity between the second image corresponding to the current frame and the second image corresponding to the previous frame; the transmission module is configured to transmit a target image according to the similarity; wherein, when the similarity is greater than or equal to a preset threshold, the target image is the first image.

[0015] According to a third aspect of the present disclosure, a computer-readable medium is provided, on which a computer program is stored. When the program is executed by a processor, the medical image segmentation method in the above-mentioned embodiment is implemented.

[0016] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs, which, when the one or more programs are executed by the one or more processors, enables the one or more processors to implement the medical image segmentation method as in the above-mentioned embodiment.

[0017] It can be seen from the above technical solutions that the medical image segmentation method and device in the exemplary embodiments of the present disclosure have at least the following advantages and positive effects:

[0018] The medical image segmentation method disclosed in the present invention can, on the one hand, perform graph-based convolution processing on the extracted first feature map to obtain a second feature map. The graph-based convolution processing can gather local information and global information of the medical image to be processed, and consider global distribution information while retaining detail information, thereby improving segmentation accuracy. On the other hand, the extracted first feature map is subjected to spatial feature extraction based on the attention mechanism to obtain a third feature map, and graph reconstruction is performed based on the second feature map and the third feature map to obtain a reconstructed feature map; considering the invariance of the target spatial position in the medical image, the attention mechanism is introduced to extract spatial position information, which can suppress the error between segmentation categories and further improve the segmentation accuracy.

[0019] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0021] Figure 1 The flowchart of the medical image segmentation method according to an embodiment of the present disclosure is schematically shown.

[0022] Figure 2 The figure schematically shows a flow chart of a feature extraction process according to an embodiment of the present disclosure.

[0023] Figure 3 The figure schematically shows a flow chart of a feature extraction process in a coding unit according to an embodiment of the present disclosure.

[0024] Figure 4 A flowchart of a graph-based convolution processing process according to an embodiment of the present disclosure is schematically shown.

[0025] Figure 5 A flowchart of a control feature extraction process based on an attention mechanism according to another embodiment of the present disclosure is schematically shown.

[0026] Figure 6 The figure schematically shows a flow chart of an upsampling process according to another embodiment of the present disclosure.

[0027] Figure 7 A schematic diagram of the processing flow of a medical image segmentation model according to another embodiment of the present disclosure is schematically shown.

[0028] Figure 8 The structural block diagram of a medical image segmentation device according to an embodiment of the present disclosure is schematically shown.

[0029] Fig. 9 A module diagram of an electronic device according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0030] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be more comprehensive and complete and will fully convey the concept of the example embodiments to those skilled in the art.

[0031] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the present disclosure.

[0032] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0033] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0034] Currently, there are few annotated images that can be used as training samples in medical images, and the size of a single medical image is large. There are many types of organs to be segmented, with large size differences and possible overlap, so the segmentation accuracy is low and the speed is slow, which seriously affects the user experience of the image. This paper proposes a medical image segmentation method based on a medical image segmentation model.

[0035] In one embodiment of the present disclosure, a medical image segmentation method is proposed. Figure 1 A schematic diagram of the process of segmenting a medical image is shown, Figure 1 As shown, the medical image segmentation method includes at least the following steps:

[0036] Step S110 , performing feature extraction on the medical image to be processed to obtain a first feature map of the medical image to be processed.

[0037] Step S120, performing graph-based convolution processing on the first feature map to obtain a second feature map; performing attention mechanism-based spatial feature extraction on the first feature map to obtain a third feature map.

[0038] Step S130, performing image reconstruction according to the second feature map and the third feature map to obtain a reconstructed feature map.

[0039] Step S140, upsampling the reconstructed feature map to obtain a segmentation result.

[0040] The medical image segmentation method of the disclosed embodiment can, on the one hand, perform graph-based convolution processing on the extracted first feature map to obtain a second feature map. The graph-based convolution processing can gather local information and global information of the medical image to be processed, and consider global distribution information while retaining detail information, thereby improving segmentation accuracy. On the other hand, perform spatial feature extraction based on the attention mechanism on the extracted first feature map to obtain a third feature map, and perform graph reconstruction based on the second feature map and the third feature map to obtain a reconstructed feature map; considering the invariance of the target spatial position in the medical image, the attention mechanism is introduced to extract spatial position information, which can suppress the error between segmentation categories and further improve segmentation accuracy.

[0041] In order to make the technical solution of the present disclosure clearer, each step of the medical image segmentation method is described below.

[0042] In step S110 , feature extraction is performed on the medical image to be processed to obtain a first feature map of the medical image to be processed.

[0043] In this exemplary embodiment, the medical images to be processed include but are not limited to CT images, MRI images, and X-ray images, and the parts that can be photographed can be the abdominal cavity, or other parts such as the chest cavity, pelvic cavity, or head. The medical images to be processed can be three-dimensional images (3D images), and for medical 3D images, they usually contain all the organ conditions of the photographed body parts, with a large amount of data and many targets. Taking the abdominal three-dimensional image as an example, a 3D image includes at least images of multiple organs such as the liver, gallbladder, pancreas, spleen, and kidneys, and this embodiment is to quickly and accurately distinguish the areas where each organ is located from the 3D image to achieve organ segmentation of the 3D image.

[0044] In this exemplary embodiment, feature extraction can be performed on the medical image to be processed to obtain a first feature map, i.e., useful information of the medical image to be processed. Exemplarily, feature extraction of different dimensions can be performed by continuously performing convolution downsampling. For example, a 3D convolution model can be used to perform multiple downsampling to extract high and low dimensional, local and global features.

[0045] In some embodiments, considering the large number of parameters and the large amount of computing resources consumed by 3D convolution, a depthwise separable convolution is used for feature extraction to reduce the number of parameters. The medical image segmentation model may include N encoding units connected in series, as shown in FIG. Figure 2As shown, feature extraction can be performed on the medical image to be processed through N coding units connected in series in sequence; wherein, the feature extraction corresponding to each coding unit includes: performing depth-separable convolution on the input of the current coding unit; performing first channel convolution and downsampling on the depth-separable convolution result to obtain the output corresponding to the current coding unit; wherein, the input of the first coding unit (coding unit 1) is the medical image to be processed, the input of the second coding unit (coding unit 2) to the Nth coding unit (coding unit N) is the output of the previous coding unit connected in series with it, and the output of the Nth coding unit is the first feature map.

[0046] In this example implementation, N coding units are connected in series in sequence, and N different dimensions of the input medical image to be processed are sequentially extracted, and the coding unit extracts local detail features based on the features extracted by the previous coding unit connected in series with it, thereby achieving a gradual and in-depth extraction of local details and ensuring the accuracy of the local detail information of the extracted first feature map. Separable convolution is to apply each convolution kernel (the number of convolution kernels is the same as the depth) to each channel for channel convolution, and the first channel convolution can be the convolution result of combining different depths using 1×1 convolution.

[0047] In some embodiments, considering that the input 3D image may be large in size, more convolution downsampling times are required, resulting in low efficiency of the feature extraction process. This example introduces dilated convolution, which specifically includes: performing dilated separable convolutions at multiple scales and multiple expansion rates on the input of the current coding unit, and obtaining multiple dilated convolution results; fusing multiple dilated convolution results to obtain the depth-separable convolution result corresponding to the current coding unit.

[0048] In this example implementation, convolution kernels of different depths may be used to perform convolution processing of different scales, and each scale may correspond to a dilation rate. The fusion processing may be a summation or a weighted summation processing, which is not limited in this example. For example, Figure 3As shown in FIG. 1 , the size of the input medical image to be processed is C×W×H (number of channels×width×height), and a depth-separable convolution is used for channel convolution. For example, a 3×3 convolution kernel is used, and three-way dilated convolutions with expansion rates of 1, 2, and 3 are used to convolve the input, that is, 3×3 convolution (expansion rate is 1, i.e., no expansion), 5×5 convolution (expansion rate is 2), and 7×7 convolution (expansion rate is 3) are used respectively; then the three-way convolution results are added for channel fusion, and the channel fusion results enter the 1×1 convolution for channel adjustment to obtain a feature map with half the number of channels of the input (C / 2). The feature map is subjected to downsampling operations (such as Max Pooling, normalization processing BN, Relu activation processing, etc.) to obtain an output feature map with a size of C / 2×W / 2×H / 2. The above is the feature extraction process within a coding unit. By gradually performing the above process through multiple coding units, a first feature map with a larger field of view (C / 8×W / 8×H / 8) can be obtained. It can be seen that this example obtains more than twice the feature space field of view with the number of parameters of a 3D convolution. Through three encoding units, an 8-fold field of view size relative to the 3D convolution can be obtained, achieving more global feature extraction with fewer convolution downsampling times.

[0049] This example addresses the problem that traditional 3D convolution requires multiple downsampling for high- and low-dimensional, local-global feature extraction, which is inefficient. By combining deep separation convolution with multi-way dilated convolution, multi-scale separation dilated convolution is formed for feature extraction, achieving fewer downsampling times, obtaining a larger field of view, and improving model processing efficiency.

[0050] In step S120, the first feature map is subjected to graph-based convolution processing to obtain a second feature map; and the first feature map is subjected to attention mechanism-based spatial feature extraction to obtain a third feature map.

[0051] In this example implementation, the second feature map is used to indicate the aggregation result of the local neighbor features and the remote context information, and the third feature map is used to indicate the spatial position relationship of each target. The local neighbor features and the remote context information in the first feature map can be learned through graph-based convolution processing, and the spatial position relationship information of each target can be obtained through spatial feature extraction based on the attention mechanism.

[0052] Exemplarily, performing graph-based convolution processing on the first feature map includes the following steps:

[0053] In the first step, the first feature graph is processed interactively in different feature spaces to construct an adjacency matrix.

[0054] In this example implementation, the adjacency matrix can be automatically generated by graph interaction of different feature spaces. For example, the first feature graph is firstly subjected to channel compression to obtain a first feature matrix corresponding to the first feature space and a second feature matrix corresponding to the second feature space; then, the first feature matrix and the second feature matrix are subjected to matrix operation to obtain an interaction matrix; and the interaction matrix is ​​subjected to nonlinear processing to obtain an adjacency matrix. Figure 4 As shown in Figure 1, for the first feature map (C / 8, W / 8, H / 8) of the graph convolution input, after the channel compression of 1×1 convolution, the feature matrices F1 and F2 of the two feature spaces are obtained. After the transposition operation of F1, matrix operations (such as matrix multiplication) are performed with F2 to realize the graph interaction of different feature spaces, and the feature interaction graph, that is, the interaction matrix of the graph space, is obtained. Then, the adjacency matrix A is obtained by nonlinear processing through the sigmoid function, A∈R n ×n , n=WH / 64, number of channels s=C / 8. The present invention uses a dynamic generation method to establish an adjacency matrix. Compared with the traditional method of taking each point of the feature graph as a node and calculating the similarity between nodes to establish an adjacency matrix, the amount of calculation in the adjacency matrix construction process is greatly reduced.

[0055] In the second step, the second channel convolution is performed on the first feature map to obtain the third feature matrix.

[0056] In this example embodiment, the second channel convolution is used to extract features in a feature space different from the adjacency matrix, for example, Figure 4 As shown, 1×1 convolution can be used for the second channel convolution to obtain the third feature matrix X, X∈R n×s .

[0057] The third step is to use the adjacency matrix to perform graph convolution on the third feature matrix.

[0058] In this example implementation, the medical image segmentation model may include M graph convolution units connected in series, and the graph convolution processing may be performed on the third feature matrix based on the adjacency matrix through the M graph convolution units connected in series; wherein, for the first graph convolution unit, the third feature matrix is ​​subjected to a graph convolution processing using the adjacency matrix and the weight parameter of the first graph convolution unit. Specifically, the formula for a graph convolution processing is as follows:

[0059]

[0060] in, represents the output of the first graph convolution unit, I represents the identity matrix, A represents the adjacency matrix, X represents the third feature matrix, and W 1 Represents the weight parameter of the first graph convolution unit, W 1 ∈R s×s .

[0061] For the second graph convolution unit to the Mth graph convolution unit, the adjacency matrix and the weight parameters of the current graph convolution unit are used to perform secondary graph convolution processing on the graph convolution result of the previous graph convolution unit connected in series with the current graph convolution unit. Specifically, the formula for secondary convolution processing is as follows:

[0062]

[0063] in, represents the output of the mth graph convolution unit, W m represents the weight parameter of the mth graph convolution unit, W m ∈R s×s .

[0064] like Figure 4 As shown, this example goes through two graph convolution units and uses two graph convolutions to obtain right After the reshape function is processed, the coordinate space feature map is restored to obtain the second feature map. The number of graph convolution units in this example is generally no more than 3. The feature coordinate space is mapped to the graph space through the processing of the graph convolution unit. The local features and remote features are efficiently aggregated through the graph convolution calculation to achieve the learning and reasoning of the global features.

[0065] Exemplarily, the process of performing spatial feature extraction based on the attention mechanism on the first feature map may include the following steps: performing a three-dimensional convolution process based on the attention mechanism on the first feature map to obtain an attention matrix. Dimensionally adjusting the attention matrix to obtain a third feature map.

[0066] In this example implementation, three different three-dimensional convolution processes are performed to obtain the Q (query) matrix, K (key) matrix, and V (value) matrix corresponding to the self-attention mechanism, and the weight distribution of the V matrix is ​​achieved by modeling the correlation between the Q matrix and the K matrix. Dimension adjustment can be achieved by a 1×1×1 three-dimensional convolution kernel so that the dimension of the third feature map after dimension adjustment is the same as the dimension of the second feature map.

[0067] For example, the spatial extraction process is Figure 5As shown in the figure, the first feature map of the input is (C / 8, W / 8, H / 8). After being processed by three different 1×1×1 3D convolution operators, the Q matrix, K matrix, and V matrix are extracted. After reshaping, the corresponding tensor sizes are (C / 16, W / 8×H / 8), (C / 16, W / 8×H / 8), and (C / 8, W / 8×H / 8). The Q matrix is ​​resized to (W / 8×H / 8, C / 16) after transposition. The transpose of Q and K are matrix operated to obtain a correlation matrix size of (W / 8×H / 8, W / 8×H / 8). After being processed by the softmax function, the matrix operation is performed with V to obtain an attention matrix with a size of (C / 8, W / 8×H / 8). Reshaping obtains a tensor (C / 8, W / 8, H / 8) with the same size as the input. After obtaining the attention matrix, the dimension can also be adjusted through a 1×1×1 3D convolution to ensure that the dimension of the third feature map is the same as that of the second feature map.

[0068] In this example, based on the invariance of the spatial positions of multiple targets, a spatial attention mechanism is used to enhance the prior guidance for the segmentation of multiple targets, which can improve the segmentation accuracy.

[0069] In step S130, a graph reconstruction is performed according to the second feature graph and the third feature graph to obtain a reconstructed feature graph.

[0070] In this example implementation, the third feature map containing the target spatial position feature can be fused with the second feature map, such as performing a dot product operation on the second feature map and the third feature map, to complete the image reconstruction and obtain a reconstructed feature map.

[0071] In step S140, upsampling is performed on the reconstructed feature map to obtain a segmentation result.

[0072] In this example implementation, the upsampling process is set corresponding to the downsampling process, and is used to increase the dimension of the feature map whose dimension is reduced by the downsampling process after the upsampling process, until it is restored to the original size of the medical image to be processed. Figure 6 As shown, the reconstructed feature map (C / 8, W / 8, H / 8) undergoes three deconvolution or upsampling operations to obtain a feature map of the restored size (C, W, H), and then the softmax layer is used to segment each target, and the target segmentation map in the medical image to be processed is obtained, that is, the segmentation result. In this example, the number of targets is preset and can be set according to the category or shooting position of the medical image. For example, the number of targets is set to the number of organs in the medical image, which is not limited in this example.

[0073] refer to Figure 7The medical image segmentation method disclosed in the present invention can be implemented by a trained medical image segmentation model, and its processing process includes: inputting the medical image to be processed (abdominal three-dimensional image) into the model, obtaining a first feature map S1 through feature extraction, performing graph-based convolution processing and attention mechanism-based spatial feature extraction on the first feature map S1, performing graph reconstruction on the convolution processing result and the extracted spatial features to obtain a reconstructed feature map S2, performing upsampling processing on the reconstructed feature map, and the model outputs a segmentation result (abdominal organ segmentation result).

[0074] In the above embodiment, the medical image segmentation model is obtained after training, so the method may also include a training step for the medical image segmentation model: obtaining training samples, the training samples may be manually annotated medical images, and a segmentation model may be trained for medical images of different body parts respectively, or a total segmentation model may be trained, which is not limited in this example. The specific training process is similar to the above processing process and will not be repeated here. For the segmentation result output by the model in each training round, it is necessary to calculate the loss function based on the segmentation result; with the goal of minimizing the loss function, the parameters of the segmentation model are adjusted, and the parameters of feature extraction, graph-based convolution processing, attention-based spatial feature extraction, and upsampling processing may be adjusted respectively.

[0075] In this example, the weighted processing result of the first loss function and the second loss function can be used as the total loss. The first loss function can be a Dice loss function, and the second loss function can be a focal loss function. The calculation formula of the total loss function L is as follows:

[0076] L=λL Dice +(1-λ)L fl

[0077] Among them, λ is the loss weight, is a learnable parameter, and L Dice represents the Dice loss function, which evaluates the loss by the similarity between the predicted value and the true value; L focal Represents the focal loss function. By increasing the weight of hard-to-classify samples in the loss function, the loss function tends to favor hard-to-classify samples, which helps to improve the accuracy of hard-to-classify samples. Taking into account the difficulty of sample classification, it helps to improve the segmentation accuracy of small objects.

[0078] On the one hand, the present invention retains the intrinsic connection of the spatial structure of multiple targets by inputting complete three-dimensional medical images, avoiding the destruction of the organ anatomical structure by the traditional batch input or block input process; on this basis, the target spatial features are extracted through the attention mechanism as a priori guidance of the unchanged anatomical position to improve the segmentation accuracy. On the other hand, by combining the deep separable convolution with the multi-way hole convolution, while reducing the number of parameters in the three-dimensional image feature extraction process, the spatial field of view is expanded, and the global features can be extracted with a smaller number of downsampling times, reducing the number of model parameters, preventing overfitting, improving the generalization ability of the model, and reducing the consumption of computing resources. Furthermore, by gathering local information and global information through graph-based convolution processing, the method of dynamically generating adjacency matrices is adopted to reduce the amount of calculation in the graph convolution process. In addition, there is no complex jump connection in the segmentation model of the present invention, the model structure is simple, and the model operation efficiency can be improved.

[0079] The following describes an apparatus embodiment of the present disclosure, which can be used to execute the medical image segmentation method described above. For details not disclosed in the apparatus embodiment of the present disclosure, please refer to the embodiment of the medical image segmentation method described above.

[0080] Figure 8 The block diagram schematically shows a medical image segmentation device according to an embodiment of the present disclosure.

[0081] Reference Figure 8 As shown, a medical image segmentation device 800 according to an embodiment of the present disclosure includes: an encoding module 810, a feature interaction module 820, a graph reconstruction module 830 and a decoding module 840. Specifically: the encoding module 810 is configured to perform feature extraction on the medical image to be processed to obtain a first feature map of the medical image to be processed; the feature interaction module 820 is configured to perform graph-based convolution processing on the first feature map to obtain a second feature map; perform spatial feature extraction based on the attention mechanism on the first feature map to obtain a third feature map; the graph reconstruction module 830 is configured to perform graph reconstruction based on the second feature map and the third feature map to obtain a reconstructed feature map; the decoding module 840 is configured to perform upsampling processing on the reconstructed feature map to obtain a segmentation result.

[0082] In some embodiments of the present disclosure, based on the aforementioned scheme, the encoding module 810 is further configured to: extract features from the medical image to be processed through N encoding units connected in series in sequence; wherein each encoding unit is configured to: perform depth-separable convolution on the input of the current encoding unit; perform first channel convolution and downsampling on the depth-separable convolution result to obtain the output corresponding to the current encoding unit; wherein the input of the first encoding unit is the medical image to be processed, the input of the second to Nth encoding units is the output of the previous encoding unit connected in series therewith, and the output of the Nth encoding unit is the first feature map.

[0083] In some embodiments of the present disclosure, based on the aforementioned scheme, the encoding unit is further configured to: perform dilated separable convolution on the input at multiple scales and multiple expansion rates to obtain multiple dilated convolution results; and fuse the multiple dilated convolution results to obtain a depth-separable convolution result corresponding to the current encoding unit.

[0084] In some embodiments of the present disclosure, based on the aforementioned scheme, the feature interaction module 820 includes a graph convolution module 821, and the graph convolution module 821 is configured to: perform graph interaction processing of different feature spaces on the first feature graph to construct an adjacency matrix; perform a second channel convolution on the first feature graph to obtain a third feature matrix; and perform graph convolution processing on the third feature matrix using the adjacency matrix.

[0085] In some embodiments of the present disclosure, based on the aforementioned scheme, the graph convolution module 821 is also configured to: perform channel compression on the first feature map to obtain a first feature matrix corresponding to the first feature space and a second feature matrix corresponding to the second feature space; perform matrix operations on the first feature matrix and the second feature matrix to obtain an interaction matrix; perform nonlinear processing on the interaction matrix to obtain an adjacency matrix.

[0086] In some embodiments of the present disclosure, based on the aforementioned scheme, the graph convolution module 821 includes: M graph convolution units connected in series in sequence, configured to perform graph convolution processing on the third feature matrix based on the adjacency matrix; wherein, the first graph convolution unit is configured to perform a graph convolution processing on the third feature matrix using the adjacency matrix and the weight parameters of the first graph convolution unit; the second graph convolution unit to the Mth graph convolution unit are configured to perform a secondary graph convolution processing on the graph convolution result of the previous graph convolution unit connected in series with the current graph convolution unit using the adjacency matrix and the weight parameters of the current graph convolution unit.

[0087] In some embodiments of the present disclosure, based on the aforementioned scheme, the feature interaction module 820 also includes an attention module 822, and the attention module 822 is configured to: perform three-dimensional convolution processing based on the attention mechanism on the first feature map to obtain an attention matrix; and perform dimension adjustment on the attention matrix to obtain a third feature map, and the dimension of the third feature map is the same as the dimension of the second feature map.

[0088] In some embodiments of the present disclosure, based on the aforementioned scheme, the device 800 also includes: a training module, which is configured to: calculate a loss function based on the segmentation result; and adjust the parameters of feature extraction, graph-based convolution processing, attention-based spatial feature extraction, and upsampling processing with the goal of minimizing the loss function.

[0089] The specific details of the above-mentioned medical image segmentation devices have been described in detail in the corresponding medical image segmentation methods, and thus will not be repeated here.

[0090] It should be noted that, although several modules or units of the device for execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0091] In this exemplary embodiment, an electronic device capable of implementing the above method is also provided.

[0092] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as a system, method or program product. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as a "circuit", "module" or "system".

[0093] Refer to the following Fig. 9 An electronic device 900 according to this embodiment of the present invention will be described. Fig. 9 The electronic device 900 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0094] like Fig. 9As shown, the electronic device 900 is in the form of a general computing device. The components of the electronic device 900 may include, but are not limited to: the at least one processing unit 910, the at least one storage unit 920, a bus 930 connecting different system components (including the storage unit 920 and the processing unit 910), and a display unit 940.

[0095] The storage unit stores program codes, which can be executed by the processing unit 910, so that the processing unit 910 performs the steps of various exemplary embodiments of the present invention described in the medical image segmentation method described above in this specification. For example, the processing unit 910 can perform the following steps: Figure 1 Step S110 shown in: feature extraction is performed on the medical image to be processed to obtain a first feature map of the medical image to be processed; step S120: graph-based convolution processing is performed on the first feature map to obtain a second feature map; spatial feature extraction based on the attention mechanism is performed on the first feature map to obtain a third feature map; step S130: graph reconstruction is performed according to the second feature map and the third feature map to obtain a reconstructed feature map; step S140: upsampling is performed on the reconstructed feature map to obtain a segmentation result.

[0096] The storage unit 920 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 9201 and / or a cache storage unit 9202 , and may further include a read-only storage unit (ROM) 9203 .

[0097] The storage unit 920 may also include a program / utility 9204 having a set (at least one) of program modules 9205, such program modules 9205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0098] Bus 930 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0099] The electronic device 900 may also communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable viewers to interact with the electronic device 900, and / or communicate with any device that enables the electronic device 900 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 950. Furthermore, the electronic device 900 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 960. As shown, the network adapter 960 communicates with other modules of the electronic device 900 via a bus 930. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 900, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0100] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0101] In this exemplary embodiment, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of this specification is stored. In some possible implementations, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps of various exemplary embodiments of the present invention described in the above "Exemplary Method" section of this specification.

[0102] The program product for implementing the above method according to an embodiment of the present invention may adopt a portable compact disk read-only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto, and in this document, a readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, apparatus, or device.

[0103] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0104] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0105] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0106] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0107] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0108] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0109] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A medical image segmentation method, characterized in that: include: Performing feature extraction on the medical image to be processed to obtain a first feature map of the medical image to be processed; Performing graph-based convolution processing on the first feature map to obtain a second feature map; performing attention-based spatial feature extraction on the first feature map to obtain a third feature map; Performing image reconstruction according to the second feature map and the third feature map to obtain a reconstructed feature map; The reconstructed feature map is upsampled to obtain a segmentation result.

2. The medical image segmentation method according to claim 1, characterized in that: The feature extraction of the medical image to be processed comprises: The feature extraction of the medical image to be processed is performed through N coding units connected in series in sequence; wherein the feature extraction corresponding to each coding unit includes: Perform depth-wise separable convolution on the input of the current coding unit; Perform a first channel convolution and downsampling on the depth-to-depth separable convolution result to obtain the output corresponding to the current coding unit; wherein the input of the first coding unit is the medical image to be processed, the input of the second coding unit to the Nth coding unit is the output of the previous coding unit connected in series with it, and the output of the Nth coding unit is the first feature map.

3. The medical image segmentation method according to claim 2, characterized in that: Perform a depth-wise separable convolution on the input of the current encoding unit, including: Perform dilated separable convolutions at multiple scales and multiple dilation rates on the input of the current coding unit, and obtain multiple dilated convolution results accordingly; The multiple dilated convolution results are fused to obtain a depth-to-depth separable convolution result corresponding to the current coding unit.

4. The medical image segmentation method according to claim 1, characterized in that: Performing graph-based convolution processing on the first feature graph includes: Performing graph interaction processing of different feature spaces on the first feature graph to construct an adjacency matrix; Performing a second channel convolution on the first feature map to obtain a third feature matrix; The adjacency matrix is ​​used to perform graph convolution processing on the third feature matrix.

5. The medical image segmentation method according to claim 4, characterized in that: Performing graph interaction processing of different feature spaces on the first feature graph includes: Perform channel compression on the first feature map to obtain a first feature matrix corresponding to the first feature space and a second feature matrix corresponding to the second feature space; Performing matrix operations on the first characteristic matrix and the second characteristic matrix to obtain an interaction matrix; The interaction matrix is ​​processed nonlinearly to obtain the adjacency matrix.

6. The medical image segmentation method according to claim 5, characterized in that: The graph convolution processing includes: performing graph convolution processing on the third feature matrix based on the adjacency matrix through M graph convolution units connected in series in sequence; wherein, For the first graph convolution unit, perform a graph convolution process on the third feature matrix using the adjacency matrix and the weight parameter of the first graph convolution unit; For the second graph convolution unit to the Mth graph convolution unit, the adjacency matrix and the weight parameter of the current graph convolution unit are used to perform secondary graph convolution processing on the graph convolution result of the previous graph convolution unit connected in series with the current graph convolution unit.

7. The medical image segmentation method according to claim 1, characterized in that: Performing spatial feature extraction based on the attention mechanism on the first feature map includes: Performing a three-dimensional convolution process based on an attention mechanism on the first feature map to obtain an attention matrix; The attention matrix is ​​dimensionally adjusted to obtain the third feature map, where the dimension of the third feature map is the same as the dimension of the second feature map.

8. The medical image segmentation method according to any one of claims 1 to 7, characterized in that: The method further comprises: Calculate a loss function according to the segmentation result; With the goal of minimizing the loss function, the parameters of the feature extraction, the graph-based convolution processing, the attention-based spatial feature extraction, and the upsampling processing are adjusted.

9. A medical image segmentation device, characterized in that: include: An encoding module is configured to perform feature extraction on the medical image to be processed to obtain a first feature map of the medical image to be processed; A feature interaction module is configured to perform graph-based convolution processing on the first feature map to obtain a second feature map; perform attention-based spatial feature extraction on the first feature map to obtain a third feature map; A graph reconstruction module, configured to perform graph reconstruction according to the second feature graph and the third feature graph to obtain a reconstructed feature graph; The decoding module is configured to perform upsampling processing on the reconstructed feature map to obtain a segmentation result.

10. An electronic device, comprising: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images