Medical image analysis method and device of end side model, equipment and storage medium

Abstract

The invention relates to a medical image analysis method and device of an end-side model, equipment and a storage medium. The method comprises the following steps: inputting an initial image into a down-sampling module; performing down-sampling on the input initial image by using a down-sampling module to obtain a down-sampling result output by the down-sampling module; inputting the down-sampling result into a deep convolutional layer to obtain a feature image output by the deep convolutional layer; performing global information simulation calculation on the feature image by using a low-dimensional state space model to obtain global context semantic information; inputting the global context semantic information into a transposed convolutional layer to obtain a transposed convolutional image output by the transposed convolutional layer; inputting the down-sampling result and the transposed convolution image into an up-sampling module for up-sampling to obtain an up-sampling result output by the up-sampling module; and obtaining a target image output by the trained end side model based on an up-sampling result. The analysis efficiency of the medical image can be improved.

Description

Technical Field

[0001] The present application relates to the field of medical image processing technology, and in particular to a medical image analysis method, device, equipment and storage medium of a terminal-side model. Background Art

[0002] The rapid development of digital technology has greatly promoted the progress of digital medicine and laid a solid information foundation for it. Digital medicine based on artificial intelligence technology has gradually played a role in medical intervention. At present, the main artificial intelligence methods for medical image analysis are: methods based on convolutional neural networks (CNN) and methods based on self-attention mechanisms. These methods have promoted the intelligent development of medical image analysis and provided important auxiliary functions for doctors in the clinical medical process.

[0003] However, the existing medical image analysis models have a relatively high computational complexity, resulting in slow processing speeds and difficulty meeting the real-time requirements of medical care. Summary of the invention

[0004] The embodiments of the present application provide a medical image analysis method, apparatus, device and storage medium of an end-side model, aiming to solve the problem that the existing medical image analysis model has a relatively high computational complexity, resulting in a relatively slow processing speed and difficulty in meeting the real-time requirements of medical treatment.

[0005] In a first aspect, an embodiment of the present application provides a medical image analysis method of an end-side model, wherein the trained end-side model includes a downsampling module, an upsampling module, and a feature extraction module, wherein the feature extraction module includes a deep convolution layer, a low-dimensional state space model, and a transposed convolution layer, and the method includes:

[0006] Feed the initial image into the downsampling module;

[0007] The input initial image is downsampled by using the downsampling module to obtain the downsampling result output by the downsampling module;

[0008] The down-sampling result is input into the deep convolution layer to obtain the feature image output by the deep convolution layer;

[0009] The feature image is simulated and calculated using a low-dimensional state space model to obtain global contextual semantic information;

[0010] The global context semantic information is input into the transposed convolution layer to obtain the transposed convolution image output by the transposed convolution layer;

[0011] The down-sampling result and the transposed convolution image are input into the up-sampling module for up-sampling, and the up-sampling result output by the up-sampling module is obtained;

[0012] Based on the upsampling result, the target image output by the trained end-side model is obtained.

[0013] In some embodiments, before performing global information simulation calculation on the feature image using the low-dimensional state space model to obtain global context semantic information, the method further includes:

[0014] Sort the feature images in the positive direction to obtain a feature sequence in the positive direction;

[0015] Sort the feature images in the reverse direction to obtain a reverse direction feature sequence;

[0016] Sort the feature images in the depth direction to obtain a feature sequence in the depth direction;

[0017] The feature image is simulated and calculated using a low-dimensional state space model to obtain global contextual semantic information, including:

[0018] Inputting the forward direction feature sequence, the reverse direction feature sequence and the depth direction feature sequence into the low-dimensional state space model;

[0019] A low-dimensional state space model is used to perform global information modeling on the forward direction feature sequence, reverse direction feature sequence and depth direction feature sequence respectively, and the global context semantic information output by the low-dimensional state space model is obtained. The global context semantic information includes forward global information, reverse global information and depth global information.

[0020] In some implementations, before inputting the global context semantic information into the transposed convolutional layer, the method further includes:

[0021] The forward global information, the reverse global information and the deep global information are fused to obtain a fusion matrix;

[0022] Perform linear transformation on the fusion matrix to obtain the query vector Q, key vector K and value vector V;

[0023] Input the transpose of the key vector K, the query vector Q, the value vector V and the preset scaling factor into the self-attention layer to obtain the attention-enhanced features output by the self-attention layer;

[0024] The global context semantic information is input into the transposed convolution layer to obtain the transposed convolution image output by the transposed convolution layer, including:

[0025] The attention-enhanced features are input into the transposed convolution layer, and the transposed convolution result of the transposed convolution layer output is obtained.

[0026] In some embodiments, the downsampling module includes i downsampling network layers;

[0027] The input initial image is downsampled by the downsampling module to obtain the downsampling result output by the downsampling module, including:

[0028] Input the initial image into the first downsampling network layer to obtain the first downsampling result output by the first downsampling network layer;

[0029] The downsampling result of the first layer is used as the input of the next downsampling network layer and the operation is repeated until the downsampling results output by each downsampling network layer are obtained.

[0030] In some implementations, the upsampling module includes i upsampling network layers, and after obtaining the downsampling enhancement results corresponding to each downsampling network layer, the method further includes:

[0031] The downsampling results output by each downsampling network layer are enhanced respectively to obtain the downsampling enhancement results corresponding to each downsampling network layer;

[0032] The downsampling enhancement results corresponding to each downsampling network layer are respectively input into the feature extraction module to obtain the transposed convolution images corresponding to each downsampling enhancement result output by the feature extraction module;

[0033] The down-sampling result and the transposed convolution image are input into the up-sampling module for up-sampling, and the up-sampling result output by the up-sampling module is obtained, including:

[0034] The downsampling result output by the i-th downsampling network layer is fused with the transposed convolution image corresponding to the i-th downsampling network layer and input into the first upsampling network layer to obtain the upsampling result output by the first upsampling network layer;

[0035] The upsampling result output by the first upsampling network layer and the transposed convolution image corresponding to the i-1th downsampling network layer are used as the input of the next upsampling network layer and the operation is repeated until the upsampling result output by the i-th upsampling network layer is obtained.

[0036] In some embodiments, the method further comprises:

[0037] When i∈[1,2], the stride of the depthwise convolution layer and the transposed convolution layer are both configured to 4, and the kernel size of the depthwise convolution layer and the transposed convolution layer are both configured to 3;

[0038] When i∈[3,4], the stride of the depth convolution layer is configured to 2, and the kernel size of the depth convolution layer is configured to 1;

[0039] Configure the transposed convolution layer with a stride of 2, a kernel size of 4, and a padding size of 1.

[0040] In some implementations, the downsampling result output by the downsampling network layer is enhanced to obtain a downsampling enhancement result corresponding to the downsampling network layer, including:

[0041] Perform the first depth convolution on the downsampling result output by the downsampling network layer to obtain the first depth convolution result;

[0042] Perform the first point-by-point convolution on the downsampling result output by the downsampling network layer to obtain the first point-by-point convolution result;

[0043] Perform a second depth convolution on the first depth convolution result to obtain a second depth convolution result;

[0044] The first point-by-point convolution result is combined with the second depth-wise convolution result to obtain a convolution result;

[0045] Perform a second point-by-point convolution on the convolution result to obtain a second point-by-point convolution result;

[0046] The downsampling result output by the downsampling network layer is fused with the second point-by-point convolution result and input into the fully connected layer for linear transformation to obtain the downsampling enhancement result output by the fully connected layer.

[0047] In a second aspect, an embodiment of the present application further provides a medical image analysis device of an end-side model, which includes a unit for executing the above method.

[0048] In a third aspect, an embodiment of the present application further provides a computer device, which includes a memory and a processor, wherein a computer program is stored in the memory, and the above method is implemented when the processor executes the computer program.

[0049] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program can implement the above method when executed by a processor.

[0050] The embodiment of the present application provides a medical image analysis method, device, equipment and storage medium of an end-side model. The method includes: inputting the initial image into a downsampling module; downsampling the input initial image using the downsampling module to obtain the downsampling result output by the downsampling module; inputting the downsampling result into a deep convolution layer to obtain a feature image output by the deep convolution layer; using a low-dimensional state space model to perform global information simulation calculation on the feature image to obtain global context semantic information; inputting the global context semantic information into a transposed convolution layer to obtain a transposed convolution image output by the transposed convolution layer; inputting the downsampling result and the transposed convolution image into an upsampling module for upsampling to obtain the upsampling result output by the upsampling module; based on the upsampling result, obtaining the target image output by the trained end-side model.

[0051] In the embodiment of the present application, the downsampling result is input into the deep convolution layer to obtain a feature image, and the deep convolution layer is used to map the feature image to a low-dimensional space. The feature map in the low-dimensional space has a smaller number of channels and a smaller spatial size, which can directly reduce the computational complexity of the convolution operation. The feature image is reduced in dimension by using a low-dimensional state space model, which can remove unimportant features, retain the most critical information, and reduce unnecessary calculations. Therefore, the low-dimensional state space model can be used to more efficiently perform global information simulation calculations in low-dimensional space and capture global context information without performing complex calculations in high-dimensional space, thereby improving processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0053] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0054] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0055] Figure 1 A schematic flowchart of a medical image analysis method of a terminal-side model provided in an embodiment of the present application.

[0056] Figure 2 A flowchart of a U-shaped sampling and prediction network framework provided in an embodiment of the present application.

[0057] Figure 3 A flowchart of a feature extraction module provided in an embodiment of the present application.

[0058] Figure 4 A schematic diagram of the flow of the GMP Block module provided in an embodiment of the present application.

[0059] Figure 5 A schematic diagram of the structure of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0061] The disclosure below provides many different embodiments or examples to realize the different structures of the present application. In order to simplify the disclosure of the present application, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present application. In addition, the present application can repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0062] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0063] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0064] It should be further understood that the term “and / or” used in the specification and appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0065] As used in this specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0066] In order to solve the technical problem in the prior art that the medical image analysis model has a high computational complexity, resulting in a slow processing speed and difficulty in meeting the real-time requirements of medical treatment, the present application provides a medical image analysis method based on an end-side model, which can improve the analysis efficiency of medical images.

[0067] Figure 1 A flow chart of a medical image analysis method of a device-side model provided in an embodiment of the present application. The trained device-side model includes a downsampling module, an upsampling module, and a feature extraction module, the feature extraction module includes a deep convolution layer, a low-dimensional state space model, and a transposed convolution layer, and the method includes:

[0068] Step 110: Input the initial image into the downsampling module.

[0069] Step 120: down-sample the input initial image using a down-sampling module to obtain a down-sampling result output by the down-sampling module.

[0070] Step 130: Input the downsampling result into the deep convolution layer to obtain a feature image output by the deep convolution layer.

[0071] Step 140: Use a low-dimensional state space model to perform global information simulation calculation on the feature image to obtain global context semantic information.

[0072] Step 150: input the global context semantic information into the transposed convolution layer to obtain the transposed convolution image output by the transposed convolution layer.

[0073] Step 160: Input the down-sampling result and the transposed convolution image into the up-sampling module for up-sampling, and obtain the up-sampling result output by the up-sampling module.

[0074] Step 170: Based on the upsampling result, obtain the target image output by the trained end-side model.

[0075] Since the feature map in the low-dimensional space has a smaller number of channels and a smaller spatial size, the amount of computation of the convolution operation can be directly reduced, and the feature image can be reduced in dimension using a low-dimensional state space model, which can remove unimportant features, retain the most critical information, and reduce unnecessary calculations. Therefore, this embodiment inputs the downsampling result into a deep convolution layer to obtain a feature image, and uses a deep convolution layer to map the feature image to a low-dimensional space. The low-dimensional state space model can be used to more efficiently perform global information simulation calculations in the low-dimensional space, capture global context information, and eliminate the need for complex calculations in the high-dimensional space, thereby improving processing efficiency and throughput.

[0076] In addition, since this embodiment captures global contextual semantic information and then maps the low-dimensional features back to the original spatial dimensions through the transposed convolution layer, the accuracy of image segmentation can be guaranteed.

[0077] Referring to the second embodiment of a medical image analysis method of a terminal-side model provided in the present application, the second embodiment may include the following steps:

[0078] Step 210: Input the initial image into the downsampling module.

[0079] Step 220: down-sample the input initial image using the down-sampling module to obtain a down-sampling result output by the down-sampling module.

[0080] Step 230: Input the downsampling result into the deep convolution layer to obtain a feature image output by the deep convolution layer.

[0081] Wherein, step 230 may adopt the following formula 1:

[0082] X sq =Dw i (X) a , Formula 1.

[0083] Among them, X sq is the feature image, and X is the downsampling result.

[0084] Step 240: sort the feature images in the forward direction to obtain a forward direction feature sequence; sort the feature images in the reverse direction to obtain a reverse direction feature sequence; sort the feature images in the depth direction to obtain a depth direction feature sequence.

[0085] Specifically, for the feature image X sq Sort by the positive direction (for example, from the upper left to the lower right of the image) to obtain the positive direction feature sequence X sqF .

[0086] For the feature image X sq Sort in the reverse direction (for example, from the lower right to the upper left of the image) to obtain the reverse direction feature sequence X sqR .

[0087] For the feature image X sq Sort by depth direction (e.g. pixel by pixel from the first frame to the last frame) to get the depth direction feature sequence X sqS .

[0088] Step 250: Input the forward direction feature sequence, the reverse direction feature sequence and the depth direction feature sequence into the low-dimensional state space model.

[0089] Step 260: Use the low-dimensional state space model to perform global information modeling on the forward direction feature sequence, the reverse direction feature sequence and the depth direction feature sequence, respectively, to obtain the global contextual semantic information output by the low-dimensional state space model.

[0090] Among them, the global context semantic information includes forward global information, reverse global information and deep global information.

[0091] Step 270: input the global context semantic information into the transposed convolution layer to obtain the transposed convolution image output by the transposed convolution layer.

[0092] In some embodiments, before step 270, i.e. inputting the global context semantic information into the transposed convolution layer to obtain the transposed convolution image output by the transposed convolution layer, the following steps may also be included:

[0093] Step 271: Fuse the forward global information, the reverse global information and the depth global information to obtain a fusion matrix.

[0094] Wherein, step 271 may adopt the following formula 2:

[0095]

[0096] Among them, M(X sqF ) is the positive global information, M(X sqR ) is the reverse global information, M(X sqS ) is the deep global information, is the fusion matrix, and M is the low-dimensional state space model.

[0097] Step 272: Perform a linear transformation on the fusion matrix to obtain a query vector Q, a key vector K and a value vector V.

[0098] Step 272 may use the following formula 3:

[0099]

[0100] Step 273: Input the transpose of the key vector K, the query vector Q, the value vector V and the preset scaling factor into the self-attention layer to obtain the attention enhancement feature output by the self-attention layer.

[0101] Step 273 may use the following formula 4:

[0102]

[0103] in, is the preset scaling factor, X a is the attention enhancement feature and T is the transpose.

[0104] Based on this, step 270 can be to input the attention enhancement feature into the transposed convolution layer to obtain the transposed convolution result output by the transposed convolution layer.

[0105] Specifically, formula 5 can be used:

[0106] X′ ＝ T i (X a ) a , Formula 5.

[0107] Among them, X′ is the transposed convolution result, that is, the output result of the feature extraction module Slim UNETR Block v2, Tw i (·) a is the transposed convolution.

[0108] Step 280: Input the down-sampling result and the transposed convolution image into the up-sampling module for up-sampling, and obtain the up-sampling result output by the up-sampling module.

[0109] Step 290: Based on the upsampling result, obtain the target image output by the trained end-side model.

[0110] In some embodiments, the downsampling module may include i downsampling network layers (where i may be 4), and steps 120 and 220 may specifically include the following steps:

[0111] Step 310: Input the initial image into the first downsampling network layer to obtain the first downsampling result output by the first downsampling network layer.

[0112] Step 320: Repeat the operation by using the downsampling result of the first layer as the input of the next downsampling network layer until the downsampling results output by each downsampling network layer are obtained.

[0113] For example, the first layer downsampling result is input into the second layer downsampling network layer, and the second layer downsampling result output by the second layer downsampling network layer is obtained;

[0114] The second layer downsampling result is input into the third layer downsampling network layer, and the third layer downsampling result output by the third layer downsampling network layer is obtained;

[0115] The 3rd layer downsampling result is input into the 4th layer downsampling network layer to obtain the 4th layer downsampling result output by the 4th layer downsampling network layer.

[0116] In this way, the downsampling result of the first layer is used as the input of the next downsampling network layer for repeated operation until the downsampling results output by each downsampling network layer are obtained.

[0117] In some implementations, the upsampling module includes i upsampling network layers, and after step 320, i.e., obtaining the downsampling enhancement results corresponding to each downsampling network layer, the method further includes:

[0118] Step 330: Perform enhancement processing on the downsampling results output by each downsampling network layer to obtain the downsampling enhancement results corresponding to each downsampling network layer.

[0119] For example, the first layer downsampling result, the second layer downsampling result, the third layer downsampling result and the fourth layer downsampling result are enhanced respectively to obtain the first layer downsampling enhancement result, the second layer downsampling enhancement result, the third layer downsampling enhancement result and the fourth layer downsampling enhancement result.

[0120] In some implementations, step 330 may include the following steps:

[0121] Step 331: Perform a first depth convolution on the downsampling result output by the downsampling network layer to obtain a first depth convolution result.

[0122] Step 332: Perform a first point-by-point convolution on the downsampling result output by the downsampling network layer to obtain a first point-by-point convolution result.

[0123] Step 333: Perform a second depthwise convolution on the first depthwise convolution result to obtain a second depthwise convolution result.

[0124] Step 334: Fuse the first point-by-point convolution result with the second depth-wise convolution result to obtain a convolution result.

[0125] Step 335: Perform a second point-by-point convolution on the convolution result to obtain a second point-by-point convolution result.

[0126] Step 336: The downsampling result output by the downsampling network layer is fused with the second point-by-point convolution result and input into the fully connected layer for linear transformation to obtain the downsampling enhancement result output by the fully connected layer.

[0127] For steps 331 to 336, the following formula 6 may be used:

[0128] X=FC i (X g +·PW i (PW i (X g )+·DW i (DW i (X g )))), Formula 6.

[0129] Among them, X g is the downsampling result output by the downsampling network layer, DW i(X g ) is the result of the first depth convolution, PW i (X g ) is the first point-by-point convolution result, DW i (DW i (X g )) is the result of the second depth convolution, PW i (X g )+·DW i (DW i (X g )) is the convolution result, PW i (PW i (X g )+·DW i (DW i (X g ))) is the result of the second point-by-point convolution, X is the downsampling enhancement result of the fully connected layer output, that is, the output result of the GMP Block, i represents the feature map of the i-th downsampling layer currently being processed, FC i Represents a fully connected layer.

[0130] Among them, PWi(.) and DWi(.) represent point-wise convolution with kernel size 1 and depth-wise convolution with kernel size 3, respectively.

[0131] In some embodiments, after the point-by-point convolution and depth-wise convolution operations, instance normalization processing and adaptive activation function processing may be performed. Specifically, when i∈[1,2], the adaptive activation function may be set to a GELU function, and when i∈[3,4], the adaptive activation function may be set to a Swish function.

[0132] In addition, the fully connected layer is also processed with an adaptive activation function. Similarly, when i∈[1,2], the adaptive activation function can be set to the GELU function, and when i∈[3,4], the adaptive activation function can be set to the Swi sh function.

[0133] Step 340: input the downsampling enhancement results corresponding to each downsampling network layer into the feature extraction module respectively, and obtain the transposed convolution image corresponding to each downsampling enhancement result output by the feature extraction module.

[0134] Among them, when i∈[1,2], the step size of the deep convolution layer and the transposed convolution layer in the feature extraction module is configured to 4, and the kernel size of the deep convolution layer and the transposed convolution layer is configured to 3.

[0135] When i∈[3,4], the stride of the depthwise convolutional layer is configured to 2, and the kernel size of the depthwise convolutional layer is configured to 1.

[0136] Configure the transposed convolution layer with a stride of 2, a kernel size of 4, and a padding size of 1.

[0137] Step 350: Fuse the downsampling result output by the i-th downsampling network layer with the transposed convolution image corresponding to the i-th downsampling network layer and input them into the first upsampling network layer to obtain the upsampling result output by the first upsampling network layer.

[0138] Step 360: Repeat the operation of using the upsampling result output by the 1st upsampling network layer and the transposed convolution image corresponding to the i-1th downsampling network layer as the input of the next upsampling network layer until the upsampling result output by the i-th upsampling network layer is obtained.

[0139] For steps 350 and 360, please refer to Figure 2 , specifically it may include the following processes.

[0140] 1) Normalize the downsampling result of the 4th layer to obtain a normalized result;

[0141] 2) Perform a deconvolution operation on the normalized result to obtain a deconvolution result;

[0142] 3) Input the deconvolution result and the 4th layer downsampling enhancement result into the 1st layer upsampling network layer to obtain the 1st layer upsampling result output by the 1st layer upsampling network layer;

[0143] 4) Inputting the first layer upsampling result and the third layer downsampling enhancement result into the second layer upsampling network layer, and obtaining the second layer upsampling result output by the second layer upsampling network layer;

[0144] 5) Inputting the upsampling result of the second layer and the downsampling enhancement result of the second layer into the upsampling network layer of the third layer, and obtaining the upsampling result of the third layer output by the upsampling network layer of the third layer;

[0145] 6) The upsampling result of the third layer and the downsampling enhancement result of the first layer are input into the fourth upsampling network layer to obtain the fourth upsampling result output by the fourth upsampling network layer.

[0146] In this way, the upsampling result output by the first upsampling network layer and the transposed convolution image corresponding to the i-1th downsampling network layer are used as the input of the next upsampling network layer for repeated operation until the upsampling result output by the i-th upsampling network layer is obtained.

[0147] After obtaining the upsampling result of the 4th layer, the upsampling result of the 4th layer can be input into a separate convolution layer for deep convolution to obtain a deep convolution image; then the Sigmoid activation function is used to calculate and predict the deep convolution image to obtain the analysis result, that is, the prediction result f(X input );

[0148] Then calculate the prediction result f(X input ) and the true value Y to evaluate the correctness of the prediction results.

[0149] Based on the above embodiments, combined with Figure 2-Figure 4 The medical image analysis model 100 provided in this application, namely SlimUNETRv2, mainly includes a U-type sampling and prediction network framework, a feature extraction module SlimUNETR Block v2 and a GMP Block module.

[0150] Among them, the U-type sampling and prediction network framework includes a downsampling module, an upsampling module and a separate convolutional layer Segment Head, such as Figure 2 As shown, the U-type sampling and prediction network framework can specifically include a four-stage downsampling process and a corresponding four-stage upsampling process.

[0151] Specifically, suppose the input initial image is a three-dimensional medical image X input (W*H*D*C), where W, H, D, and C represent the width, height, depth, and number of channels of the initial image, respectively. The i-th downsampling stage of the U-shaped sampling and prediction network framework downsamples the input initial image to X by deep convolution calculation. down , size is W / 2 i+1 *H / 2 i+1 *D / 2 i+1 *C i , where C i ∈[64,128,256,512]. After that, the corresponding upsampling stage upsamples the image to X by deconvolution calculation. up , whose size is the same as the X of the corresponding stage down Similarly, the final analysis result f(X) is calculated through a separate convolutional layer Segment Head and Sigmoid activation function. input ). At the same time, the U-shaped sampling and prediction network framework uses skip connections to connect the corresponding sampling results in the downsampling and upsampling processes to enhance the final analysis performance.

[0153] The feature extraction module SlimUNETR Block v2 can include a deep convolution layer, a low-dimensional state space model, and a transposed convolution layer. The low-dimensional state space model uses a sparse state space calculation mechanism to provide low-computational feature extraction for high-performance medical image analysis. The specific calculation framework is as follows: Figure 3Specifically, the feature extraction module is deployed in each downsampling stage and the corresponding upsampling stage of the U-shaped sampling and prediction network framework.

[0154] Among them, the GMP Block module is based on the design of differentiated step-size convolution and is used for feature smoothing and spatial information supplementation of sampling results. Its specific calculation framework is as follows: Figure 4 Specifically, the GMP Block module applies a set of convolutional layers and fully connected layers with different strides to enhance the capture of local information.

[0155] Assume that the downsampling result accepted by the GMP Block module is X g , the output result is X, and its calculation can refer to the above formula 6.

[0156] In some embodiments, when the medical image analysis model SlimUNETR v2 is trained, the prediction result f(X input ) is used to calculate the index by comparing the true value Y of the analysis result within the image (Y is the lesion / lesion analysis result accurately judged by the doctor).

[0157] Specifically, the following two main indicators can be used to evaluate the correctness of the analysis task of the medical image analysis model:

[0158] 1) Dice coefficient: This value represents the method analysis result f(X input ) is similar to the true value Y. For details, please refer to the following formula 7:

[0159]

[0160] 2) HD95 error: This value represents the method analysis result f(X input ) and the true value Y, please refer to the following formula 8 for details:

[0161] HD·95=·percentile(distances(f(X input ),Y),95%), Formula 8.

[0162] Wherein, distances(.,.) represents the distance set of the nearest points in two analysis point sets, and percentile(,95%) represents the 95% percentile of the above distance set.

[0163] In summary, the feature extraction module SlimUNETR v2 maps the input features (i.e., the downsampling results output by each downsampling network layer) to a low-dimensional space (i.e., a low-dimensional state space model) through a deep convolutional layer, and performs three-way global information exchange in the low-dimensional space to capture global contextual semantic information, and then maps the low-dimensional features back to the original space dimension through a transposed convolutional layer. In this way, global information exchange in a low-dimensional state space model can not only ensure segmentation accuracy, but also improve the efficiency of medical image analysis and processing.

[0164] Corresponding to the above medical image analysis method of the end-side model, the present application also provides a medical image analysis device of the end-side model. The medical image analysis device of the end-side model includes a unit for executing the above medical image analysis method of the end-side model, and the medical image analysis device of the end-side model can be configured in a desktop computer, a tablet computer, a laptop, or other terminals.

[0165] like Figure 5 As shown, the embodiment of the present application provides a computer device, including a processor 111, a communication interface 112, a memory 113 and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114.

[0166] Memory 113, used for storing computer programs;

[0167] In one embodiment of the present application, the processor 111 is configured to execute a program stored in the memory 113 to implement the medical image analysis method of the device-side model provided in any one of the aforementioned method embodiments, including:

[0168] Feed the initial image into the downsampling module;

[0169] The input initial image is downsampled by using the downsampling module to obtain the downsampling result output by the downsampling module;

[0170] The down-sampling result is input into the deep convolution layer to obtain the feature image output by the deep convolution layer;

[0171] The feature image is simulated and calculated using a low-dimensional state space model to obtain global contextual semantic information;

[0172] The global context semantic information is input into the transposed convolution layer to obtain the transposed convolution image output by the transposed convolution layer;

[0173] The down-sampling result and the transposed convolution image are input into the up-sampling module for up-sampling, and the up-sampling result output by the up-sampling module is obtained;

[0174] Based on the upsampling result, the target image output by the trained end-side model is obtained.

[0175] It is understood by those skilled in the art that all or part of the processes in the method for implementing the above embodiment can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a storage medium, which is a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the process steps of the embodiment of the above method.

[0176] Therefore, the embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the medical image analysis method of the end-side model provided in any of the aforementioned method embodiments are implemented. Including:

[0177] Feed the initial image into the downsampling module;

[0178] The input initial image is downsampled by using the downsampling module to obtain the downsampling result output by the downsampling module;

[0179] The down-sampling result is input into the deep convolution layer to obtain the feature image output by the deep convolution layer;

[0180] The feature image is simulated and calculated using a low-dimensional state space model to obtain global contextual semantic information;

[0181] The global context semantic information is input into the transposed convolution layer to obtain the transposed convolution image output by the transposed convolution layer;

[0182] The down-sampling result and the transposed convolution image are input into the up-sampling module for up-sampling, and the up-sampling result output by the up-sampling module is obtained;

[0183] Based on the upsampling result, the target image output by the trained end-side model is obtained.

[0184] The storage medium is a physical, non-transient storage medium, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk, etc., which can store program codes. Computer-readable storage media can be non-volatile or volatile.

[0185] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0186] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of each unit is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0187] The steps in the method of the embodiment of the present application can be adjusted in order, combined and deleted according to actual needs. The units in the device of the embodiment of the present application can be combined, divided and deleted according to actual needs. In addition, the functional units in the various embodiments of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0188] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, terminal, or network device, etc.) to perform all or part of the steps of the various embodiments of the present application.

[0189] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0190] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

[0191] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A medical image analysis method of a terminal-side model, characterized in that: The trained end-side model includes a downsampling module, an upsampling module, and a feature extraction module, the feature extraction module includes a deep convolution layer, a low-dimensional state space model, and a transposed convolution layer, and the method includes: Inputting the initial image into the downsampling module; Downsampling the input initial image using the downsampling module to obtain a downsampling result output by the downsampling module; Inputting the downsampling result into the deep convolution layer to obtain a feature image output by the deep convolution layer; Using the low-dimensional state space model to perform global information simulation calculation on the feature image to obtain global context semantic information; Inputting the global context semantic information into the transposed convolution layer to obtain a transposed convolution image output by the transposed convolution layer; Inputting the down-sampling result and the transposed convolution image into the up-sampling module for up-sampling, and obtaining the up-sampling result output by the up-sampling module; Based on the upsampling result, a target image output by the trained end-side model is obtained.

2. The method according to claim 1, characterized in that Before performing global information simulation calculation on the feature image using the low-dimensional state space model to obtain global context semantic information, the method further includes: Sorting the feature images in a forward direction to obtain a forward direction feature sequence; Sorting the feature images in a reverse direction to obtain a reverse direction feature sequence; Sorting the feature images in the depth direction to obtain a feature sequence in the depth direction; The step of using the low-dimensional state space model to perform global information simulation calculation on the feature image to obtain global context semantic information includes: Inputting the forward direction feature sequence, the reverse direction feature sequence and the depth direction feature sequence into the low-dimensional state space model; The low-dimensional state space model is used to perform global information modeling on the forward direction feature sequence, the reverse direction feature sequence and the depth direction feature sequence respectively to obtain the global context semantic information output by the low-dimensional state space model, wherein the global context semantic information includes forward global information, reverse global information and depth global information.

3. The method according to claim 2, characterized in that Before inputting the global context semantic information into the transposed convolutional layer, the method further includes: Fusing the forward global information, the reverse global information and the depth global information to obtain a fusion matrix; Performing a linear transformation on the fusion matrix to obtain a query vector Q, a key vector K and a value vector V; Inputting the transpose of the key vector K, the query vector Q, the value vector V and a preset scaling factor into a self-attention layer to obtain an attention enhancement feature output by the self-attention layer; The step of inputting the global context semantic information into the transposed convolution layer to obtain the transposed convolution image output by the transposed convolution layer comprises: The attention enhancement feature is input into the transposed convolution layer to obtain a transposed convolution result output by the transposed convolution layer.

4. The method according to claim 1, characterized in that: The downsampling module includes i downsampling network layers; The downsampling the input initial image by using the downsampling module to obtain the downsampling result output by the downsampling module includes: Inputting the initial image into the first downsampling network layer to obtain the first downsampling result output by the first downsampling network layer; The first layer of downsampling results is used as the input of the next layer of downsampling network layer to repeat the operation until the downsampling results output by each downsampling network layer are obtained.

5. The method according to claim 4, characterized in that The upsampling module includes i upsampling network layers. After obtaining the downsampling enhancement results corresponding to each downsampling network layer, the method further includes: The downsampling results output by each downsampling network layer are enhanced respectively to obtain the downsampling enhancement results corresponding to each downsampling network layer; Inputting the downsampling enhancement results corresponding to each downsampling network layer into the feature extraction module respectively, and obtaining the transposed convolution image corresponding to each downsampling enhancement result output by the feature extraction module; The step of inputting the down-sampling result and the transposed convolution image into the up-sampling module for up-sampling to obtain the up-sampling result output by the up-sampling module comprises: The downsampling result output by the i-th downsampling network layer is fused with the transposed convolution image corresponding to the i-th downsampling network layer and input into the first upsampling network layer to obtain the upsampling result output by the first upsampling network layer; The upsampling result output by the first upsampling network layer and the transposed convolution image corresponding to the i-1th downsampling network layer are used as the input of the next upsampling network layer and the operation is repeated until the upsampling result output by the i-th upsampling network layer is obtained.

6. The method according to claim 5, characterized in that The method further comprises: When i∈[1,2], the step sizes of the depth convolution layer and the transposed convolution layer are both configured to be 4, and the kernel sizes of the depth convolution layer and the transposed convolution layer are both configured to be 3; When i∈[3,4], the step size of the depth convolution layer is configured to be 2, and the kernel size of the depth convolution layer is configured to be 1; The stride of the transposed convolutional layer is configured to be 2, the kernel size to be 4, and the padding size to be 1.

7. The method according to claim 1, characterized in that The downsampling result output by the downsampling network layer is enhanced to obtain the downsampling enhancement result corresponding to the downsampling network layer, including: Perform the first depth convolution on the downsampling result output by the downsampling network layer to obtain the first depth convolution result; Perform the first point-by-point convolution on the downsampling result output by the downsampling network layer to obtain the first point-by-point convolution result; Performing a second depth convolution on the first depth convolution result to obtain a second depth convolution result; Fusing the first point-by-point convolution result with the second depth-wise convolution result to obtain a convolution result; Performing a second point-by-point convolution on the convolution result to obtain a second point-by-point convolution result; The downsampling result output by the downsampling network layer is fused with the second point-by-point convolution result and input into the fully connected layer for linear transformation to obtain the downsampling enhancement result output by the fully connected layer.

8. A medical image analysis device of a terminal side model, characterized in that: The method comprises a unit for executing the method according to any one of claims 1 to 7.

9. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 can be implemented.

Images