Image segmentation method and device for eyeball, rectus oculi muscle, and optic nerve

Abstract

The image segmentation method and device applied to the eyeball, rectus oculi muscle, and optic nerve can help doctors avoid the eyeball, rectus oculi muscle, and optic nerve during surgery, thereby improving the accuracy of surgery and reducing unnecessary trauma to the human body. The method includes: (1) in the encoding stage, the network adopts four-fold downsampling, uses the residual convolution block as the basic convolution unit, and sets the convolution kernel size to 3*3; (2) adds the Transformer module to extract and fuse the features of the global information, reducing the loss of distant features in pixel-level prediction; (3) in the decoding stage, uses the compressed attention module to suppress irrelevant information in the image, strengthen the learning of key information, and make the network pay more attention to the key features in the image.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to an image segmentation method for an eyeball, rectus oculi muscle, and optic nerve, and an image segmentation device for an eyeball, rectus oculi muscle, and optic nerve. Background Art

[0002] During eye surgeries like optic nerve decompression surgery, doctors must carefully consider the complex structure of the eye, which contains numerous tiny yet vital organs. Furthermore, due to individual differences, relying solely on experience increases surgical risk, and inadvertently damaging vital tissue can have serious consequences. Therefore, precise organ segmentation and surgical navigation can alert surgeons to their locations and prevent them from being damaged.

[0003] In a surgical navigation system, effective analysis of the patient's preoperative MRI, CT, and other medical images is crucial for intraoperative surgical navigation. Accurately segmenting each target tissue is a fundamental and crucial requirement. First, surgical navigation can help expand the surgeon's field of view during surgery, revealing previously unobservable information. This ultimately shortens the overall surgical time, reduces surgical trauma and complications, and effectively helps improve surgical precision. Second, by analyzing the surgical process through the surgical navigation system, subsequent surgical analysis can further evaluate the procedure, positively impacting subsequent surgeries. Summary of the Invention

[0004] In order to overcome the defects of the existing technology, the technical problem to be solved by the present invention is to provide an image segmentation method applied to the eyeball, rectus oculi muscle, and optic nerve, which can help doctors better avoid the eyeball, rectus oculi muscle, and optic nerve during surgery during surgical navigation, thereby improving the accuracy of surgery and reducing some unnecessary trauma to the human body.

[0005] The technical solution of the present invention is: this image segmentation method applied to the eyeball, rectus oculi muscle, and optic nerve comprises the following steps:

[0006] (1) In the encoding stage, the network adopts four-fold downsampling, uses the residual convolution block as the basic convolution unit, and the convolution kernel size is set to 3*3;

[0007] (2) Adding the Transformer module to extract and fuse global information features, reducing the loss of distant features in pixel-level prediction;

[0008] (3) In the decoding stage, a compressed attention module is used to suppress irrelevant information in the image.

[0009] Strengthen the learning of key information so that the network pays more attention to the key features in the image.

[0010] In the encoding stage, the network of the present invention adopts four-times downsampling, uses the residual convolution block as the basic convolution unit, and sets the convolution kernel size to 3*3; in the decoding stage, the compressed attention module is used to suppress irrelevant information in the image, strengthen the learning of key information, and make the network pay more attention to the key features in the image; the Transformer module is added between the encoder and decoder to extract and fuse features of global information, improve the global context perception ability of the target area, and effectively reduce the loss problem of longer-distance features in pixel-level prediction, so as to help doctors better avoid the eyeball, rectus muscle, and optic nerve during surgery during surgical navigation, thereby improving the accuracy of surgery and reducing some unnecessary trauma to the human body.

[0011] Also provided is an image segmentation device for the eyeball, rectus oculi muscle, and optic nerve, comprising:

[0012] The encoding module is configured to downsample the network four times, using the residual convolution block as the basic convolution unit and the convolution kernel size is set to 3*3;

[0013] The Transformer module is configured to extract and fuse global information to reduce the loss of distant features in pixel-level prediction;

[0014] The decoding module is configured to use a compressed attention module to suppress irrelevant information in the image.

[0015] information, strengthen the learning of key information, and make the network pay more attention to the key features in the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 The present invention is a flowchart of an image segmentation method for eyeball, rectus oculi muscle and optic nerve. DETAILED DESCRIPTION

[0017] like Figure 1 As shown, the image segmentation method applied to the eyeball, rectus oculi muscle, and optic nerve includes the following steps:

[0018] (1) In the encoding stage, the network adopts four-fold downsampling, uses the residual convolution block as the basic convolution unit, and the convolution kernel size is set to 3*3;

[0019] (2) Adding the Transformer module to extract and fuse global information features, reducing the loss of distant features in pixel-level prediction;

[0020] (3) In the decoding stage, a compressed attention module is used to suppress irrelevant information in the image.

[0021] Strengthen the learning of key information so that the network pays more attention to the key features in the image.

[0022] In the encoding stage, the network of the present invention adopts four-times downsampling, uses the residual convolution block as the basic convolution unit, and sets the convolution kernel size to 3*3; in the decoding stage, the compressed attention module is used to suppress irrelevant information in the image, strengthen the learning of key information, and make the network pay more attention to the key features in the image; the Transformer module is added between the encoder and decoder to extract and fuse features of global information, improve the global context perception ability of the target area, and effectively reduce the loss problem of longer-distance features in pixel-level prediction, so as to help doctors better avoid the eyeball, rectus muscle, and optic nerve during surgery during surgical navigation, thereby improving the accuracy of surgery and reducing some unnecessary trauma to the human body.

[0023] Preferably, in step (2), given the output feature map F of the encoder, in order to ensure the comprehensive representation of each data, a 3×3×3 convolution layer is used to increase the channel dimension from 128 to 512; considering that the input of the Transformer module is a Sequence, the spatial dimension and the depth dimension are folded into one dimension to obtain a d×N feature map f, N = H×W×D; in order to encode the position information that is crucial in the segmentation task, learnable position embeddings are introduced and fused with the feature map f by directly adding them to create feature embeddings as follows:

[0024] z0=f+PE=W × F+PE (1)

[0025] Where W is the linear projection operation, PE ∈ R a×N represents the position embedding, z0 ∈ R a×N represents feature embedding.

[0026] Preferably, in step (2), the Transformer module is composed of L Transformer layers, each Transformer layer has a standard architecture consisting of a multi-head attention MHA block and a feed-forward network FFN; the lth i The output of the Transformer layer (l∈[1, 2, ..., L]) is calculated as follows:

[0027] z′l=MHA(LN(z l-1 ))+z l-1 (2)

[0028] z l =FFN(LN(z′ l))+z ′ l (3)

[0029] Among them, LN(*) represents layer normalization, z l It is the first i The output of a Transformer layer.

[0030] Preferably, in step (3), a compressed attention SA module is used, which includes two convolution channels: a conventional convolution channel, which includes two basic convolution blocks composed of convolution, batch normalization, and linear rectification function ReLU; a squeeze-attention channel, which first performs a non-full compression operation by average pooling to expand the receptive field while retaining high-level feature information, and then extracts high-level features by two basic convolution blocks, and finally restores the image size to the same as the output of the conventional convolution channel through bilinear interpolation upsampling.

[0031] Preferably, in step (3), the output X of the SA module out Expressed as:

[0032] X out =X attn *X res +-X attn (4)

[0033] Among them, X res is the output of the regular convolution channel, X attn is the output of the squeeze-attention channel, X res , X attn Respectively expressed as:

[0034] X res =F(X in ;θ,Ω) (5)

[0035]

[0036] Among them, F(·) is the residual function defined by the parameters θ and Ω of the two basic convolutional blocks, Up(·

[0037] ) is a bilinear interpolation upsampling function that restores the feature map to the size of the input image, σ(·) is a linear rectification function ReLU, Output of the inner attention convolution operation:

[0038]

[0039] Among them, F attn (·) represents the parameters θ of the two attention convolution blocks attn ,Ω attnThe attention convolution function defined, AvgPool(·) is the average pooling layer to reduce the input image size.

[0040] Preferably, the method further comprises supervising the network using a hybrid loss function during training, where the hybrid function is defined as follows:

[0041] Loss = α1l CE +α2l Dice +α3l SSIM (8)

[0042] Among them, l CE is the CE loss function, l Dice is the Dice loss function, l SSIM is the SSIM loss function, and α1, α2, and α3 are hyperparameters. Experiments show that the segmentation accuracy is highest when α1, α2, and α3 are set to 0.2, 0.5, and 0.3, respectively. Therefore, this patent uses this weight as the optimal weight for the network model.

[0043] Preferably, the CE loss function is used to measure the difference between the target true value and the predicted value, which is defined as follows:

[0044] lcE=-∑ (x , y )[G(x,y)log(P(x,y))+(1-G(x,y))log(1-P(x,y))] (9)

[0046] Among them, G(x, y)∈{0, 1} is the gold standard of pixel point (x, y), P(x, y) is the predicted probability of the neural network model for classifying pixel point (x, y); the CE loss function does not consider the neighborhood of the pixel, and performs equally weighted calculations on the foreground and background, which helps to converge the loss value of each pixel.

[0047] Preferably, the Dice loss function is defined as:

[0048]

[0049] Among them, G(x, y)∈{0, 1} is the gold standard of pixel point (x, y), P(x, y) is the network's predicted probability of classifying pixel point (x, y); the Dice loss function is used to measure the similarity between the network segmentation result and the gold standard, and pays more attention to the prediction of the target foreground.

[0050] Preferably, the SSIM loss function is used as part of the loss function during training to learn the structural information in the gold standard. The definition of SSIM is as follows:

[0051]

[0052] Where x={x j :j=1,...,N 2}, y={y j :j=1,...,N 2} are the pixel values ​​of the corresponding points in the N×N image blocks cropped from the gold standard G(x, y) and the network predicted segmentation result P(x, y), μ x , μ y are the means of x and y, σ x , σ y are the standard deviations of x and y, σ xy is the covariance of x and y, and the hyperparameters C1 and C2 are set to C1 = 0.01 respectively 2 , C2=0.03 2 ,SSIM loss function considers the local neighborhood of each pixel and assigns higher weights to the boundary area, which helps to optimize the image boundary.

[0053] Those skilled in the art will appreciate that all or part of the steps in the above-described method can be accomplished by instructing the relevant hardware through a program. The program can be stored in a computer-readable storage medium. When executed, the program includes each step of the above-described method. The storage medium can be ROM / RAM, a magnetic disk, an optical disk, a memory card, etc. Therefore, corresponding to the method of the present invention, the present invention also includes an image segmentation device for the eyeball, rectus oculi muscle, and optic nerve. The device is generally represented in the form of functional modules corresponding to the steps of the method. The device includes:

[0054] The encoding module is configured to downsample the network four times, using the residual convolution block as the basic convolution unit and the convolution kernel size is set to 3*3;

[0055] The Transformer module is configured to extract and fuse global information to reduce the loss of distant features in pixel-level prediction;

[0056] The decoding module is configured to use a compressed attention module to suppress irrelevant information in the image.

[0057] information, strengthen the learning of key information, and make the network pay more attention to the key features in the image.

[0058] Accurately extracting the eyeball, rectus oculi muscle, and optic nerve from CT images can provide doctors with precise information about the eye's structure. Due to the complex structure of the eye, uneven distribution of contrast agents in CT images, significant noise, and proximity to complex surrounding tissue, manual and automatic segmentation of the eyeball, rectus oculi muscle, and optic nerve present significant challenges. Therefore, this patent utilizes a convolutional neural network based on a hybrid loss function and integrated with a Transformer to achieve automatic and precise extraction of the eyeball, rectus oculi muscle, and optic nerve. The entire network adopts a U-shaped architecture, which effectively integrates low-level and high-level features. Low-level features in CT images contain local information such as edges and texture, which facilitates the recovery of details of the eyeball, optic nerve, and rectus oculi muscle, but do not reflect their overall shape. High-level features have a larger receptive field and provide high-level semantic information, including position, which helps the network predict the overall structure and position of each organ. During the encoding phase, the network employs four-fold downsampling, uses a residual convolution block as the basic convolution unit, and has a convolution kernel size of 3*3. During the decoding phase, a compressed attention module is used to suppress irrelevant information in the image, strengthen the learning of key information, and focus the network on key features in the image. A Transformer module is added between the encoder and decoder to extract and fuse global information, improving the global context perception of the target area and effectively reducing the loss of distant features in pixel-level predictions.

[0059] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention are still within the scope of protection of the technical solution of the present invention.

Claims

1. An image segmentation method for the eyeball, rectus oculi muscle, and optic nerve, characterized by: It includes the following steps: (1) In the encoding stage, the network adopts four-fold downsampling, uses the residual convolution block as the basic convolution unit, and the convolution kernel size is set to 3*3; (2) Adding the Transformer module to extract and fuse global information features, reducing the loss of distant features in pixel-level prediction; (3) In the decoding stage, a compressed attention module is used to suppress irrelevant information in the image, strengthen the learning of key information, and make the network pay more attention to the key features in the image; In step (2), given the encoder's output feature map F, in order to ensure a comprehensive representation of each data, a 3×3×3 convolutional layer is used to increase the channel dimension from 128 to 512; considering that the input of the Transformer module is a Sequence, the spatial dimension and the depth dimension are folded into one dimension to obtain a d×N feature map f, N = H×W×D; in order to encode the position information that is crucial in the segmentation task, learnable position embeddings are introduced and fused with the feature map f by directly adding them to create the feature embedding as follows: z0=f+PE=W×F+PE (1) Where W is the linear projection operation, PE∈R d×N Represents position embedding, z0∈R d×N Represents feature embedding; In the step (3), a compression attention SA module is used, which includes two convolution channels: a conventional convolution channel, which includes two basic convolution blocks composed of convolution, batch normalization, and linear rectification function ReLU; a squeeze-attention channel, which first performs a non-full compression operation by average pooling to expand the receptive field while retaining high-level feature information, and then extracts high-level features by two basic convolution blocks, and finally restores the image size to the same as the output of the conventional convolution channel through bilinear interpolation upsampling.

2. The image segmentation method for the eyeball, rectus oculi muscle, and optic nerve according to claim 1, characterized in that: In step (2), the Transformer module consists of L Transformer layers, each of which has a standard architecture consisting of a multi-head attention MHA block and a feed-forward network FFN; the lth i The output of the Transformer layer (l∈[1,2,…,L]) is calculated as follows: z′ l =MHA(LN(z l-1 ))+z l-1 (2) from l =FFN(LN(z′ l ))+z′ l (3) Among them, LN(*) represents layer normalization, z l It is the first i The output of a Transformer layer.

3. The image segmentation method for the eyeball, rectus oculi muscle, and optic nerve according to claim 2, characterized in that: In step (3), the output X of the SA module out Expressed as: X out =X attn *X res +X attn (4) Among them, X res is the output of the regular convolution channel, X attn is the output of the squeeze-attention channel, X res , X attn Respectively expressed as: X res =F(X in ;θ,Ω) (5) Among them, F(·) is the residual function defined by the parameters θ and Ω of the two basic convolution blocks, Up(·) is the bilinear interpolation upsampling function that restores the feature map to the size of the input image, and σ(·) is the linear rectification function ReLU. is the output of the attention convolution operation: Among them, F attn (·) represents the parameters θ of the two attention convolution blocks attn ,Ω attn The attention convolution function defined, AvgPool(·) is the average pooling layer to reduce the input image size.

4. The image segmentation method for the eyeball, rectus oculi muscle, and optic nerve according to claim 3, characterized in that: The method also includes using a mixed loss function to supervise the network during training. The mixed function is defined as follows: Loss=α1l CE +α2l Dice +α3l SSIM (8) Among them, l CE is the CE loss function, l Dice is the Dice loss function, l SSIM is the SSIM loss function, and α1, α2, and α3 are hyperparameters.

5. The image segmentation method for the eyeball, rectus oculi muscle, and optic nerve according to claim 4, characterized in that: The CE loss function is used to measure the difference between the target true value and the predicted value, which is defined as follows: l CE =-∑ (x,y) [G(x,y)log(P(x,y))+(1-G(x,y))log(1-P(x,y))] (9) Among them, G(x,y)∈{0,1} is the gold standard of pixel point (x,y), P(x,y) is the predicted probability of the neural network model for the classification of pixel point (x,y); the CE loss function does not consider the neighborhood of the pixel and performs equally weighted calculations on the foreground and background, which helps to converge the loss value of each pixel.

6. The image segmentation method for the eyeball, rectus oculi muscle, and optic nerve according to claim 5, characterized in that: The Dice loss function is defined as: Among them, G(x,y)∈{0,1} is the gold standard of pixel point (x,y), P(x,y) is the network's predicted probability of classifying pixel point (x,y); the Dice loss function is used to measure the similarity between the network segmentation result and the gold standard, and pays more attention to the prediction of the target foreground.

7. The image segmentation method for the eyeball, rectus oculi muscle, and optic nerve according to claim 6, characterized in that: The SSIM loss function is used as part of the loss function during training to learn the structural information in the gold standard. The definition of SSIM is as follows: Where x={x j :j=1,…,N 2 }, y={y j :j=1,…,N 2 } are the pixel values ​​of the corresponding points in the N×N image blocks cropped from the gold standard G(x,y) and the network predicted segmentation result P(x,y), μ x , μ y are the means of x and y, σ x , σ y are the standard deviations of x and y, σ xy is the covariance of x and y, and the hyperparameters C1 and C2 are set to C1 = 0.01 respectively 2 , C2=0.03 2 ,SSIM loss function considers the local neighborhood of each pixel and assigns higher weights to the boundary area, which helps to optimize the image boundary.

8. An image segmentation device for the eyeball, rectus oculi muscle, and optic nerve, for executing the method according to claim 1, characterized in that: It includes: The encoding module is configured to downsample the network four times, using the residual convolution block as the basic convolution unit and the convolution kernel size is set to 3*3; The Transformer module is configured to extract and fuse global information to reduce the loss of distant features in pixel-level prediction; The decoding module is configured to use the compressed attention module to suppress irrelevant information in the image, strengthen the learning of key information, and make the network pay more attention to the key features in the image.

Images