Double-branch vision Transform eye CT lesion accurate detection method

Abstract

According to the technical scheme, the invention discloses a method for accurately detecting eye CT (Computed Tomography) lesions through double-branch vision Transformer. The dual-branch ViT structure provided by the invention aims at the characteristics (1) of orbit CT (Computed Tomography): lesions are concentrated in a relatively small area, but the global perception ability still needs to be reserved; 2, the anatomical structure is fine and sensitive to down-sampling operation; however, full-size CT input easily causes unobvious features), overall and local branches can be mutually referenced and supplemented through a cross attention fusion mechanism, the feature capture ability of small focuses is significantly improved, effective cognition of the overall anatomical structure is maintained, and the probability that the features of the small focuses are not obvious is reduced. While the details of the lesion area in the orbit CT are efficiently and accurately captured, the overall anatomical background information is taken into account.

Description

Technical Field

[0001] This invention relates to a method for accurate detection of ocular lesions using CT scans. Background Technology

[0002] Orbital CT scans typically cover a large area; if thin-slice scanning is used, with each slice approximately 1 mm thick, about 100-150 slices are required. Figure 1 As shown. However, lesions are usually concentrated in only a few specific layers, especially the layer containing the orbit, which contains delicate structures such as the optic nerve (about 1.5-2 mm in diameter) and extraocular muscles (about 4 mm thick), and exists only in a few layers. When radiologists perform orbital CT lesion detection, although they need to focus primarily on the orbit, they should also report lesions on other scan planes, such as brain lesions, sinusitis, etc.

[0003] Traditional image processing methods typically address the above problems in the following ways: 1) Directly downsample the entire CT image, for example, compressing it from 128 layers to 32 layers, which is equivalent to extracting 1 layer from every 4 layers. However, since the diameter of the optic nerve is less than 2mm, it is only equivalent to 2 thin-slice CT images. This method is simple but will lead to the loss of important anatomical details. 2) No downsampling or only slight downsampling, but the model has difficulty effectively focusing on lesions, and it will bring huge computational costs and memory usage; 3) Training the model using only CT slices of the orbital region may result in a mismatch with the report description. For example, the report may mention "brain lesion", but the brain lesion is located outside the orbital region. In this case, "brain lesion" in the report becomes an incorrect label for that slice.

[0004] Therefore, traditional image processing methods downsample the complete CT data, but this method is prone to losing subtle anatomical information in key lesion areas. Conversely, if no downsampling is performed at all, the model struggles to focus on the lesion area, and the memory usage is extremely high. Summary of the Invention

[0005] The technical problem this invention aims to solve is that traditional single-branch ViT networks cannot simultaneously meet the global and local resolution requirements, while existing image processing methods lack effective global-local feature interaction mechanisms.

[0006] To address the aforementioned technical problems, the present invention discloses a precise detection method for ocular CT lesions using a dual-branch ViT model, characterized in that a dual-branch ViT model is employed to process ocular CT images, wherein: The bi-branch ViT model utilizes global branch and local branch features to obtain global branch features that provide overall anatomical background information and local branch features that can characterize detailed lesion features. The dual-branch ViT model uses a cross-attention mechanism to deeply fuse the overall branch features as the query with the local branch features as the key and value, thereby obtaining fused features that can simultaneously represent global contextual information and local lesion details. The bi-branch ViT model outputs the final ocular CT lesion prediction result based on fused features.

[0007] Preferably, the overall branch is processed at a lower resolution on the complete ocular CT image to obtain the overall branch features used to provide overall anatomical background information.

[0008] Preferably, the local branching process processes high-resolution CT images of the orbital lesion region to capture the local branching features that characterize detailed lesion features. Preferably, the overall branch is Layer stacking structure, .

[0009] Preferably, the local branch is Layer stacking structure, .

[0010] Preferably, each layer in the overall branch has the same structure as each layer in the local branch.

[0011] Preferably, the processing of the input data by the l-th layer in the overall branch or the l-th layer in the local branch includes the following steps: After performing layer normalization on the input of the l-th layer of the overall branch or the local branch, a multi-head self-attention operation is performed. The output of the multi-head self-attention operation is then residually connected with the original input of the l-th layer to obtain intermediate features. After performing layer normalization on the intermediate features, a multilayer perceptron operation is performed. The output of the multilayer perceptron operation is then residually connected with the original intermediate features to obtain the output of the l-th layer of the global branch or the local branch.

[0012] Preferably, when l=1, the input of the l-th layer of the overall branch is a complete ocular CT image, or the input of the l-th layer of the local branch is a high-resolution CT image of the orbital lesion area.

[0013] Preferably, when l>1, the input of the l-th layer in the overall branch or the l-th layer in the local branch is the output of the (l-1)-th layer.

[0014] Preferably, the high-resolution CT image of the orbital lesion area is obtained by the following steps: Based on the anatomical landmarks of the orbit, the acquisition area was defined as a cubic region extending from the supraorbital foramen or supraorbital notch superiorly, to the infraorbital foramen inferiorly, and posteriorly to the posterior segment of the optic foramen. The registered CT sequence is automatically cropped to extract slices from the cubic region determined in the previous step; The cropped area is the high-resolution CT image of the orbital lesion area.

[0015] The dual-branch ViT structure proposed in this invention is specifically designed and optimized for the characteristics of orbital CT (1. lesions are concentrated in a small area, but global perception capability still needs to be preserved; 2. the anatomical structure is delicate and sensitive to downsampling operations, but full-size CT input can easily lead to unclear features). Through the cross-attention fusion mechanism, the overall and local branches can refer to and complement each other, which significantly improves the feature capture capability of small lesions, while maintaining effective cognition of the overall anatomical structure. It can achieve efficient and accurate capture of the details of the lesion area in orbital CT while taking into account the overall anatomical background information. Attached Figure Description

[0016] Figure 1 The CT scan range of the orbit is shown (side view). Detailed Implementation

[0017] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0018] The present invention discloses a dual-branch visual Transformer method for accurate detection of ocular CT lesions. After obtaining fusion features that can simultaneously represent global contextual information and local lesion details using a dual-branch ViT model, the final ocular CT lesion prediction result is output based on the fusion features.

[0019] The dual-branch ViT model constructed in the above steps includes the following three parts: Global ViT: Processes complete CT images at a lower resolution, providing overall anatomical background information; Local ViT: Focuses on the orbital lesion area with high resolution, capturing detailed lesion features; Cross-Attention Mechanism: Through the cross-attention module, the features of the overall branch (as the query) and the features of the local branch (as the key and value) are deeply integrated, enabling the model to obtain both global contextual information and local lesion details at the same time.

[0020] (a) Overall Branches The total number of branches is Layers, for example The operations for each layer are as follows:

[0021]

[0022] Except for the first layer, the input of the l-th layer is the output of the (l-1)-th layer. The input to the first layer is the sequence of complete CT image blocks after embedding.

[0023] For the output of layer l-1 First, perform layer normalization, represented as: Through the Normalization stabilizes the training process and accelerates convergence. After normalization, the mean and variance of the feature distribution are adjusted, reducing internal covariate bias.

[0024] Then, a multi-head self-attention operation is performed, represented as... By calculating attention weights between different locations in the sequence, global dependencies are captured, enabling the model to integrate overall anatomical background information from complete CT images, such as areas like the brain, eyes, and sinuses.

[0025] Then, a residual connection is performed to connect the output of the multi-head self-attention system to the original input. Addition, represented as To avoid the gradient vanishing problem and thus obtain intermediate features .

[0026] For intermediate features Perform layer normalization, represented as .

[0027] Subsequently, a multilayer perceptron operation is performed, represented as... Features are further extracted through nonlinear transformation.

[0028] Finally, residual connections are used again to connect the output of the multilayer perceptron with the intermediate features. Addition, represented as To obtain the final features of the output of the l-th layer. .

[0029] By the above Layer stacking and overall branching gradually learn the global contextual features of the complete CT at low resolution.

[0030] (ii) Local branches Similar to global branches, local branches have a total of Layers, for example The operations for each layer are as follows:

[0031]

[0032] The operations of each layer in the local branch are exactly the same as those in the global branch, including layer normalization, multi-head self-attention, residual connections, and multilayer perceptron. The difference is that the input of the first layer in the local branch is a high-resolution CT image of the orbital lesion area, rather than a complete CT image. This allows the local branch to focus on fine structures such as the optic nerve and extraocular muscles, capturing the local details of the lesion.

[0033] In this embodiment of the invention, the orbital lesion area required for the local branch is obtained by cropping from the complete CT image through a predefined anatomical range. The specific steps are as follows: Step 1, Orbital Region: Based on the anatomical landmarks of the orbit, the acquisition area is defined as a cubic region extending from the supraorbital foramen (or supraorbital notch) superiorly to the infraorbital foramen inferiorly and posteriorly to the posterior segment of the optic foramen. This ensures that the region covers all key orbital structures, such as the optic nerve and extraocular muscles. Step 2, Image cropping: Use image processing tools such as ITK-SNAP to automatically crop the registered CT sequence and extract slices within the cubic region.

[0034] Step 3, Resolution Processing: The cropped area is input into the local branch at its original high resolution (e.g., 128×128×32) to avoid information loss caused by downsampling. This method ensures that the local branch can focus on the lesion area while maintaining consistency with the spatial range of the overall branch.

[0035] (III) Cross-attention fusion mechanism The cross-attention fusion mechanism operates on the global branch features of the overall branch output and the local branch features of the local branch output as follows:

[0036] In the formula: This is a cross-attention mechanism; This is a holistic branch feature, representing global anatomical background information; The features of the local branches represent detailed information about local lesions; Q (query) uses the features of the overall branches as the query vector to locate the areas of interest in the global context; K (key) and V (value) both use the features of the local branches as key and value vectors to provide local details as a reference.

[0037] The cross-attention fusion mechanism generates attention weights by calculating the similarity between Q and K, and then uses these weights to perform a weighted summation on V. This enables queries in the overall branch to "retrieve" key details in local branches, achieving deep fusion of global and local features, and ultimately outputting... It is a fusion feature that includes both overall context and local lesion information, and is used for downstream classification tasks.

[0038] To verify the bibranch ViT model provided in this embodiment of the invention, a retrospective collection of 1000 orbital non-enhanced CT scans performed at the Ninth People's Hospital affiliated with Shanghai Jiao Tong University School of Medicine from January 2021 to December 2024 was conducted. All CT images were manually examined to ensure there were no missing image data, low-quality imaging, or obvious artifacts. Each CT scan was binary-labeled (present / absent) on the following four lesion tags, including: orbital fracture, proptosis, head trauma, and head lesions (including all head lesions such as brain and eye lesions).

[0039] From each orbital CT scan, a thin-slice sequence (0.5–1.5 mm thick) was extracted for analysis. The bed frame was removed using the maximum connected subgraph algorithm, and registration was performed using ITK-SNAP software. The acquisition range of the orbital region was defined as a cubic region extending superiorly to the supraorbital foramen (or supraorbital notch), inferiorly to the infraorbital foramen, and posteriorly to the posterior segment of the optic foramen.

[0040] This invention employs a deep learning model based on 3D Vision Transformer (3D-ViT) for orbital CT lesion classification. The model implementation directly utilizes the officially released PyTorch code library. The default architecture and initialized weights of this model were used in the experiments. The input data consisted of orbital CT sequences. Each orbital CT sample was uniformly resampled to 256×256×64 voxels, while orbital region samples were uniformly resampled to 128×128×32 voxels. The CT sequences were segmented into 16×16×16 3D blocks and embedding vectors were generated through linear projection. Pixel intensity values ​​were normalized to the [0, 1] interval. Model training followed the default parameter settings of the official library. Specifically, the cross-entropy loss function was used, the optimizer was AdamW, the initial learning rate was 3e-4, and the weight decay coefficient was 0.01. The model was trained for 100 epochs on the training set with a batch size of 8. 1000 orbital CT scans were randomly split into training and test sets at a 7:3 ratio. After training on the training set, the model was evaluated on the test set, with the AUC of five metrics being the primary evaluation indicator. To demonstrate the advantages of the proposed bi-branch ViT model, three parallel experimental groups were set up: a global single-branch group (complete CT only), a local single-branch group (orbital region CT only), and a bi-branch group (complete CT + orbital region CT). The results are shown in the table below.

[0041]

[0042] As can be seen, the bi-branch ViT model proposed in this invention exhibits the best performance in all four lesion detection tasks (two orbital tasks and two whole-head tasks).

Claims

1. A method for precise detection of ocular lesions using a dual-branch visual Transformer CT scanner, characterized in that, A bi-branch ViT model was used to process ocular CT images. Specifically, the bi-branch ViT model utilizes both global and local branch features to obtain global branch features that provide overall anatomical background information, and local branch features that characterize detailed lesion features. The bi-branch ViT model then uses a cross-attention mechanism to deeply fuse the global branch features (as the query) with the local branch features (as the key and value), resulting in fused features that simultaneously represent global contextual information and local lesion details. Based on these fused features, the bi-branch ViT model outputs the final ocular CT lesion prediction results.

2. The method for accurate detection of ocular lesions using dual-branch visual Transformer CT as described in claim 1, characterized in that, The overall branch is processed at a lower resolution on the complete ocular CT image to obtain the overall branch features used to provide overall anatomical background information.

3. The method for accurate detection of ocular lesions using dual-branch visual Transformer CT as described in claim 2, characterized in that, The local branch processes high-resolution CT images of orbital lesion areas to capture the local branch features that characterize detailed lesion features.

4. The method for accurate detection of ocular lesions using dual-branch visual Transformer CT as described in claim 3, characterized in that, The overall branch is Layer stacking structure, .

5. The method for accurate detection of ocular lesions using a dual-branch visual Transformer CT scanner as described in claim 4, characterized in that... The local branch is Layer stacking structure, .

6. The method for accurate detection of ocular lesions using dual-branch visual Transformer CT as described in claim 5, characterized in that, Each layer in the overall branch has the same structure as each layer in the local branch.

7. The method for accurate detection of ocular lesions using dual-branch visual Transformer CT as described in claim 6, characterized in that, The processing of input data by the l-th layer in the overall branch or the l-th layer in the local branch includes the following steps: after performing layer normalization on the input of the l-th layer of the overall branch or the local branch, a multi-head self-attention operation is performed, and the output of the multi-head self-attention operation is residually connected with the original input of the l-th layer to obtain intermediate features; after performing layer normalization on the intermediate features, a multilayer perceptron operation is performed, and the output of the multilayer perceptron operation is residually connected with the original intermediate features to obtain the output of the l-th layer of the overall branch or the local branch.

8. The method for accurate detection of ocular lesions using dual-branch visual Transformer CT as described in claim 7, characterized in that, When l=1, the input of the l-th layer of the overall branch is a complete ocular CT image, or the input of the l-th layer of the local branch is a high-resolution CT image of the orbital lesion area.

9. The method for accurate detection of ocular lesions using dual-branch visual Transformer CT as described in claim 8, characterized in that, When l>1, the input of the l-th layer in the overall branch or the l-th layer in the local branch is the output of the (l-1)-th layer.

10. The method for accurate detection of ocular lesions using a dual-branch visual Transformer CT scanner as described in claim 1, characterized in that, The following steps are used to obtain the high-resolution CT image of the orbital lesion area: Based on the anatomical landmarks of the orbit, a cubic region is defined as the acquisition range extending from the supraorbital foramen or supraorbital notch superiorly, to the infraorbital foramen inferiorly, and posteriorly to the posterior segment of the optic foramen; the registered CT sequence is automatically cropped, and slices are extracted from the cubic region determined in the previous step; the cropped region is the high-resolution CT image of the orbital lesion area.

Images