Spine 3D model reconstruction method based on multi-scale symmetric architecture learning

Abstract

The invention discloses a spine 3D model reconstruction method based on multi-scale symmetric architecture learning. The method comprises the following steps: rotating a spine X-ray image to a set angle; repeating the rotated X-ray images for D times and splicing the X-ray images to form a high-channel 3D volume; inputting the high-channel 3D volume into a generative adversarial network to obtain a 3D prediction model of the spine; standard deviations of different generated images of the 3D prediction model are calculated to evaluate quality differences of the generated images, and a generator is selected to globally optimize or locally optimize the 3D prediction model based on the quality differences; the 3D prediction model obtains a reconstruction model of the spine by minimizing the loss of the generator. According to the invention, the problem of low fidelity caused by excessive dependence on prior information in the existing method for reconstructing the three-dimensional structure of the spine in X-Ray is solved.

Description

Technical Field

[0001] The present invention relates to the field of biomedical image technology, and in particular to a method for reconstructing a 3D spine model based on multi-scale symmetric architecture learning. Background Art

[0002] Adolescent idiopathic scoliosis (AIS) is a common spinal disease in adolescents, and its etiology is usually related to unbalanced bone growth and development or genetic factors. Since the treatment process of scoliosis is long, regular imaging examinations are required to dynamically evaluate the progression of the disease. X-Ray imaging has become the preferred method for measuring scoliosis angles in clinical practice due to its advantages such as low radiation dose, low cost, and fast imaging speed. The morphology and curvature of the spine are usually assessed by taking anteroposterior and lateral X-Ray images. However, since the working principle of X-Ray images is to use rays to penetrate human tissues and project tissues of different densities onto a two-dimensional plane to generate images, the tissues will inevitably overlap in the image. This imaging feature makes it difficult to intuitively present the three-dimensional morphology of the anatomical structure of the spine, limiting the comprehensive analysis of complex structures.

[0003] In contrast, CT scans can clearly display the three-dimensional structure of the spine and provide more detailed diagnostic information. However, CT scans have a large radiation dose and are costly, making them unsuitable for long-term follow-up in adolescents. In addition, CT imaging is usually performed in the supine position, which may affect the natural curvature of the spine and thus bias the assessment of scoliosis. Therefore, reconstructing the three-dimensional structure of the spine from X-Rays has become a research direction that has attracted much attention in recent years.

[0004] Some early studies proposed methods that rely on manually annotated rib centerlines. These methods iteratively fit the spinal curves through numerical analysis and interpolation techniques. However, such methods rely heavily on manual intervention, resulting in inconsistent reconstruction results. With the development of deep learning, researchers began to use convolutional neural networks (CNNs) to capture deep image features and transform 2D X-Ray features into 3D spinal structures. Although these methods can extract semantically meaningful representations of the underlying 3D scene from 2D views, their asymmetric architectures may not fully capture all the information encoded in the input 2D images. This information loss weakens the correlation between the reconstructed 3D output and the original 2D input. In addition, this approach may lead to over-reliance on the model's internal prior distribution (e.g., Gaussian distribution) at the expense of inherent information in the 2D signal, thereby reducing the fidelity of the reconstruction.

[0005] The information disclosed in this background technology section is only intended to enhance the understanding of the overall background of the invention and should not be regarded as an acknowledgment or any form of suggestion that the information constitutes the prior art already known to a person skilled in the art. Summary of the invention

[0006] In order to overcome the defects of the existing technology, a method for reconstructing a 3D model of the spine based on multi-scale symmetric architecture learning is provided to solve the problem of low fidelity caused by over-reliance on prior information in the existing method of reconstructing the three-dimensional structure of the spine in X-Ray.

[0007] To achieve the above purpose, a method for reconstructing a 3D spine model based on multi-scale symmetric architecture learning is provided, comprising the following steps:

[0008] Rotating a plurality of orthogonal X-ray images of the spine to a collective angle at which the shooting angles of the X-ray images are consistent;

[0009] Repeating the rotated X-ray image D times and stitching the repeated X-ray images to form a high-channel 3D volume, so that the target depth of the high-channel 3D volume is adapted to the target depth of the 3D model of the spine;

[0010] Inputting the high-channel 3D volume into a generative adversarial network, performing preliminary feature extraction through the encoder 3D convolution of the generator of the generative adversarial network, performing deep feature learning of the encoder GLMamba block to extract features and restore high-resolution features, the decoder of the generator concatenates the extracted features with the restored high-resolution features, and fuses the final features into continuous values ​​through the convolution block of the decoder to obtain a 3D prediction model of the spine;

[0011] Calculate and obtain standard deviations of different generated images of the 3D prediction model to evaluate the quality difference of the generated images, and select the generator to globally optimize or locally optimize the 3D prediction model based on the quality difference;

[0012] The 3D prediction model obtains a reconstructed model of the spine by minimizing the loss of the generator.

[0013] Furthermore, the generator uses a U-shaped network as a basic architecture.

[0014] Furthermore, the generator globally optimizes the 3D prediction model including using a dual regularized optimal transfer learning method and a geometrically debiased Sinkhorn divergence as a cost function.

[0015] Furthermore, the generator locally optimizes the 3D prediction model by constructing a mask matrix to mark low-quality areas and performing different levels of optimization according to the value of the local standard deviation.

[0016] The beneficial effect of the present invention is that the spine 3D model reconstruction method based on multi-scale symmetric architecture learning of the present invention effectively retains the information content of the input 2D x-ray without relying on other prior information by fusing the symmetric reconstruction network of Mamba. First, the 2D view is preprocessed, and a rough registration connection is performed after the x-ray reaches the same depth as the matching 3D model by repeating the x-ray, and a mapping reconstruction framework of the 3D to 3D symmetric architecture is established. Then it is input into the designed generative adversarial network for same-dimensional translation, and the visual state space model (VSS) and CNN are effectively fused by the designed global-local Mamba (GLMamba) module for the generator, and embedded in the U-shaped structure to model the entire volume features of different scales so that the features can be better connected to the context to be truly retained. The spine 3D model reconstruction method based on multi-scale symmetric architecture learning of the present invention adopts a novel dynamic discrimination strategy, and discriminates global and local information at the same time without adding additional judges, which effectively increases the delicate degree of generating 3D spine models. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0018] Figure 1 A schematic diagram of a framework of a generative adversarial network model according to an embodiment of the present invention.

[0019] Figure 2 4 is an architectural diagram of a global-local Mamba (GLMamba) module according to an embodiment of the present invention.

[0020] Figure 3 This is a diagram of the architecture of a parallel Mamba-Convolution Mamba-CNN hybrid (MCP) module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It is also necessary to explain that, for ease of description, only the parts related to the invention are shown in the accompanying drawings.

[0022] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0023] Reference Figures 1 to 3 As shown, the present invention provides a method for reconstructing a 3D spine model based on multi-scale symmetric architecture learning, comprising the following steps:

[0024] S1. Rotate multiple orthogonal X-ray images of the spine to a collective angle with a consistent shooting angle of the X-ray images.

[0025] Rotate two orthogonal X-ray images of the spine (usually the anteroposterior and lateral views) to their respective shooting angles. Because the X-ray images are taken from different angles during acquisition, they need to be aligned by rotation for subsequent processing. This operation ensures that the X-ray images are consistent with their actual shooting angles so that they correctly reflect the actual status of the spine.

[0026] The initial angles of the two orthogonal X-ray images that were not rotated at first are both 0°. Since different X-ray images are taken from different perspectives and have corresponding geometric spatial physical meanings, they must be rotated to the corresponding geometric angles before subsequent processing.

[0027] S2. Repeat the rotated X-ray image D times and stitch the repeated X-ray images to form a high-channel 3D volume, so that the target depth of the high-channel 3D volume is adapted to the target depth of the 3D model of the spine.

[0028] Create a high-channel 3D volume that matches the target depth of the 3D model, re-establishing the symmetric architecture for the 3D model. For each 2D X-ray view, repeat D times to form a 3D data block, where D is the depth dimension of the 3D volume. Concatenate these repeated views in the channel dimension to form a high-channel volume that extends into 3D space.

[0029] Specifically, the input projection for each 2D X-ray view is I i ∈R C×H×W , where C is the number of channels, H is the height, and W is the width, and each view is repeated D times to match the depth of the output 3D object.

[0030] D stands for depth, the third dimension of a 3D volume, which is mathematically represented as: Get I i ∈R C ×H×W×D .

[0031] Then connect them in the channel dimension to form a 3D volume V with higher channels cat ∈R NC×H×W×D Each 2D input view is expanded into 3D space and then input into the network to form a mapping relationship with the corresponding target 3D volume to form a symmetric architecture.

[0032] S3. The high-channel 3D volume is input into the generative adversarial network, and preliminary feature extraction is performed through the encoder 3D convolution of the generator of the generative adversarial network. The global-local Mamba (GLMamba) block of the encoder performs deep feature learning to extract features and restore high-resolution features. The decoder of the generator concatenates the extracted features with the restored high-resolution features, and the final features are fused into continuous values ​​through the convolution block of the decoder to obtain a 3D prediction model of the spine.

[0033] The generator uses U-Net as the basic architecture. Specifically, the U-Net is used as the basic architecture of the generator, which contains an initialization layer and multiple global-local Mamba (GLMamba) blocks. Preliminary feature extraction is performed through 3D convolution, and then deep feature learning is performed through the global-local Mamba (GLMamba) block. At the end of each stage, the spatial resolution is reduced and the feature dimension is increased through the downsampling layer to extract higher-level feature representations. The decoding part concatenates the features extracted by the encoder with the restored high-resolution features, integrates spatial information and semantic information, and finally obtains the prediction results of continuous values ​​through 1×1×1 convolution blocks.

[0034] In this embodiment, the 2D projection I i The pseudo 3D volume V is formed by D overlapping connections cat Directly mapped to the real 3D pattern V, however, the key to achieving high-quality conversion is to make full use of volumetric data to capture the feature information of the three-dimensional space.

[0035] Unet is adopted as the basic architecture of the generator, and the encoder consists of an initialization layer and four global-local Mamba (GLMamba) blocks.

[0036] Given an input image V cat ∈R NC×H×W×D , where V cat It is the pseudo 3D data obtained during preprocessing. The initialization layer is the first step of the entire network. A 3D convolution with a kernel size of 7×7×7 and a stride of 2×2×2 is used for V cat Perform preliminary extraction to obtain shallow feature maps

[0037] After that, F0 is fed into the global-local Mamba (GLMamba) block step by step for four stages of deep representation learning. At the same time, at the end of each stage, the data feature dimension is downsampled from dim[l] to dim[l+1] with the help of the downsampling layer, achieving a doubling of the growth and reducing the spatial resolution to half of the original. Where l∈{1, 2, 3, 4}. This can reduce the number of tokens while maintaining the feature dimension, and reduce the computational complexity while extracting higher-level feature representations. In this way, the encoder can learn feature representations of different scales at each stage, thereby providing richer contextual information and multi-scale perception capabilities.

[0038] For the decoding part, the features extracted by the global-local Mamba (GLMamba) block in each layer of the encoder are sent to the residual block composed of two 3×3×3 convolutional layers. The further extracted fused features are concatenated with the features restored at 2 times the resolution in the previous stage along the channel dimension, and then input into the residual block together to integrate high-resolution spatial information and low-resolution semantic information, in preparation for feature recovery in the next stage. Finally, a 1×1×1 convolutional block is used to fuse the final features into continuous values ​​to obtain the final prediction result.

[0039] In this embodiment, the global-local Mamba (GLMamba) is used to enhance the effective fusion of the convolutional block (CNN) and the visual state space model (VSS) for extracting deep long-distance and local semantic features of the image.

[0040] Specifically, we use advanced residual connections to enhance the long-range spatial modeling capability of the proposed parallel Mamba-convolution (MCP) block. This introduces almost no new parameters and computational complexity. Next, the global-local mamba (GLMamba) block uses another normalization layer (LayerNorm) to normalize MLP is then used to optimize the feature mapping process from input to output and integrate these features into more abstract and useful information.

[0041] For the lth global-local Mamba (GLMamba) block in the above steps, the calculation process can be defined as: Where MCP represents the proposed parallel Mamba-CNN hybrid block (MCP block), LN represents layer normalization, and MLP represents multi-layer perception layer.

[0042] In this embodiment, an efficient parallel Mamba-CNN hybrid block (MCP block) uses a visual state space model (VSS) to demonstrate Mamba's potential for long sequence modeling in processing images.

[0043] Assume the input tensor is

[0044] First, it is input into a 1×1×1 convolutional layer whose output channel number is also C for preliminary feature extraction.

[0045] Then the tensor is evenly split into two tensors fCNN and f VSS , the size is This operation has two benefits. First, the number of feature channels subsequently input to the CNN and visual state space model (VSS) modules is halved, which reduces the model complexity. Second, local and non-local features can be processed independently and in parallel, which is conducive to better feature extraction.

[0046] Afterwards, the tensor f CNN Send it to the residual network to get At the same time, f VSS Sent to the visual state space model (VSS) module to get Then, connect and To restore it to a tensor of the same size as the input And pass through a 1×1×1 convolution block to fuse local and non-local features.

[0047] Finally, a skip connection is established between f and the output to obtain the final learned fusion feature f.

[0048] The calculation process of the above steps is shown as follows:

[0049] f CNN ,f VSS =Split(Conv 1×1×1 (f)) (1);

[0050]

[0051] Where Split represents the segmentation operation, Conv 1×1×1 represents a 1×1×1 3d convolutional layer, Represents tensor concatenation operation, Res represents residual network, and VSS represents visual state space module.

[0052] S4. Calculate the standard deviation of different generated images of the 3D prediction model to evaluate the quality difference of the generated images, and select the generator to globally optimize or locally optimize the 3D prediction model based on the quality difference.

[0053] By evaluating the quality differences of the generated images, the optimization strategy is dynamically adjusted to improve the reconstruction quality. The standard deviation between images is used to evaluate the quality differences, and global or local optimization is selected.

[0054] The generator global optimization 3D prediction model includes an optimal transfer learning method using dual regularization and geometrically debiased Sinkhorn divergence as the cost function.

[0055] Generator Local Optimization The 3D prediction model includes constructing a mask matrix to mark low-quality areas and performing different levels of optimization depending on the value of the local standard deviation.

[0056] Specifically, since the quality difference between images can reflect the difference in feature distribution between some images as a whole and the real images, the standard deviation σ between different images is used to select different optimization modes. The specific calculation is as follows:

[0057]

[0058] Among them, μ k The mean between different receptive fields within the image, μ is the mean quality of all images.

[0059] If σ is high, it means that there is a large range of quality differences between images, so global optimization should be performed, because in the generative network, images are learned from coarse to fine; on the contrary, this batch of generated images are relatively close in receptive field, and the overall performance is relatively uniform. At this time, it is necessary to focus on local optimization and adjust the feature performance of each image in the local area in a targeted manner to make it more consistent.

[0060] Therefore, for global optimization, we directly pursue the process of model optimization:

[0061]

[0062] Among them, S ε It is an optimal transfer learning (OT) learning method with double regularization and uses the Sinkhorn divergence with geometric debiasing as the cost function to measure the difference between the generator's predicted features and the true features; G is the generator, D is the discriminator, f is the operation function of the discriminator output (using the hinge loss function), and λ is a parameter that controls the degree of regularization.

[0063] For local optimization, a mask matrix h of the same size as the discriminator output is constructed * to mark low-quality areas. Compared with the quality difference between images, the quality difference between different receptive fields of the whole image can be a good measure of local differences. If the quality difference between different receptive fields in the same image is large, it means that some local areas in the image are generated well, while other areas may have defects. Therefore, the local optimization is divided into three levels according to the value of the local standard deviation σ Ⅰ ,α Ⅱ ,α Ⅲ , corresponding to the area generated by the mask. The standard deviation σ is calculated as follows:

[0064]

[0065] The higher the level (level III), the larger the local optimization range defined (the larger the α constant), so when the output value is lower than α, the corresponding value in the mask matrix is ​​1, otherwise it is 0. Therefore, if σ is small, the optimization process is transformed into the following formula:

[0066]

[0067] in represents the dot product, H = {h1,h2,…,h n ,…} is a set of mask matrices, which is equivalent to the generator finding the optimal low-quality region h in gradient descent * , and h * It is possible to clarify the direction of generator optimization and try to adjust its parameters so that the discriminator output on these low-quality areas (after operations such as with the mask matrix) can change in a direction closer to the real image, thereby reducing the low quality of these areas and ultimately achieving the goal of reducing losses.

[0068] S5. The 3D prediction model obtains the reconstructed model of the spine by minimizing the loss of the generator.

[0069] Based on the feedback from the discriminator, the parameters of the generator are adjusted. A loss function is used to measure the difference between the predicted features of the generator and the true features, and these parameters are optimized to reduce the difference.

[0070] Specifically, the loss calculation for the discriminator is as follows:

[0071]

[0072] The spine 3D model reconstruction method based on multi-scale symmetric architecture learning of the present invention effectively retains the information content of the input 2D x-ray without relying on other prior information by fusing the symmetric reconstruction network of Mamba. First, the 2D view is preprocessed, and a rough registration connection is performed after the x-ray reaches the same depth as the matching 3D model by repeating the x-ray, and a mapping reconstruction framework of the 3D to 3D symmetric architecture is established. Then it is input into the designed generative adversarial network for same-dimensional translation, and the visual state space model (VSS) and CNN are effectively fused by the designed global-local Mamba (GLMamba) block for the generator, and embedded in the U-shaped structure to model the entire volume features of different scales so that the features can be better connected to the context to be truly retained. The spine 3D model reconstruction method based on multi-scale symmetric architecture learning of the present invention adopts a novel dynamic discrimination strategy, and discriminates global and local information at the same time without adding additional judges, which effectively increases the delicate degree of generating 3D spine models.

[0073] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the inventive concept. For example, the above features are replaced with (but not limited to) technical features with similar functions disclosed in the present application.

Claims

1. A method for reconstructing a 3D spine model based on multi-scale symmetric architecture learning, characterized in that: The following steps are involved: Rotating a plurality of orthogonal X-ray images of the spine to a collective angle at which the shooting angles of the X-ray images are consistent; Repeating the rotated X-ray image D times and stitching the repeated X-ray images to form a high-channel 3D volume, so that the target depth of the high-channel 3D volume is adapted to the target depth of the 3D model of the spine; Inputting the high-channel 3D volume into a generative adversarial network, performing preliminary feature extraction through the encoder 3D convolution of the generator of the generative adversarial network, performing deep feature learning of the encoder GLMamba block to extract features and restore high-resolution features, the decoder of the generator concatenates the extracted features with the restored high-resolution features, and fuses the final features into continuous values ​​through the convolution block of the decoder to obtain a 3D prediction model of the spine; Calculate and obtain standard deviations of different generated images of the 3D prediction model to evaluate the quality difference of the generated images, and select the generator to globally optimize or locally optimize the 3D prediction model based on the quality difference; The 3D prediction model obtains a reconstructed model of the spine by minimizing the loss of the generator.

2. The method for reconstructing a 3D spine model based on multi-scale symmetric architecture learning according to claim 1, characterized in that: The generator uses a U-shaped network as the basic architecture.

3. The method for reconstructing a 3D spine model based on multi-scale symmetric architecture learning according to claim 1, characterized in that: The generator globally optimizes the 3D prediction model including using a dual regularized optimal transfer learning method and geometrically debiased Sinkhorn divergence as a cost function.

4. The method for reconstructing a 3D spine model based on multi-scale symmetric architecture learning according to claim 1, characterized in that: The generator locally optimizes the 3D prediction model including constructing a mask matrix to mark low-quality areas and performing different levels of optimization according to the value of the local standard deviation.

Images