Image segmentation-based vertebral body three-dimensional reconstruction method, bone detection method and system

Abstract

The invention relates to the technical field of image processing, and provides a vertebral body three-dimensional reconstruction method based on image segmentation, and a bone detection method and system.The method comprises the steps that firstly, an SAD-UNetR image segmentation model capable of achieving accurate segmentation of an image target is constructed through improvement of a Swinin-UNetR network, three-dimensional modeling is conducted on the basis of an accurate segmentation image, and a three-dimensional reconstruction model is obtained; an accurate three-dimensional model of a vertebra structure can be obtained; based on an accurate segmentation result, a bone mineral density prediction regression model is provided at the same time, bone substances are identified based on multi-instance learning and an attention mechanism, a bone substance classification result and bone mineral density data can be provided, intelligent assistance is provided for diagnosis of doctors, radiation risks, economic consumption and time consumption caused by special bone mineral density measurement are avoided, and the accuracy of bone mineral density measurement is improved. Measurement means of bone detection are expanded, and the method has a wide application prospect.

Description

Technical Field

[0001] The present disclosure relates to the technical field related to image processing, and in particular to a vertebral three-dimensional reconstruction method, a bone detection method and a system based on image segmentation. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art.

[0003] Deep learning provides powerful tools and methods for medical image analysis, especially image segmentation, which significantly improves the accuracy and efficiency of automated analysis. This not only reduces the workload of doctors, but also improves the effectiveness of diagnosis and treatment, and promotes the development and application of medical imaging technology.

[0004] Most of the existing medical auxiliary diagnosis tools focus on the diagnosis of organs such as the abdomen and brain, and there are few medical auxiliary systems for lumbar spine-related diseases. The inventor found in the study that the existing medical auxiliary diagnosis system uses deep learning U-shaped networks such as Swin-UNetR network, U-Net network and other model networks for image processing. It cannot accurately identify the structural boundaries of target organs and tissues during automatic segmentation. The overall segmentation accuracy is low, and the position and channel features of the image cannot be integrated, resulting in inaccurate segmentation results. Inaccurate segmentation results may cause the reconstruction results to be distorted, missing or blurred, thereby affecting the identification and diagnosis of the lesion site. In addition, the human-computer interaction interface of the existing auxiliary diagnosis system is unfriendly and has complex operation procedures, unintuitive interface design and other problems. Doctors may need a long learning time when operating, which reduces the efficiency of use. Thereby affecting the doctor's experience. Summary of the invention

[0005] In order to solve the above problems, the present disclosure proposes a vertebral three-dimensional reconstruction method, bone detection method and system based on image segmentation, improves the existing image segmentation model, performs high-precision segmentation and three-dimensional reconstruction of CT images, and can observe the patient's vertebral and soft tissue structure in a more intuitive and accurate manner, thereby accurately identifying important information such as the lesion site and nerve course; at the same time, based on the structure after image segmentation, bone detection is realized by adopting multi-instance learning and attention mechanism, which can provide bone classification results and bone density data, provide doctors with three-dimensional model images and bone detection data for diagnosis, and realize efficient intelligent medical assistance.

[0006] In order to achieve the above objectives, the present disclosure adopts the following technical solutions:

[0007] The first aspect of the present disclosure provides a method for three-dimensional reconstruction of a vertebra based on image segmentation, comprising the following steps:

[0008] Acquire spinal CT plain scan image data to be processed;

[0009] The image data is input into the improved image segmentation model based on the Swin-UNetR network to obtain the segmented image of the spinal CT image;

[0010] The improved image segmentation model adds an AWM grayscale adaptive network before the encoder of the Swin-UNetR network to adaptively adjust the grayscale of the input image; and sets a DA-Block module on the jump connection after the encoder to extract the position features and channel features of the image;

[0011] For the segmented images, the three-dimensional reconstruction method is used to process the image data and extract information to construct a three-dimensional model of the vertebral structure.

[0012] The second aspect of the present disclosure provides a bone detection method based on image segmentation, comprising the following steps:

[0013] Acquire spinal CT plain scan image data to be processed;

[0014] The image data is input into the improved image segmentation model based on the Swin-UNetR network to obtain the segmented image of the spinal CT image;

[0015] The improved image segmentation model adds an AWM grayscale adaptive network before the encoder of the Swin-UNetR network to adaptively adjust the grayscale of the input image; and sets a DA-Block module on the jump connection after the encoder to extract the position features and channel features of the image;

[0016] For the segmented image, patch location proposal is performed, and instances are selected from the segmented image to form bags as instance-level feature bags;

[0017] Transform instance-level features in instance-level feature bags through patch networks;

[0018] Combine the transformed instances through attention multi-instance learning and pooling;

[0019] Classification is performed based on the combined bag-level features to obtain the degree of porosity and the corresponding bone density value.

[0020] The third aspect of the present disclosure provides a vertebral three-dimensional reconstruction system based on image segmentation, comprising:

[0021] A data acquisition module is configured to acquire spinal CT plain scan image data to be processed;

[0022] Image segmentation module: configured to input image data into an improved image segmentation model based on the Swin-UNetR network to obtain a segmented image of the spinal CT image;

[0023] The improved image segmentation model adds an AWM grayscale adaptive network before the encoder of the Swin-UNetR network to adaptively adjust the grayscale of the input image; and sets a DA-Block module on the jump connection after the encoder to extract the position features and channel features of the image;

[0024] Three-dimensional modeling module: It is configured to process the image data and extract information using a three-dimensional reconstruction method for the segmented image, and to construct a three-dimensional model of the vertebral structure.

[0025] A fourth aspect of the present disclosure provides a bone detection system based on image segmentation, comprising:

[0026] A data acquisition module is configured to acquire spinal CT plain scan image data to be processed;

[0027] Image segmentation module: configured to input image data into an improved image segmentation model based on the Swin-UNetR network to obtain a segmented image of the spinal CT image;

[0028] The improved image segmentation model adds an AWM grayscale adaptive network before the encoder of the Swin-UNetR network to adaptively adjust the grayscale of the input image; and sets a DA-Block module on the jump connection after the encoder to extract the position features and channel features of the image;

[0029] Instance-level feature generation module: configured to propose patch locations for the segmented image and select instances from the segmented image to form bags as instance-level feature bags;

[0030] Transformation module: configured to transform instance-level features in the instance-level feature bag through a patch network;

[0031] Instance combination module: configured to combine transformed instances through attention multi-instance learning and pooling;

[0032] Prediction output module: It is configured to perform classification based on the combined bag-level features to obtain the degree of osteoporosis and the corresponding bone density value.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] The present invention improves the Swin UNetR network model to obtain a SAD-UNetR image segmentation model with image-specific features that integrate position and channel and adaptively adjust the grayscale value of the input image, and integrates a DA-Block module in the jump connection after the encoder of the SAD-UNetR image segmentation model; this module focuses on extracting the position information and channel features of the image, and can more accurately capture the spatial and semantic information of the target area and improve the accuracy of the segmentation task. The accurate segmentation of the spinal image was successfully achieved, and the three-dimensional modeling based on the accurate segmentation results can obtain an accurate three-dimensional model.

[0035] At the same time, based on accurate image segmentation results, the fusion of advanced multi-instance learning and attention mechanism provides an efficient and accurate bone detection method, which effectively improves the performance of osteoporosis classification and bone density prediction. It can effectively capture local and global structural features to improve the performance of osteoporosis classification and bone density prediction, distinguish different states of normal bone mass, osteopenia and osteoporosis, and accurately predict bone density values.

[0036] The advantages of the present disclosure and the advantages of additional aspects will be described in detail in the following specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings constituting a part of the present disclosure are used to provide a further understanding of the present disclosure. The exemplary embodiments of the present disclosure and the description thereof are used to explain the present disclosure but do not constitute a limitation of the present disclosure.

[0038] Figure 1 is a flow chart of a vertebral three-dimensional reconstruction method based on image segmentation according to Embodiment 2 of the present disclosure;

[0039] Figure 2 Schematic diagram of the structure of the SAD-UNetR image segmentation model of Embodiment 1 of the present disclosure;

[0040] FIG3( a ) is a schematic diagram of the structure of an improved Swin Transformer model of Example 1 of the present disclosure;

[0041] FIG3( b ) is a schematic diagram of the structure of the AWM grayscale adaptive network of Embodiment 1 of the present disclosure;

[0042] Figure 4 is a schematic diagram of the structure of a three-dimensional DA-Block dual attention module of Embodiment 1 of the present disclosure;

[0043] FIG5( a ) is a medical modeling diagram of a human lumbar spine structure in the prior art;

[0044] FIG5( b ) is a flowchart of 3D modeling after image segmentation in Example 1 of the present disclosure;

[0045] Figure 6 is a flow chart of a bone detection method according to Embodiment 2 of the present disclosure;

[0046] Figure 7 It is a schematic diagram of the structure of the bone density prediction regression model DA-MIDL++ of Example 2 of the present disclosure;

[0047] Figure 8 is a comparison diagram of image segmentation results using Swin UNet in the experiment of Example 1 of the present disclosure and SAD-UNetR of the present embodiment;

[0048] Fig. 9 It is a schematic diagram of the structure of the human-computer interaction platform of Example 2 of the present disclosure. DETAILED DESCRIPTION

[0049] The present disclosure is further described below in conjunction with the accompanying drawings and embodiments.

[0050] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present disclosure belongs.

[0051] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof. It should be noted that, in the absence of conflict, the various embodiments in the present disclosure and the features in the embodiments can be combined with each other. The embodiments will be described in detail below in conjunction with the accompanying drawings.

[0052] Example 1

[0053] In the technical solutions disclosed in one or more embodiments, Figures 1 to 9 As shown, the vertebral three-dimensional reconstruction method based on image segmentation includes the following steps:

[0054] Step 1, obtaining the spinal CT plain scan image data to be processed;

[0055] Step 2: Input the image data into the improved image segmentation model based on the Swin-UNetR network to obtain a segmented image of the spinal CT image; the segmentation result includes the segmented target areas of the vertebral body, intervertebral disc and nerve;

[0056] The improved image segmentation model adds an AWM grayscale adaptive network before the encoder of the Swin-UNetR network to adaptively adjust the grayscale of the input image; a DA-Block module is set on the jump connection after the encoder to extract the position features and channel features of the image;

[0057] Step 3: Using a 3D reconstruction method to process the segmented image data and extract information to construct a 3D model of the vertebral structure;

[0058] In this embodiment, the Swin UNetR network model in this embodiment is improved to obtain a SAD-UNetR image segmentation model with adaptive adjustment of the grayscale of the input image, and a DA-Block module is integrated in the jump connection after the encoder of the SAD-UNetR image segmentation model; this module focuses on extracting the location information and channel features of the image, and can more accurately capture the spatial and semantic information of the target area and improve the accuracy of the segmentation task. The accurate segmentation of the spinal image is successfully achieved, and the three-dimensional modeling based on the accurate segmentation results can obtain an accurate three-dimensional model;

[0059] A further technical solution is to pre-process the acquired spinal CT medical image before image segmentation, mainly including cropping, gray value adjustment and removal of uninteresting areas. The specific pre-processing process is as follows:

[0060] 1) Cropping image size: Cropping the acquired spinal CT plain scan image to a set size;

[0061] Specifically, the original 512*512 CT image is cropped to a 256*256 CT image;

[0062] This embodiment achieves lightweight processing by reducing the size of the image while retaining key information areas. Through lightweight processing, the data structure can be effectively simplified, redundant information can be reduced, and more efficient data support can be provided for the operation of the algorithm, thereby improving the efficiency of the segmentation algorithm and the accuracy of three-dimensional reconstruction.

[0063] 2) Grayscale value adjustment: set the grayscale value range of the cropped image between -200 and 200;

[0064] This embodiment can effectively highlight the structural details in the image by adjusting the grayscale value range, making the target area more visible and enhancing the contrast and clarity of the image. By limiting the grayscale value range, the common bone artifact phenomenon in CT images can be effectively removed.

[0065] Bone artifacts are image artifacts caused by the scattering effect of X-rays when passing through bone tissue, which will affect the quality and accuracy of the image. By adjusting the grayscale value range, this artifact can be reduced or removed, further improving the clarity and accuracy of the image and providing clearer input data for segmentation and 3D reconstruction algorithms.

[0066] 3) Remove slices of no interest: Remove slices of no interest in the image, such as surrounding blank areas and irrelevant structures.

[0067] By identifying and removing regions of no interest, interference can be reduced, key structures can be highlighted, and the size and complexity of image data can be reduced, reducing the consumption of computing memory. Reducing the redundant parts of image data is conducive to building clinical workstations. Streamlined image data sets can improve the response speed and performance of workstations, providing clinicians with a smoother and more efficient medical imaging diagnosis and processing environment. It provides a more efficient data processing foundation for subsequent segmentation and 3D reconstruction tasks.

[0068] 4) Standardized image: The average pixel value is subtracted from the pixel value of each point in the image and then divided by the standard deviation of the pixel value to normalize the spinal CT plain scan image;

[0069] In this embodiment, by preprocessing the CT images, the data can be optimized and the data quality can be improved, providing a clearer and more accurate data basis for subsequent segmentation and 3D reconstruction tasks. The optimized data helps the segmentation algorithm to more accurately identify and segment the target structure, while providing better input for the 3D reconstruction algorithm, improving the accuracy and visualization of the segmentation and reconstruction effects.

[0070] Furthermore, the improvement of the Swin-UNetR network also includes setting a three-dimensional DA-Block dual attention module in the corresponding layers of the encoder and decoder;

[0071] The improved Swin-UNetR network includes a multi-layer encoded encoder and a corresponding connected decoder to form a U-shaped network structure. The output end of each layer of the encoder is provided with a three-dimensional DA-Block dual attention module, which is connected to the corresponding decoding layer of the decoder through the three-dimensional DA-Block dual attention module to obtain the SAD-UNetR image segmentation model.

[0072] Furthermore, the encoder of the SAD-UNetR image segmentation model adopts an improved Swin Transformer model; the encoder of the SAD-UNetR image segmentation model combines the self-attention mechanism of Swin Transformer and the U-shaped network design, which can effectively extract feature representations in three-dimensional CT images and pass these representations to the decoder through jump connections. The decoder is combined with the encoder through U-shaped network design and jump connections, which can effectively convert the feature representation extracted by the encoder into the final segmentation result.

[0073] In this embodiment, an improved Swin Transformer model, an AWM grayscale adaptive network, and a three-dimensional DA-Block dual attention module are introduced on the basis of the U-shaped network. Grayscale adaptive adjustment, multi-layer encoding, and decoding are performed in sequence based on the SAD-UNetR image segmentation model, wherein after each layer of feature encoding, the output encoding features are input to the decoder through a dual-channel attention operation for decoding to obtain a segmented image.

[0074] In the SAD-UNetR image segmentation model, the input is a three-dimensional CT image.

[0075] For the input 3D CT image, the grayscale value is first adaptively adjusted through the AWM grayscale adaptive network;

[0076] In some embodiments, as shown in FIG3( b ), the AWM grayscale adaptive network includes an input layer, a first convolution layer, two parallel processing branches, a second convolution layer of each branch, a global average pooling layer (Global Average Pooling, GAP), a sigmoid activation layer, and an addition fusion module; the outputs of the two processing branches are connected to a fully connected layer (Normalization) and a threshold processing layer (Thresholding);

[0077] A feasible technical solution is to adaptively adjust the grayscale of the image based on the AWM grayscale adaptive network, including the following steps:

[0078] Step 21, the input image x is sent to the first convolutional layer (Conv), and the convolution operation is performed to generate a feature map f, and the feature dimension is: N×C×D×H×W;

[0079] Step 22: After two parallel second convolutional layers, the two sets of convolutional layers further process the feature maps to generate features of different scales, and the feature dimensions are: N×1×D×H×W;

[0080] Step 23, the outputs of the two second convolutional layers are processed in two parallel branches respectively, and the features of the two branches are added after global average pooling and sigmoid activation respectively, and the weights Ww and Wc are scaled based on the two parameters α and β, and the grayscale adjustment feature map is obtained based on the weights;

[0081] During network training, Ww and Wc are scaled by learning two parameters α and β.

[0082] in, Figure 4 The t in is a hyperparameter used to adjust the learned parameters α and β. The role of t is to ensure that the learned parameters are within a reasonable range to avoid excessive bias towards certain grayscale values, thereby affecting the segmentation effect.

[0083] Step 24, perform a full connection operation (Normalization) on the grayscale adjustment feature map and the original input image to obtain a feature map after adaptive grayscale adjustment, and generate an output image X′ through threshold processing;

[0084] In this embodiment, the AWM grayscale adaptive network is set. Figure 2 Abbreviated as AWM in the text, it improves the contrast of the corresponding structure by adaptively adjusting the grayscale value in the image, thereby adjusting the distribution of grayscale values, improving the learning ability of the basic network in soft tissues, and effectively improving the segmentation effect of nerves and intervertebral discs. In medical image processing, the AWM grayscale adaptive network can adaptively increase the attention to different tissues and lesion areas, assist the model to focus on important features, and effectively suppress the influence of noise or irrelevant parts.

[0085] In some embodiments, the network structure of the three-dimensional DA-Block (Dual Attention Block) dual attention module (hereinafter referred to as DA-Block module) can be as follows: Figure 4 As shown, including:

[0086] 2.1) Input layer: used to input the feature map output by the encoder, a four-dimensional tensor with a shape of (n,n,n,n);

[0087] 2.2) Feature extraction upper branch: including the third convolutional layer, the position attention module (PAM) and the fourth convolutional layer connected in sequence;

[0088] The third convolutional layer (Conv): The third convolutional layer performs a convolution operation on the input tensor and outputs three feature maps B, C, and D with an output shape of (n,n / 16,n,n);

[0089] Position Attention Module (PAM): It is used to combine and calculate the three feature maps B, C and D output by the third convolutional layer based on the position attention operation to obtain the PAM output feature map with a shape of (n,n / 16,n,n);

[0090] The fourth convolutional layer (Conv): The output feature map of PAM is convolved through the fourth convolutional layer to obtain the upper branch feature map, and the feature map maintains the shape of (n,n / 16,n,n);

[0091] 2.3) Feature extraction lower branch: including the fifth convolutional layer, the channel attention module (CAM) and the sixth convolutional layer;

[0092] Fifth convolutional layer (Conv): The input tensor passes through a convolutional layer, and the output shape is (n,n / 16,n,n);

[0093] Channel Attention Module (CAM): The feature map output by the fifth convolutional layer is input into two fully connected layers, and two different feature representations are obtained after being processed by the activation function. The two feature representations are combined and calculated to obtain the CAM output feature map with a shape of (n,n / 16,n,n); the CAM output feature map passes through the sixth convolutional layer to obtain the next branch feature map, and the shape is maintained as (n,n / 16,n,n);

[0094] 2.4) Feature fusion module: The upper branch feature map and the lower branch feature map are added and fused element by element to obtain a fused feature map with a shape of (n,n / 16,n,n).

[0095] 2.5) Output layer: The fused feature map is subjected to a convolution operation to output the final feature map with a shape of (n,n,n,n).

[0096] Based on the above overall structure, DA-Block can fully utilize the advantages of position and channel features to enhance the feature extraction capability of the model.

[0097] The above DA-Block module includes a feature position extraction block (PAM) and a channel feature extraction block (CAM), and all convolutions in the DA-Block module adopt depthwise separable convolution;

[0098] In PAM, the local feature A output by the encoder is first input, and after being processed by the convolutional layer, three new feature maps are obtained, namely B, C and D. Then, these three feature maps are combined and calculated to obtain the output of the position attention module.

[0099] The role of PAM is to capture the spatial dependency between any two positions in the feature map and update specific features by the weighted sum of all position features. These weights are determined by the feature similarity between the two positions, so PAM is very effective in extracting meaningful spatial features.

[0100] In CAM, the input feature passes through a convolution layer to obtain a new feature map. Then, this feature map is input into two fully connected layers respectively, and two different feature representations are obtained after being processed by the activation function. These two feature representations are then combined and calculated to obtain the output of the channel attention module. The role of CAM is to capture the correlation between channels in the feature map and extract meaningful channel features through weighted sum.

[0101] In the above scheme, the DA-Block module, as a feature extraction module, integrates image-specific features of position and channel, thereby realizing feature extraction for unique attributes of the image. Especially in the U-Net-shaped architecture, the professional feature extraction capability of DA-Block is crucial. The Swin Transformer is good at extracting global features using the attention mechanism, but it is not specifically for image-specific attributes. In contrast, DA-Block excels in feature extraction based on position and channel, making it possible to obtain a more detailed and accurate feature set. Therefore, in this embodiment, the DA-Block module is set at the jump connection to enhance the segmentation of the model.

[0102] Each layer of the encoder in the SAD-UNetR image segmentation model includes Linear Embed, Block, and Merging Blocks connected in sequence, where Block adopts the improved Swin Transformer model;

[0103] Linear Embed: Map patches to an embedding space to generate feature representations.

[0104] Improved Swin Transformer model: This module extracts features through the self-attention mechanism and uses a moving window for calculation to effectively capture long-distance dependencies.

[0105] Merging Blocks: At different stages, merging blocks are used to reduce the resolution of feature maps and merge features;

[0106] In this embodiment, the improved Swin Transformer model proposes an important innovation, as shown in Figure 3(a), that is, the traditional normalization layer (LayerNorm, abbreviated as LN in the figure) is replaced with a re-parameterized batch normalization layer (BatchNorm) in the Swin Transformer architecture. This improvement can solve the computational overhead problem of LayerNorm in the inference phase while maintaining the training stability and performance of the model. The innovative method of PREpBN is used to gradually replace the normalization layer (LayerNorm) in Swin Transformer with a re-parameterized batch normalization layer (BatchNorm) to improve training stability and computational efficiency.

[0107] The normalization layer (LayerNorm) is a commonly used normalization method in the Transformer architecture, which can stabilize the learning of the model during training. However, the normalization layer (LayerNorm) requires statistical calculations during inference, which leads to higher computational latency, especially on resource-constrained devices. In contrast, BatchNorm can be merged with the linear layer during the inference phase, thereby significantly reducing the inference latency, but using BatchNorm directly may cause training crashes and performance degradation.

[0108] Batch Normalization (BatchNorm for short): Improve the stability and convergence speed of the model by standardizing the input of each small batch.

[0109] In order to effectively replace the normalization layer (LayerNorm) with batch normalization (BatchNorm), this embodiment proposes a gradual replacement strategy (PRepBN). In the early stage of training, the model mainly relies on the normalization layer (LayerNorm) to ensure the stability of training. As the training progresses, we gradually increase the proportion of batch normalization (BatchNorm), and finally achieve full BatchNorm at the end of training.

[0110] The formula for the gradual replacement strategy is: PRepBN(X)=γLN(X)+(1-γ)RepBN(X). This process is controlled by a hyperparameter γ, which is set to 1 at the beginning of training and gradually reduced to 0 to achieve a smooth transition.

[0111] We also introduce Reparameterized Batch Normalization (RepBN), which is formulated as: RepBN(X) = RepBN(X) + ηX, where η is a learnable parameter that can be jointly optimized during training. In this way, RepBN not only maintains the advantages of Batch Normalization (BatchNorm), but also enhances the stability and performance of the model during training.

[0112] In this embodiment, the innovative method of PRepBN is used to gradually replace the normalization layer (LayerNorm) in the Swin transformer with a re-parameterized batch normalization layer (BatchNorm), so as to improve training stability and computational efficiency. This improvement not only improves the stability of model training, but also significantly improves computational efficiency, so that the model can converge more quickly and process larger data sets.

[0113] In the above solution, replacing the normalization layer (LayerNorm) with the re-parameterized batch normalization (BatchNorm) not only improves the reasoning efficiency of the Swin Transformer, but also ensures the stability of training through a gradual replacement strategy. This innovation provides new ideas for the design of efficient Transformer architectures and has important theoretical and practical application value.

[0114] The Swin Transformer Block contains the following modules:

[0115] W-MSA (Windowed Multi-Head Self-Attention): This module calculates self-attention within a fixed-size window, allowing the model to capture feature relationships in a local area. By using multiple heads, the model can learn different feature representations.

[0116] SW-MSA (Moving Window Multi-Head Self-Attention): When calculating self-attention, the window moves between each layer to capture the feature relationship across windows. This mechanism enhances the model's understanding of global information.

[0117] Layer Normalization (LayerNorm for short): The normalization layer is used to standardize input features to improve the stability and convergence speed of training.

[0118] MLP (Multi-layer Perceptron): This module consists of two fully connected layers and usually includes an activation function (such as ReLU) to further process the features and introduce non-linearity.

[0119] The decoder of the SAD-UNetR image segmentation model is another important component of the model, which is responsible for converting the feature representation extracted by the encoder into the final segmentation result. In the decoder part, a U-shaped network is used to pass the feature representation extracted by the encoder to the decoder through a jump connection. At the decoding layer of each decoder, an output feature representation of the corresponding size is obtained;

[0120] The decoder adopts a CNN-based decoder, designs a U-shaped network structure, and is symmetrical with the encoder. The decoder of this U-shaped network includes multiple stages, each of which has a decoder module to gradually restore the resolution of the feature representation. At each stage, the decoder is connected to the corresponding stage of the encoder through a jump connection to utilize the feature representation extracted by the encoder. This connection mechanism helps the decoder to better understand the input data and improve the accuracy of segmentation. At the end of each stage, the output feature representation of the decoder is reshaped to the resolution corresponding to the encoder so that it can be fused and processed with the feature representation of the encoder. Final segmentation output: The final output of the decoder is the segmentation result of the spinal CT image, including vertebrae, intervertebral discs, and nerves.

[0121] In this embodiment, the Swin UNetR network model is improved to obtain the SAD-UNetR image segmentation model of this embodiment, which successfully achieves accurate segmentation of the spine image.

[0122] In the SAD-UNetR image segmentation model, the Soft Dice loss function is used for training to evaluate the similarity between the model's prediction results and the true labels, and the model parameters are optimized by minimizing this loss function.

[0123] The Soft Dice loss function is a loss function used for pixel-level segmentation tasks. For each pixel position, the Dice coefficient between the predicted result and the true label is calculated separately to evaluate the similarity between the two. It can effectively evaluate the accuracy of the segmentation result. It can ensure that the model can accurately evaluate the match between the predicted result and the true label at each pixel position.

[0124] During the training process, the goal of the model is to minimize the Soft Dice loss function, that is, to maximize the Dice coefficient between the predicted results and the true labels by adjusting the model parameters, thereby improving the accuracy of segmentation.

[0125] The following is a description of other structures in the SAD-UNetR image segmentation model;

[0126] Patch Partition module: This module is placed between the encoder's AWM and the encoder's second layer to divide the grayscale-adjusted input data into non-overlapping 3D patches for subsequent processing;

[0127] The upsampling connection of the encoder in the SAD-UNetR image segmentation model is connected through the Residual Block, which is used to input the data of the previous layer of the decoder to the next layer of the decoder through the residual block;

[0128] Residual Block: This module contains two 3×3×3 convolutional layers and uses instance normalization for feature processing to help the model learn deeper features.

[0129] Deconv (deconvolution): This module is used to increase the resolution of the feature map through deconvolution operations.

[0130] Segmentation Output: The final segmentation output module uses a 1×1×1 convolutional layer and a sigmoid activation function to generate segmentation results corresponding to different tumor regions.

[0131] In step 4, the 3D reconstruction method can use the graphics and image processing library VTK (Visualization Toolkit) to perform 3D reconstruction of the CT image, and transform the spine and related mask data from the 3D voxel space into a polygonal mesh with a geometric shape and topological structure, that is, a 3D model of the spine.

[0132] Specifically, the mask data obtained by segmentation in step 2 is converted into three-dimensional voxel data, and the voxel data is converted into polygonal meshes using the Marching Cubes algorithm in VTK to obtain the geometric shape of the spine and its surrounding structures.

[0133] According to a further technical solution, in step 4, the obtained three-dimensional model is surface rendered and projected to obtain a multi-view display result; after the three-dimensional model is obtained in step 3, a visual display is performed.

[0134] An alternative technical solution is to use an image processing and rendering system based on OpenCV and OpenGL to replace VTK for 3D reconstruction during 3D modeling.

[0135] As shown in Figure 5(a), it is the entire spine of a human body, wherein the enlarged part is the lumbar vertebra position of the reconstruction task. As shown in Figure 5(b), the vertebral structure obtained by the three-dimensional modeling method after image segmentation of this embodiment is similar to the physical vertebral structure marked in Figure 5(a). It can be seen that accurate three-dimensional reconstruction can be achieved through the method of this embodiment.

[0136] Specifically, OpenCV is used for image processing and segmentation, while OpenGL is used for real-time rendering of three-dimensional structures.

[0137] Another alternative technical solution is to use MATLAB's image processing and visualization system to replace VTK for 3D reconstruction during the 3D modeling process;

[0138] Specifically, MATLAB and its image processing toolbox are used to perform analysis, segmentation, and three-dimensional visualization of CT images.

[0139] Through the above methods, the three-dimensional information of the spine and its surrounding structures can be efficiently and accurately extracted from the spinal CT images, and the vertebral structure can be accurately reconstructed in three dimensions. Doctors can observe the model from different angles and distances through operations such as rotation, scaling, and translation to evaluate the patient's condition. In addition, by comparing the patient's model with the model of a normal person, doctors can more accurately identify the location of the lesion and improve the accuracy of diagnosis. This allows doctors to understand the patient's spinal condition more clearly and comprehensively, thereby making an accurate clinical diagnosis, which is of great reference value for subsequent diagnosis and treatment decisions.

[0140] In order to illustrate the effect of the three-dimensional reconstruction method of this embodiment, an experiment was conducted, and the specific description is as follows.

[0141] In the experiment of this embodiment, the SAD-UNetR method was compared with five deep learning-based methods (i.e., FCN, UNet, 3D U-Net, UNetR, Swin unet, and nnunet).

[0142] The comparison of existing deep learning methods is explained as follows:

[0143] Fully Convolutional Network (FCN): FCN is a pixel-level classification of images, which solves the problem of semantic-level image segmentation. Unlike CNN, FCN can accept images of any size and restore them to the same size as the input image using a deconvolution layer. Therefore, predictions can be generated for each pixel while retaining the spatial information in the original input image, and finally pixel-by-pixel classification is performed on the upsampled feature map.

[0144] UNetr: This architecture uses pure Transformers as encoders to learn sequential representations of the input volume and effectively capture global multi-scale information. It also follows the successful "U-shaped" network design of the encoder and decoder. The Transformer encoder is directly connected to the decoder through jump connections at different resolutions to calculate the final semantic segmentation output.

[0145] 3D U-Net: The network structure of 3D UNet is based on the UNet architecture and is extended to suit the semantic segmentation task of processing 3D volume data. All 2D convolution operations in the original U-net are replaced with 3D convolution operations to process 3D volume data. The architecture contains an encoder path and a decoder path. The former is used to capture the context in the image and the latter is used to achieve precise positioning using transposed convolution. Skip connections are used as a feature map fusion method. This method effectively solves the problem of medical image segmentation.

[0146] Swin UNetR: Swin UNetR combines the advantages of Swin Transformer and UNetR architectures. Swin Transformer is a Transformer-based model that introduces the concept of shifted windows to limit the scope of self-attention, thereby reducing computational complexity and improving the efficiency of the model. In Swin UNetR, Swin Transformer acts as an encoder to capture the global contextual information of the image with its powerful representation learning ability. SwinUNetR processes image patches using Transformer instead of relying on convolutional layers, which enables more efficient processing of long-distance dependencies in images. In addition, the architecture combines low-level features with high-level features through skip connections, which helps improve the recovery of details, especially in complex medical image segmentation tasks.

[0147] nnUNet: nnUNet is an adaptive neural network architecture that automatically configures the network structure and training strategy to adapt to a specific dataset. The nnUNet framework is based on U-Net, but achieves excellent performance through thorough analysis of each new dataset and automated preprocessing, network configuration, and post-processing steps. The key advantages of nnUNet are its adaptability and automation. It not only automatically selects the appropriate network depth, width, and training strategy, but also adjusts data augmentation and regularization techniques according to the needs of specific tasks. This approach greatly simplifies the model tuning process, enabling it to achieve optimal performance on a variety of medical image datasets.

[0148] In order to make an objective and accurate comparison, the same experimental settings, including preprocessed images, optimizers, loss functions, learning rates, and epochs, were used for the improved SAD-UNetR algorithm and four deep learning-based methods (i.e., FCN, UNet, 3D U-Net, UNetR, Swin unet, and nnunet). The experimental conclusions were drawn by comparing the experimental results.

[0149] The experimental results are shown in Table 1, and comparison is made from three indicators. In the table, Dice Coefficient (DC) is also called Dice similarity coefficient, which is used to evaluate the overlap between the predicted result and the actual result; Intersection over Union (IoU) is the ratio of the intersection over union of the predicted result and the actual result, and Sensitivity (SEN) is the recall rate. Compared with the prior art, the SAD-UNetR image segmentation model provided in this embodiment is optimal in Dice similarity coefficient and SEN, and the IoU index also reaches above 0.85, which has a good segmentation effect.

[0150] Table 1 Comparison of experimental results

[0151]

[0152] During the training process, the SAD-UNetR image segmentation model of this embodiment is implemented using PyTorch3 and MONAI4, and is trained on a GPU with an NVIDIA GeForce RT×3090, with a learning rate set to 0.0008. All input images are normalized to zero mean and unit standard deviation based on non-zero voxels. During training, 64*64*64 patches are randomly cropped from the 3D image volume. A random axis mirror flip is applied to all 3 axes with a probability of 0.5. In addition, the feature map size is 64, adaptive moment estimation (Adam) is used, epoch is set to 100, and momentum is set to 0.9.

[0153] like Figure 8 As shown, a comparison chart of the image segmentation prediction results using Swin UNet and SAD-UNetR of this embodiment is shown. It can be seen from the figure that by visually comparing the segmentation results of the Swin UnetR network and the SAD-UNetR network with the true labels, it can be clearly seen that the prediction results of the SAD-UNetR network are more superior in visual effect and are more consistent with the true labels.

[0154] Example 2

[0155] Based on Example 1, this embodiment provides a bone detection method based on the vertebral three-dimensional reconstruction method based on image segmentation described in Example 1, which can quickly and accurately detect and identify the patient's bone condition after image segmentation of lumbar CT, including the following steps:

[0156] Step 1, obtaining the spinal CT plain scan image data to be processed;

[0157] Step 2: Input the image data into the improved image segmentation model based on the Swin-UNetR network to obtain a segmented image of the spinal CT image; the segmentation result includes the segmented target areas of the vertebral body, intervertebral disc and nerve;

[0158] The improved image segmentation model adds an AWM grayscale adaptive network before the encoder of the Swin-UNetR network to adaptively adjust the grayscale of the input image; and sets a DA-Block module on the jump connection after the encoder to extract the position features and channel features of the image;

[0159] Step 3: Propose patch locations for the segmented image and select instances from the segmented image to form bags as instance-level feature bags.

[0160] Step 4: Transform the instance-level features in the instance-level feature bag through the patch network;

[0161] Step 5: Combine the transformed instances through attention multi-instance learning and pooling;

[0162] Step 6: Classify based on the combined bag-level features to obtain the degree of osteoporosis and the corresponding bone density value.

[0163] This embodiment provides an efficient and accurate bone detection method through innovative image segmentation, advanced multi-instance learning and attention mechanism, which effectively improves the performance of osteoporosis classification. It can effectively capture local and global structural features to improve the performance of osteoporosis classification and distinguish different states of normal bone mass, osteopenia and osteoporosis.

[0164] The image segmentation method of the above steps 1 to 2 is the same as the implementation method of steps 1 to 2 in Example 1. The improved image segmentation model used, namely the SAD-UNetR image segmentation model, is the same as the SAD-UNetR image segmentation model in Example 1, and will not be repeated in this embodiment.

[0165] In step 3, instance-level feature bags are obtained based on patch position proposals, including the following steps:

[0166] Step 31, evenly dividing the segmented image of the CT image into a plurality of cubic patches of a set size;

[0167] These patches are local areas, each of which contains a certain number of voxels to achieve three-dimensional space covering the entire CT image;

[0168] Step 32, using a t-test method to evaluate the significant difference, and determining the location of the patch with statistical significance for osteoporosis in the segmented image;

[0169] The t-test method was used to evaluate the significant differences between the experimental group (osteoporosis patients) and the control group (healthy individuals); the t-test method, specifically: statistical analysis was performed on each patch, and the differences between the experimental group and the control group were calculated to identify the patch locations that were statistically significant for osteoporosis identification.

[0170] Step 33, calculating the average value of voxel features: obtaining the voxel features in each patch, and finding the average value to represent the overall features of each patch;

[0171] Specifically, the voxel features may include density and morphological features of the voxels;

[0172] Step 34: Calculate the average voxel feature of each patch and compare between groups to generate a p-value map covering the entire CT image; select the patch with the smallest p-value to form an instance-level feature bag as input for subsequent feature extraction and classification.

[0173] Among them, each patch position corresponds to a p-value, which indicates the significant difference between the experimental group and the control group. The smaller the p-value, the more important the patch is in distinguishing osteoporosis from normal state. According to the generated p-value graph, several patch positions with the smallest p-value are selected to form the input bag. These patch positions are the areas with the strongest correlation with osteoporosis and can effectively capture the characteristics of osteoporosis.

[0174] Furthermore, during the training process, osteoporosis classification was considered as a multiple instance learning problem and the corresponding model was constructed. The training data was organized into multiple bags, each bag containing multiple unlabeled instances, and only the overall label of the bag was available. Patches from patients with normal bone mass, osteopenia, and osteoporosis were considered as different bags.

[0175] The above instance-level feature bag extraction method ensures that the model can focus on the most representative local areas, laying a solid foundation for subsequent feature extraction and classification.

[0176] In some embodiments, a bone density prediction regression model DA-MIDL++ is constructed to implement steps 4 to 6 of the above-mentioned bone quality detection method, such as Figure 7 As shown in the figure, the bone density prediction regression model DA-MIDL++ includes: patch networks (Patch-Nets), attention multi-instance learning pooling (Attention MIL Pooling) module and attention-aware global classifier and prediction branch.

[0177] Patch-Nets: used to transform instance-level features in instance-level feature bags and extract feature representations with spatial attention;

[0178] Attention multi-instance learning (MIL) pooling module: configured to use attention multi-instance learning and pooling to combine the transformed instances;

[0179] Attention-aware global classifier for classification based on combined bag-level features to obtain the degree of osteoporosis

[0180] The prediction branch is used to predict the corresponding bone density value based on the combined bag-level features;

[0181] Patch-Net extracts feature representations with spatial attention from input 3D image patches. Patch-Net is configured to perform the following processes: first, learn spatial attention-aware patch-level feature representations, and second, output an influence score to indicate the influence of the patch on the bag label.

[0182] Figure 7 The symbols in are explained as follows:

[0183] original patch: the original patch, the orange square represents the original input patch;

[0184] Patch-level features: Patch-level features. The blue squares represent patch-level features extracted from the original patches.

[0185] Global features: global features are obtained by summarizing multiple patch-level features and are used to describe the overall image characteristics;

[0186] Attention-aware global features: Attention-aware global features, which are adjusted by the attention mechanism and emphasize more important features.

[0187] Affect score: The gray square represents the impact score, which is used to measure the importance of each patch to the overall feature;

[0188] Attention weights: gray cubes represent the calculated attention weights used to weight features.

[0189] Dimensional expansion: light blue blocks represent feature dimension expansion operations to accommodate subsequent calculations.

[0190] Conv3d: 3D convolution, the yellow rectangle represents a three-dimensional convolution operation, which is used to extract features;

[0191] Max polling: The blue cube represents the maximum pooling operation, which is used for dimensionality reduction and feature selection.

[0192] Fully Connected: Fully connected layer. The green rectangle represents the fully connected layer, which maps the features to the classification space.

[0193] Softmax: softmax activation function. The blue rectangle represents the softmax activation function, which is used for probability calculation in multi-classification tasks.

[0194] Sigmoid: sigmoid activation function. The gray rectangle represents the sigmoid activation function, which is used for probability calculation in binary classification tasks.

[0195] Global max pooling: The light green rectangle represents the global maximum pooling operation, which is used to extract the global maximum feature;

[0196] Global average pooling: The light blue rectangle represents the global average pooling operation, which is used to extract the global average feature.

[0197] Channel max pooling: Channel max pooling. The blue rectangle represents the channel max pooling operation, which is used to extract the maximum feature of each channel.

[0198] Channel average pooling: The light gray rectangle represents the channel average pooling operation, which is used to extract the average features of each channel;

[0199] Concatenate: A plus sign in a circle indicates a concatenation operation, which concatenates multiple features together.

[0200] Element-wise multiplication: element-wise multiplication. A multiplication sign in a circle indicates an element-wise multiplication operation, where the elements at corresponding positions are multiplied.

[0201] Element-wise addition: element-wise addition. A plus sign in a circle indicates an element-wise addition operation, and the elements at corresponding positions are added.

[0202] Tensor multiplication: A cross in a circle represents a tensor multiplication operation, which performs multiplication operations on tensors of multiple dimensions.

[0203] A feasible technical solution is Figure 7As shown, Patch-Net includes multiple 3D convolutional layers (Conv3d), a maximum pooling branch, and a spatial attention module;

[0204] Among them, multiple 3D convolutional layers are Conv1, Conv2, Conv3 and Conv4; the maximum pooling branch includes a global average pooling layer, a fully connected layer and a sigmoid activation layer, which is used to output the impact score, that is, the ability of the patch to affect the bag label; the spatial attention module includes two pooling layers of channel maximum pooling and channel average pooling, as well as convolutional layers and activation layers;

[0205] The max pooling branch extracts more abstract features and reduces the size of the feature map. The spatial attention block generates a spatial attention map by performing channel max pooling and average pooling on the feature map, emphasizing the importance of different parts in the patch. Finally, the feature representation output by Patch-Net is combined with the spatial attention map to generate a patch-level feature representation with spatial attention, thereby enhancing the ability to capture structural changes related to osteoporosis.

[0206] In some embodiments, the attention multi-instance learning pooling (Attention MIL Pooling) module includes: a channel average pooling layer, a global average pooling layer and a global maximum pooling layer are connected to the channel average pooling layer, and the global average pooling layer and the global maximum pooling layer are respectively connected to multiple convolutional layers to generate two patch attention maps, and then the attention weights are obtained through activation operations for combination;

[0207] A technical solution that can be implemented is the Attention Multi-Instance Learning Pooling (MIL Pooling) module, which is used to generate an attention map for each patch to identify the impact of each patch in the overall classification. The specific process is as follows:

[0208] Step 41, each patch-level feature output by Patch-Net is processed in parallel by adopting global average pooling and global maximum pooling to generate two different feature descriptors;

[0209] Step 42: The two descriptors are further learned through a convolutional layer to generate two patch attention maps;

[0210] Step 43, combining the influence score output by Patch-Net with the two obtained attention maps to form a comprehensive patch attention map, thereby combining the instances transformed by Patch-Net;

[0211] The attention pooling method of the above-mentioned Attention Multi-Instance Learning Pooling (MIL Pooling) module not only considers the characteristics of each patch, but also assigns different weights to each patch, thereby effectively emphasizing the patch features that are crucial to the classification results and improving the performance of the model in osteoporosis classification and bone density value prediction tasks.

[0212] The attention-aware global classification and prediction branch is designed to process the bag-level feature representations from the attention multi-instance learning pool and make the final classification decision. The classifier further extracts features through a two-layer convolutional network, which uses the high-level feature extraction capabilities of the convolutional layers to capture the correlation between patches. Specifically, the first two convolutional layers of this module have 128 and 64 filters respectively, and the feature maps are subsequently downsampled to a fixed size through adaptive 3D average pooling. Finally, after flattening, the features are input into two fully connected layers to generate two scores representing the probabilities of negative and positive classes. In this way, the attention-aware global classifier and prediction branch can comprehensively consider the different weights of patch-level features, thereby improving the accuracy of osteoporosis classification and bone density value prediction.

[0213] This embodiment provides an improved dual-attention multi-instance deep learning network (DA-MIDL++) for osteoporosis classification and bone density prediction of lumbar CT images. Based on the segmentation of lumbar CT images, the model can quickly and accurately screen osteoporosis and provide accurate navigation support for subsequent treatment plans and surgical planning. The main innovation of the DA-MIDL++ model is to combine multi-instance learning and attention mechanisms to achieve accurate classification of osteoporosis and bone density prediction through several key modules in the network. This method significantly reduces the radiation risk, economic consumption and time cost brought by specialized bone density measurement. In addition, the model also expands the scope of osteoporosis screening, can be more widely used in clinical practice, and has a very broad application prospect.

[0214] Further technical solutions include establishing a human-computer interaction platform to realize human-computer interaction in the three-dimensional reconstruction process and the bone detection process; displaying segmentation results, reconstruction display and bone density prediction results to users through a human-computer interaction interface;

[0215] A feasible implementation method, a human-computer interaction platform, includes:

[0216] Presentation layer, where users (doctors) interact with the system through the human-computer interface;

[0217] Specific operations include: importing images, starting segmentation, viewing segmentation results, reconstructing displays, and performing bone density prediction;

[0218] The business logic layer is used to implement the bone detection method for vertebral three-dimensional reconstruction based on image segmentation described in Example 1, and output the intermediate processing results during the detection process; the specific operations and corresponding implementation functions are as follows:

[0219] Medical image reading and storage: read and store medical image data;

[0220] 2D Segmentation and Display: Perform 2D segmentation on the image and display the results.

[0221] 3D segmentation: Perform 3D segmentation on the image.

[0222] Three-dimensional reconstruction of vertebral structure: Three-dimensional reconstruction of vertebral structure;

[0223] Osteoporosis classification and bone density prediction: Classify osteoporosis and predict bone density.

[0224] The bottom layer is used to carry the algorithm library, the underlying framework, and the trained model;

[0225] Algorithm library: provides support for various algorithms;

[0226] Bottom framework: provides the basic architecture for system operation;

[0227] Trained segmentation model: pre-trained image segmentation model;

[0228] PyQt5 library: a development library for graphical user interfaces;

[0229] VTK library: a library for 3D computing and visualization;

[0230] Osteoporosis classification and BMD prediction model: a model used to achieve osteoporosis classification and bone density prediction.

[0231] The above-mentioned human-computer interaction platform can open the corresponding code on the compilation platform PyCharm or Visual Studio Code, and enter the UI interface after running the code on the compilation platform; in the system initialization interface, click the "Import Data" module to import the CT image, and select the CT image to be processed in the folder where the CT image is stored for import. After importing the spinal CT image, click Reconstruct Display, and the axial plane, coronal plane, sagittal plane, and three-dimensional reconstruction part of the original spinal CT image can be clearly displayed in the interface.

[0232] After clicking the "Start Segmentation" module, the input spinal CT image can be used to extract the region of interest through the segmentation model. At the same time, the segmented CT image is used to obtain the corresponding 3D reconstruction results through the 3D reconstruction module, which are displayed in the corresponding processing box. For each segmentation task, the corresponding segmentation accuracy and segmentation reconstruction time will be displayed after completion. Drag the scroll bar in the interface icon to view different slices of the spinal CT image. Rotate in the 3D reconstruction area to switch different perspectives so as to observe the 3D structure of the vertebra in all directions; after clicking the Start Detection module, the segmented vertebra is tested for bone quality through the DA-MIDL++ model.

[0233] Optionally, the human-computer interaction platform can be built on a cloud platform, with an image processing backend built through cloud services (such as AWS and Azure), and segmentation and reconstruction tasks can be achieved by setting up deep learning models. Web or mobile applications can be developed for doctors to use, and doctors can access and analyze CT images anytime and anywhere through their devices. The platform can provide a variety of processing options, allowing doctors to choose different algorithms and models for analysis to obtain the best results.

[0234] A further technical solution is to visualize the processing results in medical images based on virtual reality (VR) technology; specifically:

[0235] (1) Use game engines such as Unity or Unreal Engine in combination with the VTK library for 3D reconstruction and visualization;

[0236] (2) designing VR interfaces and interactive elements to enable physicians to manipulate CT images and vertebral structures in a virtual environment;

[0237] (3) Use the immersive experience provided by VR headsets to help doctors better understand anatomical structures and lesions.

[0238] Example 3

[0239] Based on Example 1, this embodiment provides a vertebral body three-dimensional reconstruction system based on image segmentation, including:

[0240] A data acquisition module is configured to acquire spinal CT plain scan image data to be processed;

[0241] Image segmentation module: configured to input image data into an improved image segmentation model based on the Swin-UNetR network to obtain a segmented image of the spinal CT image;

[0242] The improved image segmentation model adds an AWM grayscale adaptive network before the encoder of the Swin-UNetR network to adaptively adjust the grayscale of the input image; and sets a DA-Block module on the jump connection after the encoder to extract the position features and channel features of the image;

[0243] Three-dimensional modeling module: It is configured to process the image data and extract information using a three-dimensional reconstruction method for the segmented image, and to construct a three-dimensional model of the vertebral structure.

[0244] It should be noted here that the various modules in this embodiment correspond one-to-one to the various steps in Example 1, and the specific implementation process is the same, which will not be repeated here.

[0245] Example 4

[0246] Based on Example 1, this embodiment provides a bone detection system based on image segmentation, including:

[0247] A data acquisition module is configured to acquire spinal CT plain scan image data to be processed;

[0248] Image segmentation module: configured to input image data into an improved image segmentation model based on the Swin-UNetR network to obtain a segmented image of the spinal CT image;

[0249] The improved image segmentation model adds an AWM grayscale adaptive network before the encoder of the Swin-UNetR network to adaptively adjust the grayscale of the input image; and sets a DA-Block module on the jump connection after the encoder to extract the position features and channel features of the image;

[0250] Instance-level feature generation module: configured to propose patch locations for the segmented image and select instances from the segmented image to form bags as instance-level feature bags;

[0251] Transformation module: configured to transform instance-level features in the instance-level feature bag through a patch network;

[0252] Instance combination module: configured to combine transformed instances through attention multi-instance learning and pooling;

[0253] Prediction output module: It is configured to perform classification based on the combined bag-level features to obtain the degree of osteoporosis and the corresponding bone density value.

[0254] It should be noted here that the various modules in this embodiment correspond one-to-one to the various steps in Example 2, and the specific implementation process is the same, which will not be repeated here.

[0255] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

[0256] Although the above describes the specific implementation methods of the present disclosure in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present disclosure. Technical personnel in the relevant field should understand that on the basis of the technical solution of the present disclosure, various modifications or variations that can be made by those skilled in the art without creative work are still within the scope of protection of the present disclosure.

Claims

1. Vertebral 3D reconstruction method based on image segmentation, It is characterized in that The steps include: Acquire spinal CT plain scan image data to be processed; The image data is input into the improved image segmentation model based on the Swin-UNetR network to obtain the segmented image of the spinal CT image; The improved image segmentation model adds an AWM grayscale adaptive network before the encoder of the Swin-UNetR network to adaptively adjust the grayscale of the input image; a DA-Block module is set on the jump connection after the encoder to extract the position features and channel features of the image; For the segmented images, the three-dimensional reconstruction method is used to process the image data and extract information to construct a three-dimensional model of the vertebral structure.

2. The method for three-dimensional reconstruction of vertebral bodies based on image segmentation according to claim 1, Features: The network structure of the DA-Block module includes: Input layer: used to input the feature map output by the Swin-UNetR network encoder; Feature extraction upper branch: including the third convolutional layer, the position attention module and the fourth convolutional layer connected in sequence; Feature extraction lower branch: including the fifth convolutional layer, channel attention module and sixth convolutional layer connected in sequence; Feature fusion module: The feature map output by the upper branch of feature extraction and the feature map output by the lower branch of feature extraction are added and fused element by element to obtain a fused feature map; Output layer: The fused feature map is subjected to convolution operation to output the final feature map; All convolutional layers in the DA-Block module adopt depthwise separable convolution.

3. The method for three-dimensional reconstruction of vertebral bodies based on image segmentation according to claim 1, Features: The AWM grayscale adaptive network includes an input layer, a first convolutional layer, and two parallel processing branches. Each branch includes a second convolutional layer, a global average pooling layer, a sigmoid activation layer, and an addition fusion module connected in sequence. The outputs of the two processing branches are connected to a fully connected layer and a threshold processing layer. Alternatively, the image grayscale is adaptively adjusted based on the AWM grayscale adaptive network, including the following steps: The input image x is sent to the first convolutional layer, and the convolution operation is performed to generate the feature map f; After two parallel second convolutional layers, these two sets of convolutional layers further process the feature map f to generate features of different scales; The outputs of the two second convolutional layers are processed in two parallel branches respectively. After global average pooling and sigmoid activation, the features of the two branches are added together, and the weights Ww and Wc are scaled based on the two parameters α and β to obtain the grayscale adjustment feature map based on the weights. The grayscale adjusted feature map is fully connected with the original input image to obtain the feature map after adaptive grayscale adjustment, and the output image is generated through threshold processing.

4. The method for three-dimensional reconstruction of vertebral bodies based on image segmentation according to claim 1, Features: The encoder of the Swin-UNetR network adopts an improved Swin Transformer model. The improved Swin Transformer model replaces the normalization layer with a re-parameterized batch normalization layer and adopts a gradual replacement strategy. During the training process, the normalization layer in the Swin transformer is gradually replaced with a re-parameterized batch normalization layer according to the set parameters. Alternatively, the 3D reconstruction method uses a graphics and image processing library VTK to perform 3D reconstruction of CT images.

5. Bone detection method based on image segmentation, It is characterized in that The steps include: Acquire spinal CT plain scan image data to be processed; The image data is input into the improved image segmentation model based on the Swin-UNetR network to obtain the segmented image of the spinal CT image; The improved image segmentation model adds an AWM grayscale adaptive network before the encoder of the Swin-UNetR network to adaptively adjust the grayscale of the input image; a DA-Block module is set on the jump connection after the encoder to extract the position features and channel features of the image; For the segmented image, patch location proposal is performed, and instances are selected from the segmented image to form bags as instance-level feature bags; Transform instance-level features in instance-level feature bags through patch networks; Combine the transformed instances through attention multi-instance learning and pooling; Classification is performed based on the combined bag-level features to obtain the degree of osteoporosis and the corresponding bone density value.

6. The bone detection method based on image segmentation as claimed in claim 5, It is characterized in that The instance-level feature bag is obtained based on the patch position proposal, which includes the following steps: The segmented image of the CT image is evenly divided into a plurality of cubic patches of a set size; The t-test method was used to evaluate the significant differences and determine the locations of statistically significant osteoporosis patches in the segmented images. Get the voxel features in each patch and calculate the average value as the overall feature representing each patch; The mean of voxel features of each patch was calculated and compared between groups to generate a p-value map covering the entire CT image; the patch with the smallest p-value was selected to form an instance-level feature bag.

7. The bone detection method based on image segmentation as claimed in claim 5, It is characterized in that Construct a bone density prediction regression model, including: patch network, attention multi-instance learning pooling module, attention-aware global classifier and prediction branch; The patch network includes multiple 3D convolutional layers, a maximum pooling branch, and a spatial attention module; it is used to transform the instance-level features in the instance-level feature bag and extract feature representations with spatial attention; Attention multi-instance learning pooling module: configured to use attention multi-instance learning and pooling to combine the transformed instances; the attention multi-instance learning pooling module includes a channel average pooling layer, the channel average pooling layer is connected to a global average pooling layer and a global maximum pooling layer, the global average pooling layer and the global maximum pooling layer are respectively connected to multiple convolutional layers, two patch attention maps are generated, and then the attention weights are obtained through activation operations for combination; Attention-aware global classifier, used to classify based on the combined bag-level features to obtain the degree of osteoporosis; The prediction branch is used to predict the corresponding bone density value based on the combined bag-level features.

8. The bone detection method based on image segmentation as claimed in claim 5, It is characterized in that A human-computer interaction platform is established to realize human-computer interaction in the three-dimensional reconstruction process and the bone detection process; a visual display is realized through the human-computer interaction interface, including segmentation results, reconstruction display and bone density prediction results. 9.Vertebral 3D reconstruction system based on image segmentation, It is characterized in that include: A data acquisition module is configured to acquire spinal CT plain scan image data to be processed; Image segmentation module: configured to input image data into an improved image segmentation model based on the Swin-UNetR network to obtain a segmented image of the spinal CT image; The improved image segmentation model adds an AWM grayscale adaptive network before the encoder of the Swin-UNetR network to adaptively adjust the grayscale of the input image; and sets a DA-Block module on the jump connection after the encoder to extract the position features and channel features of the image; Three-dimensional modeling module: It is configured to process the image data and extract information using a three-dimensional reconstruction method for the segmented image, and to construct a three-dimensional model of the vertebral structure.

10. Bone detection system based on image segmentation, It is characterized in that include: A data acquisition module is configured to acquire spinal CT plain scan image data to be processed; Image segmentation module: configured to input image data into an improved image segmentation model based on the Swin-UNetR network to obtain a segmented image of the spinal CT image; The improved image segmentation model adds an AWM grayscale adaptive network before the encoder of the Swin-UNetR network to adaptively adjust the grayscale of the input image; and sets a DA-Block module on the jump connection after the encoder to extract the position features and channel features of the image; Instance-level feature generation module: configured to propose patch locations for the segmented image and select instances from the segmented image to form bags as instance-level feature bags; Transformation module: configured to transform instance-level features in the instance-level feature bag through a patch network; Instance combination module: configured to combine transformed instances through attention multi-instance learning and pooling; Prediction output module: It is configured to perform classification based on the combined bag-level features to obtain the degree of osteoporosis and the corresponding bone density value.

Images