X-ray image segmentation method and system fused with CT data

Abstract

The invention provides a CT data fused X-ray image segmentation method and system. The method comprises the steps of obtaining a to-be-segmented X-ray image and a CT image at a corresponding position; inputting the X-ray image and the CT image into a preset image segmentation model to enable the image segmentation model to extract a two-dimensional feature from the X-ray image and extract a three-dimensional feature from the CT image, performing feature fusion on the two-dimensional feature and the three-dimensional feature, and generating a corresponding segmentation image according to the fused feature; wherein the segmentation model is constructed based on a neural network model and is obtained by training according to a plurality of historical CT images and a plurality of corresponding historical DRR images, the historical DRR images are obtained by performing DRR projection on the historical CT images, and the accuracy of automatic segmentation of each tissue structure in the X-ray image is improved.

Description

Technical Field

[0001] The present application relates to the technical field of machine learning and image processing, and in particular to a method and system for segmenting an X-ray image fused with CT data. Background Art

[0002] Ankle replacement surgery is a complex orthopedic procedure that aims to restore joint function, relieve pain, and improve the quality of life of patients by replacing damaged or degenerated ankle joints. With the aging population and the increase in sports injuries, the incidence of ankle diseases has gradually increased, which has led to an increasing demand for ankle replacement surgery. However, the success of the operation depends on accurate preoperative diagnosis and planning, which are highly dependent on the accurate analysis of medical images. As one of the important tools for preoperative evaluation, ankle X-ray images have the advantages of non-invasiveness, low cost, and easy access, so they are widely used in clinical practice. In the preoperative planning of ankle replacement surgery, doctors need to accurately identify the anatomical structures of the ankle joint, such as the tibia, talus, and fibula, to assess the health of the joint and formulate a surgical plan. However, manual analysis of X-ray images is not only time-consuming and laborious, but also limited by the subjective experience of doctors, which may lead to diagnostic differences between different doctors. Therefore, automated ankle X-ray image segmentation technology has emerged. It automatically extracts the various anatomical structures of the joint through computer vision algorithms, which can improve the consistency and accuracy of diagnosis, thereby optimizing surgical planning. The successful application of X-ray image segmentation can significantly improve the accuracy of preoperative planning. Automatic segmentation can accurately measure the joint space and help doctors evaluate the degree of joint degeneration; the segmented images can be used to simulate surgery to help doctors choose the most appropriate joint replacement components; by comparing preoperative and postoperative images, automatic segmentation can help evaluate the effectiveness of surgery and detect potential complications. In addition to its application in preoperative planning and postoperative evaluation, segmented ankle X-ray images can also be used for 3D image reconstruction. By segmenting and reconstructing X-ray images from multiple angles, doctors can generate a three-dimensional model of the patient's ankle joint. This model can more intuitively display the anatomical structure and lesion site of the joint, providing an important reference for precise surgical positioning and the design of replacement components.

[0003] In order to solve the problem of medical image segmentation, researchers have developed a variety of image segmentation techniques, mainly including the following categories: Methods based on traditional image processing: These methods use classic techniques such as edge detection, region growing and threshold segmentation to extract boundaries in medical images. Methods based on machine learning: With the development of machine learning technology, supervised learning methods such as support vector machine (SVM) and random forest have been applied to medical image segmentation. These methods learn the features of the image by training the model, so as to achieve better results in segmentation. Methods based on deep learning: In recent years, deep learning models such as convolutional neural networks (CNN) have performed well in the field of medical image segmentation, especially network structures such as U-Net and ResNet have been widely used in medical image segmentation tasks. Deep learning models can automatically extract multi-level image features, thereby achieving accurate segmentation in complex backgrounds. Methods based on attention mechanisms: Recently developed models such as Swin Transformer and Vision Transformer capture the contextual information of images globally through self-attention mechanisms, showing stronger capabilities in medical image segmentation. The unique challenges of ankle X-ray image segmentation are: first, the contrast is low, and the density difference between bones and soft tissues is small, resulting in unclear bone boundaries in the image; second, ankle X-ray images are two-dimensional projection images, and bones, soft tissues and other anatomical structures may overlap during the imaging process, resulting in blurred boundaries of anatomical structures. Although common segmentation methods perform well in images with high contrast and clear boundaries, they often fail to achieve the desired effect in ankle X-ray images. Summary of the invention

[0004] In response to the above technical problems, the present application provides an X-ray image segmentation method and system that integrates CT data to improve the accuracy of automatic segmentation of various tissue structures in X-ray images.

[0005] In a first aspect, an embodiment of the present application provides an X-ray image segmentation method for fused CT data, comprising:

[0006] Acquire the X-ray image to be segmented and the CT image of the corresponding position;

[0007] Inputting the X-ray image and the CT image into a preset image segmentation model, so that the image segmentation model extracts two-dimensional features from the X-ray image and three-dimensional features from the CT image, and after fusing the two-dimensional features with the three-dimensional features, generates a corresponding segmentation map according to the fused features;

[0008] The segmentation model is constructed based on a neural network model and is obtained by training according to a number of historical CT images and a number of corresponding historical DRR images, and the historical DRR images are obtained by performing DRR projection on the historical CT images.

[0009] The embodiment of the present application provides an X-ray image segmentation method that integrates CT data, and realizes the automatic segmentation of X-ray images through a preset image segmentation model. In the image segmentation process, the embodiment of the present application extracts the two-dimensional features of the X-ray image and the three-dimensional features of the CT image respectively and performs feature fusion, so that the image segmentation model can obtain more reference information related to the X-ray image, and helps the image segmentation model to more accurately segment the X-ray image. In addition, during the training process of the image segmentation model, several historical CT images are processed by DRR projection to obtain several corresponding historical DDR images. Through this technical means, the effective information of high-dimensional data is fully utilized, and two-dimensional DDR image data is generated based on three-dimensional CT image data, so that the corresponding X-ray image data is not required during the model training process. While ensuring the training effect, the demand for training data is reduced and the utilization rate of CT image data is improved. Compared with the prior art, since the embodiment of the present application introduces high-dimensional CT image data in both the model training and image segmentation processes, it fully utilizes the local details and global spatial structure information of the image, improves the model's feature capture capability under low contrast and fuzzy boundary conditions, ensures high precision and robustness of segmentation or classification tasks, effectively solves the problem of blurred boundaries of tissue parts in X-ray images, and improves the accuracy of automatic segmentation of various tissue structures in X-ray images.

[0010] In a possible implementation, the X-ray image and the CT image are input into a preset image segmentation model so that the image segmentation model extracts two-dimensional features from the X-ray image and three-dimensional features from the CT image, and after fusing the two-dimensional features with the three-dimensional features, generates a corresponding segmentation map according to the fused features, including:

[0011] According to the historical DRR image, a histogram normalization operation is performed on the X-ray image to generate a corresponding processed X-ray image;

[0012] The processed X-ray image and the CT image are input into a preset image segmentation model so that the image segmentation model generates a corresponding segmentation map according to the processed X-ray image and the CT image.

[0013] Furthermore, the step of performing a histogram normalization operation on the X-ray image according to the historical DRR image to generate a corresponding processed X-ray image includes:

[0014] According to a preset pixel range, respectively calculating the histograms of the historical DRR image and the X-ray image to obtain corresponding DRR image histograms and X-ray image histograms;

[0015] Calculating the cumulative distribution functions of the DRR image histogram and the X-ray image histogram respectively to obtain corresponding DRR image cumulative distribution functions and X-ray image cumulative distribution functions;

[0016] Determining a pixel mapping function according to the DRR image cumulative distribution function and the X-ray image cumulative distribution function;

[0017] According to the pixel mapping function, the X-ray image is pixel mapped to generate the processed X-ray image.

[0018] In an embodiment of the present application, the grayscale histogram of the X-ray image to be input into the image segmentation model is normalized so that its histogram matches the historical DRR image during training, and then the processed X-ray image and CT image are input into the image segmentation model, thereby reducing the impact of the difference between the distribution of model training data and model inference data, and further improving the accuracy of the model's image segmentation.

[0019] Furthermore, the image segmentation model generates a corresponding segmentation map according to the processed X-ray image and the CT image, including:

[0020] Performing two-dimensional feature encoding on the processed X-ray image, extracting multi-scale semantic features, and generating a corresponding two-dimensional feature map;

[0021] Performing three-dimensional feature coding on the CT image, extracting three-dimensional spatial information, and obtaining three-dimensional coding features;

[0022] The three-dimensional coded features are subjected to three-dimensional feature separation, and three corresponding feature maps are generated after dimension order adjustment, dimension flipping, feature dimension reduction and convolution operations, including a coronal feature map, a sagittal feature map and an axial feature map;

[0023] Based on the cross attention mechanism, the two-dimensional feature map is cross-fused with the three feature maps respectively to obtain three corresponding fused feature maps;

[0024] After performing a splicing operation, a convolution operation, and a batch normalization operation on the three fused feature maps, a normalized fused feature map is obtained;

[0025] Decoding the normalized fused feature map, converting the features in the normalized fused feature map into a two-dimensional space, and obtaining a two-dimensional fused feature map;

[0026] Based on a preset number of segmentation categories, two-dimensional convolution and feature mapping are performed on the two-dimensional fusion feature map to generate the segmentation map.

[0027] The embodiment of the present application provides a method for generating a segmentation map based on processed X-ray images and CT images, by extracting three-dimensional features according to different slice directions, generating coronal feature maps, sagittal feature maps and axial feature maps, and performing multi-level fusion with two-dimensional features, effectively solving the boundary fuzziness problem caused by overlapping anatomical structures in ankle joint X-ray images, enhancing the model's all-round understanding of complex structures, and improving the accuracy of bone segmentation. In the feature fusion stage, the embodiment of the present application introduces a cross-attention mechanism, which achieves accurate integration of information by adaptively modeling the dependency between two-dimensional and three-dimensional features, especially when dealing with problems such as small differences in bone and soft tissue density and unclear boundaries in X-ray images, which can effectively improve the performance of the model.

[0028] In a possible implementation, the segmentation model is constructed based on a neural network model and trained according to a number of historical CT images and a number of corresponding DRR images, including:

[0029] Acquire a number of historical CT images and a number of corresponding DRR images at a preset position;

[0030] Constructing corresponding real segmentation maps according to each of the DRR images;

[0031] Combining each of the historical CT images, each of the DRR images, and each of the real segmentation maps to construct a training data set;

[0032] Constructing an initial image segmentation model based on a neural network model, wherein the initial image segmentation model includes a two-dimensional feature extraction module, a three-dimensional feature extraction module, a feature fusion module, a feature decoder and a segmentation head;

[0033] The training data set is input into the initial image segmentation model, and each model parameter in the initial image segmentation model is updated through a preset loss function based on supervised learning to obtain the segmentation model.

[0034] The embodiment of the present application limits the content of CT images and DRR images during the model training process, and only uses a number of historical CT images of preset positions and a number of corresponding DRR images to construct training data, so that the image segmentation model obtained through training can specialize in a certain human body position. For example, when the preset position is the ankle joint, the model can learn the knowledge of the unique bone morphology and soft tissue morphology of the ankle joint based on the X-ray image characteristics of the ankle joint. The image segmentation model obtained through training can be specifically used for image segmentation of the ankle joint position, further improving the segmentation accuracy of the model.

[0035] Furthermore, the loss function is constructed by combining the Dice loss function and the cross entropy loss function.

[0036] The embodiment of the present application adopts a loss function constructed by combining Dice loss and cross entropy loss during the training process, wherein Dice loss is used to measure the degree of overlap between the model prediction and the true label, and is mainly used in image segmentation tasks. The higher the Dice coefficient, the higher the match between the prediction result and the true segmentation result. Cross-entropy loss is mainly used for pixel-level classification tasks to ensure that the model can accurately predict the category to which it belongs at each pixel. These two loss functions are combined for model training, while taking into account the optimization of class imbalance and global accuracy, ensuring that the model performs well in both classification accuracy and segmentation area overlap.

[0037] In one possible implementation, performing DRR projection on the historical CT image to obtain the historical DRR image includes: simulating the process of performing X-ray projection on the historical CT image based on the Siddon algorithm and GPU acceleration technology, and then generating the corresponding historical DRR image according to the simulation result.

[0038] In a second aspect, accordingly, an embodiment of the present application provides an X-ray image segmentation system for fusion CT data, including an acquisition module, an image segmentation module, and a model building module;

[0039] Wherein, the acquisition module is used to acquire the X-ray image to be segmented and the CT image of the corresponding position;

[0040] The image segmentation module is used to input the X-ray image and the CT image into a preset image segmentation model, so that the image segmentation model extracts two-dimensional features from the X-ray image and three-dimensional features from the CT image, and after fusing the two-dimensional features and the three-dimensional features, generates a corresponding segmentation map according to the fused features;

[0041] The model building module is used to build the segmentation model based on a neural network model and train according to several historical CT images and corresponding several historical DRR images, and the historical DRR images are obtained by performing DRR projection on the historical CT images.

[0042] In a possible implementation, the image segmentation module inputs the X-ray image and the CT image into a preset image segmentation model, so that the image segmentation model extracts two-dimensional features from the X-ray image and three-dimensional features from the CT image, and after fusing the two-dimensional features and the three-dimensional features, generates a corresponding segmentation map according to the fused features, including:

[0043] According to the historical DRR image, a histogram normalization operation is performed on the X-ray image to generate a corresponding processed X-ray image;

[0044] The processed X-ray image and the CT image are input into a preset image segmentation model so that the image segmentation model generates a corresponding segmentation map according to the processed X-ray image and the CT image.

[0045] Furthermore, the image segmentation model generates a corresponding segmentation map according to the processed X-ray image and the CT image, including:

[0046] Performing two-dimensional feature encoding on the processed X-ray image, extracting multi-scale semantic features, and generating a corresponding two-dimensional feature map;

[0047] Performing three-dimensional feature coding on the CT image, extracting three-dimensional spatial information, and obtaining three-dimensional coding features;

[0048] The three-dimensional coded features are subjected to three-dimensional feature separation, and three corresponding feature maps are generated after dimension order adjustment, dimension flipping, feature dimension reduction and convolution operations, including a coronal feature map, a sagittal feature map and an axial feature map;

[0049] Based on the cross attention mechanism, the two-dimensional feature map is cross-fused with the three feature maps respectively to obtain three corresponding fused feature maps;

[0050] After performing a splicing operation, a convolution operation, and a batch normalization operation on the three fused feature maps, a normalized fused feature map is obtained;

[0051] Decoding the normalized fused feature map, converting the features in the normalized fused feature map into a two-dimensional space, and obtaining a two-dimensional fused feature map;

[0052] Based on a preset number of segmentation categories, two-dimensional convolution and feature mapping are performed on the two-dimensional fusion feature map to generate the segmentation map. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 : A flow chart of an X-ray image segmentation method for fused CT data provided in an embodiment of the present application.

[0054] Figure 2 : A schematic diagram of the results of a histogram normalization operation in an X-ray image segmentation method for fused CT data provided in an embodiment of the present application.

[0055] Figure 3 : A schematic diagram of the model architecture of an image segmentation model in an X-ray image segmentation method that fuses CT data provided in an embodiment of the present application.

[0056] Figure 4: A structural schematic diagram of a two-dimensional feature extraction module in an X-ray image segmentation method that integrates CT data provided in an embodiment of the present application.

[0057] Figure 5 : A structural schematic diagram of a three-dimensional feature extraction module in an X-ray image segmentation method that integrates CT data provided in an embodiment of the present application.

[0058] Figure 6 : A schematic diagram of the structure of a three-dimensional feature separator in an X-ray image segmentation method that integrates CT data provided in an embodiment of the present application.

[0059] Figure 7 : A structural schematic diagram of a feature fusion module in an X-ray image segmentation method for fusing CT data provided in an embodiment of the present application.

[0060] Figure 8 : A schematic diagram of an image segmentation result in an X-ray image segmentation method for fused CT data provided in an embodiment of the present application.

[0061] Fig. 9 : A flow chart of constructing an image segmentation model in an X-ray image segmentation method integrating CT data provided in an embodiment of the present application.

[0062] Fig.10 : A schematic diagram of the data structure of a training data set and an inference data set in an X-ray image segmentation method that integrates CT data provided in an embodiment of the present application.

[0063] Fig.11 : A structural diagram of an X-ray image segmentation system that integrates CT data provided in an embodiment of the present application. DETAILED DESCRIPTION

[0064] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0065] It should be noted that the step numbers in the text are only for the convenience of explanation of the specific embodiments and do not serve to limit the order in which the steps are executed. In the description of this application, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features.

[0066] Embodiment 1:

[0067] like Figure 1 As shown, the first embodiment provides an X-ray image segmentation method for fusion CT data, including steps S1-S2:

[0068] Step S1, obtaining an X-ray image to be segmented and a CT image of a corresponding position;

[0069] Step S1, inputting the X-ray image and the CT image into a preset image segmentation model, so that the image segmentation model extracts two-dimensional features from the X-ray image and three-dimensional features from the CT image, and after fusing the two-dimensional features with the three-dimensional features, generates a corresponding segmentation map according to the fused features;

[0070] The segmentation model is constructed based on a neural network model and is obtained by training according to a number of historical CT images and a number of corresponding historical DRR images, and the historical DRR images are obtained by performing DRR projection on the historical CT images.

[0071] The embodiment of the present application provides an X-ray image segmentation method that integrates CT data, and realizes the automatic segmentation of X-ray images through a preset image segmentation model. In the image segmentation process, the embodiment of the present application extracts the two-dimensional features of the X-ray image and the three-dimensional features of the CT image respectively and performs feature fusion, so that the image segmentation model can obtain more reference information related to the X-ray image, and helps the image segmentation model to more accurately segment the X-ray image. In addition, during the training process of the image segmentation model, several historical CT images are processed by DRR projection to obtain several corresponding historical DDR images. Through this technical means, the effective information of high-dimensional data is fully utilized, and two-dimensional DDR image data is generated based on three-dimensional CT image data, so that the corresponding X-ray image data is not required during the model training process. While ensuring the training effect, the demand for training data is reduced and the utilization rate of CT image data is improved. Compared with the prior art, since the embodiment of the present application introduces high-dimensional CT image data in both the model training and image segmentation processes, it fully utilizes the local details and global spatial structure information of the image, improves the model's feature capture capability under low contrast and fuzzy boundary conditions, ensures high precision and robustness of segmentation or classification tasks, effectively solves the problem of blurred boundaries of tissue parts in X-ray images, and improves the accuracy of automatic segmentation of various tissue structures in X-ray images.

[0072] In a possible implementation, in step S2, the X-ray image and the CT image are input into a preset image segmentation model so that the image segmentation model extracts two-dimensional features from the X-ray image and three-dimensional features from the CT image, and after fusing the two-dimensional features with the three-dimensional features, generates a corresponding segmentation map according to the fused features, including:

[0073] According to the historical DRR image, a histogram normalization operation is performed on the X-ray image to generate a corresponding processed X-ray image;

[0074] The processed X-ray image and the CT image are input into a preset image segmentation model so that the image segmentation model generates a corresponding segmentation map according to the processed X-ray image and the CT image.

[0075] Furthermore, the step of performing a histogram normalization operation on the X-ray image according to the historical DRR image to generate a corresponding processed X-ray image includes:

[0076] According to a preset pixel range, respectively calculating the histograms of the historical DRR image and the X-ray image to obtain corresponding DRR image histograms and X-ray image histograms;

[0077] Calculating the cumulative distribution functions of the DRR image histogram and the X-ray image histogram respectively to obtain corresponding DRR image cumulative distribution functions and X-ray image cumulative distribution functions;

[0078] Determining a pixel mapping function according to the DRR image cumulative distribution function and the X-ray image cumulative distribution function;

[0079] According to the pixel mapping function, the X-ray image is pixel mapped to generate the processed X-ray image.

[0080] In an embodiment of the present application, the grayscale histogram of the X-ray image to be input into the image segmentation model is normalized so that its histogram matches the historical DRR image during training, and then the processed X-ray image and CT image are input into the image segmentation model, thereby reducing the impact of the difference between the distribution of model training data and model inference data, and further improving the accuracy of the model's image segmentation.

[0081] In a preferred embodiment, the result of performing a histogram normalization operation on the ankle joint X-ray image is as follows: Figure 2 As shown, the specific process is as follows:

[0082] Calculate the histogram for the X-ray image I that will be input into the model inf and historical DRR images I during training DRR Calculate the histogram H respectivelyinf (x) and H DRR (x), where x is a preset pixel value range (e.g., 0 to 255).

[0083] Calculate the cumulative distribution function (CDF) and calculate the cumulative distribution function C for X-ray images and DRR images respectively inf (x) and C DRR (x), the formula is:

[0084]

[0085] Matching mapping, through the cumulative distribution function C inf (x) and C DRR (x), find the mapping relationship between each X-ray image pixel value x inf The corresponding DRR image pixel value x DRR ,Right now:

[0086] x i ′ nf =(C DRR ) -1 (C inf (x inf ));

[0087] Apply mapping: Apply the mapped pixel values ​​to the X-ray image I inf , we get the image I after histogram normalization processing ′ , whose histogram matches the historical DRR images during training.

[0088] Furthermore, the image segmentation model generates a corresponding segmentation map according to the processed X-ray image and the CT image, including:

[0089] Performing two-dimensional feature encoding on the processed X-ray image, extracting multi-scale semantic features, and generating a corresponding two-dimensional feature map;

[0090] Performing three-dimensional feature coding on the CT image, extracting three-dimensional spatial information, and obtaining three-dimensional coding features;

[0091] The three-dimensional coded features are subjected to three-dimensional feature separation, and three corresponding feature maps are generated after dimension order adjustment, dimension flipping, feature dimension reduction and convolution operations, including a coronal feature map, a sagittal feature map and an axial feature map;

[0092] Based on the cross attention mechanism, the two-dimensional feature map is cross-fused with the three feature maps respectively to obtain three corresponding fused feature maps;

[0093] After performing a splicing operation, a convolution operation, and a batch normalization operation on the three fused feature maps, a normalized fused feature map is obtained;

[0094] Decoding the normalized fused feature map, converting the features in the normalized fused feature map into a two-dimensional space, and obtaining a two-dimensional fused feature map;

[0095] Based on a preset number of segmentation categories, two-dimensional convolution and feature mapping are performed on the two-dimensional fusion feature map to generate the segmentation map.

[0096] The embodiment of the present application provides a method for generating a segmentation map based on processed X-ray images and CT images, by extracting three-dimensional features according to different slice directions, generating coronal feature maps, sagittal feature maps and axial feature maps, and performing multi-level fusion with two-dimensional features, effectively solving the boundary fuzziness problem caused by overlapping anatomical structures in ankle joint X-ray images, enhancing the model's all-round understanding of complex structures, and improving the accuracy of bone segmentation. In the feature fusion stage, the embodiment of the present application introduces a cross-attention mechanism, which achieves accurate integration of information by adaptively modeling the dependency between two-dimensional and three-dimensional features, especially when dealing with problems such as small differences in bone and soft tissue density and unclear boundaries in X-ray images, which can effectively improve the performance of the model.

[0097] In a preferred embodiment, the model architecture of the image segmentation model is as follows: Figure 3As shown, it mainly includes a two-dimensional feature extraction module, a three-dimensional feature extraction module, a feature fusion module, a decoder and a segmentation head. The three-dimensional feature encoder is based on the encoder part of 3D Unet, and is used to extract deep spatial information from the three-dimensional space through three-dimensional convolution operations. The three-dimensional feature separator is used to process different slice directions in the three-dimensional features, such as separating independent features from the coronal, axial and sagittal planes. This separation helps to further extract different features in each direction, which facilitates a more refined analysis of the volume data. The feature fusion module is used to fuse the two-dimensional features with the three-dimensional separated features, combining the advantages of both to generate a richer and more comprehensive feature expression. Two-dimensional features can capture local details and edge information, while three-dimensional features can capture the volume structure and depth information of the object. By fusing these two types of features, the model can simultaneously obtain fine information on the two-dimensional plane and contextual information in the three-dimensional space, thereby achieving more accurate predictions in complex image analysis tasks. The feature decoder is implemented based on the decoder of the ResUnet network, and its main function is to gradually restore and reconstruct the features extracted and fused by the encoder, and convert these high-dimensional features into the same spatial dimension as the input data. The segmentation head includes a two-dimensional convolution layer and a softmax layer, which is used to further refine the segmentation result based on the decoder output to obtain the final segmentation map.

[0098] Specifically, the structure of the two-dimensional feature extraction module is as follows: Figure 4 As shown, the two-dimensional feature extraction module is based on the encoder part of Swin-Unet. In this embodiment, the two-dimensional feature extraction process is as follows:

[0099] Input a 2D image of the ankle joint of size 3×224×224. In the first step, a convolution operation of size 4×4 and stride 4 is used to reduce the spatial resolution of the image and increase the number of channels at the same time. Subsequently, the feature map obtained by the convolution operation is flattened and the dimension is transposed, and then the position encoding is added to retain the spatial position information. Next, the image features enter the standard Swin-Unet encoder structure, first passing through two Swin Transformer Blocks. Swin TransformerBlock uses a self-attention mechanism to model the features locally and globally to capture the contextual information of the image. Then a Patch Merging operation is performed, which reduces the spatial dimension of the feature map by downsampling and increases the number of channels to achieve feature compression and information aggregation. Then enter the second encoder block, which still includes two Swin TransformerBlocks and a Patch Merging operation to further extract the multi-scale features of the image. In the last step of encoding, only two Swin Transformer Blocks are passed, and no Patch Merging operation is performed. In this process, the feature map S i (i=1,2,3) sizes are 96×56×56, 192×28×28, 384×14×14 respectively. These feature maps extract rich image information at different scales, providing strong support for subsequent decoding and image segmentation. For subsequent processing, the obtained features are convolved with a size of 1×1 and a step size of 1 to increase the number of channels. The processed feature maps The sizes are 256×56×56, 512×28×28, and 1024×14×14 respectively.

[0100] like Figure 5 As shown, the three-dimensional feature extraction module includes a three-dimensional feature encoder and a three-dimensional feature separator. In this embodiment, the process of encoding features by the three-dimensional feature encoder is as follows:

[0101] Input a 3D CT image of size 1×128×128×128. The network first passes through a series of modules consisting of 1×1 convolution, residual block (ResBlock) and average pooling. Each residual block is followed by an average pooling operation with a window size of 2, which gradually reduces the spatial size of the feature map. The whole process is repeated four times. The number of channels of the 1×1 convolution in the four operations is 16, 64, 128, and 256 respectively. After each operation, the resolution of the feature map is halved, while the number of channels increases. The detailed structure of the residual block is given on the right. First, a convolution operation of size 3×3, stride 1, and padding 1 is performed on the input of the residual block. After the convolution, ReLU activation and batch normalization (Batch Norm) operation are performed; then the input features are added to the features obtained by convolution, and then another convolution operation of size 3×3, stride 1, and padding 1 is performed, also accompanied by ReLU activation and batch normalization. The benefit of ReLU is that it introduces nonlinearity, thereby enhancing the network's expressiveness and enabling the model to fit complex features; batch normalization can effectively accelerate the training process, stabilize the gradient flow, and avoid gradient vanishing or explosion. The process of passing input information directly to output through residual connections can effectively alleviate the gradient vanishing problem. In this process, the feature map obtained The sizes are 16×128×128×128, 64×64×64×64, 128×32×32×32, and 256×16×16×16 respectively.

[0102] The structure of the three-dimensional feature separator is as follows Figure 6 As shown, in this embodiment, the process of separating features by the three-dimensional feature separator is as follows:

[0103] The final output of the 3D feature extractor As input, The initial dimension of the feature is (C, H, W, D). In order to separate the three-dimensional feature into coronal, sagittal and axial features, the three-dimensional feature must be dimensionally adjusted first. The specific measures for adjustment are:

[0104]

[0105] It means rearranging its dimensions into (C, W, H, D). It means rearranging its dimensions into (C, H, D, W). It means rearranging its dimensions to (C, D, W, H), and Flip(·, -2) means flipping the second-to-last dimension of the feature map.

[0106] The adjusted features will be divided into Figure 6 The process is as shown in the figure. 3D_cor As an example, the processing flow of the other two features is the same. First, feature dimensionality reduction is performed, f 3D_cor The size of is (256,16,16,16). In the process of dimensionality reduction, f 3D_cor The first two dimensions of are flattened into a new dimension, and the flattened feature size is (4096, 16, 16), and then a convolution block is used to further process the features. The convolution block structure used in the whole process includes the following steps: first, a 1×1 convolution with a step size of , which is used to reduce the number of channels; followed by a 3×3 convolution with a step size of 1, which resizes the feature map to 14×14 to keep consistent with the output feature size of the 2D feature extraction module. After the convolution operation, a GeLU activation function and batch normalization layer are followed to improve the nonlinear expression ability of the model and stabilize the training process.

[0107] After passing through the three-dimensional feature separator, the coronal feature f with a feature map size of (1024, 14, 14) can be obtained. cor , sagittal characteristics f sag and axial features f axial .

[0108] The structure of the feature fusion module is as follows: Figure 7 As shown, in this embodiment, the feature fusion process is as follows:

[0109] First, the output of the two-dimensional feature extractor f 2D The output f of the three-dimensional feature extractor is cor ,f sag ,f axial Use the attention mechanism to cross-integrate to get f cor_2D ,f sag_2D ,f axial_2D ; then f cor_2D ,f sag_2D ,f axial_2D The concatenation operation is used to concatenate the features along the channel dimension to obtain an integrated feature map. The concatenated feature map is subjected to a 1×1 convolution operation, the main purpose of which is to reduce the number of channels, thereby reducing the computational complexity and retaining the key information of the input features. This is followed by a 3×3 convolution operation, which is used to extract spatial features and help the model further capture local structural information while maintaining the spatial resolution of the feature map. The convolved feature map is subjected to the GeLU activation function to introduce nonlinear transformations and increase the expressive power of the model. Finally, a batch normalization operation is performed to obtain the final fusion feature f of the feature fusion module. fuse , normalize the values ​​of feature maps to speed up training and stabilize model performance.

[0110] Among them, the process of cross-fusion using the attention mechanism is as follows:

[0111] Coronal features cor For example, the sagittal feature f sag and axial features f axial The processing flow is the same as that of the coronal feature f cor and the image two-dimensional feature f 2D After 1×1 convolution, flattening and transposition operations, they are adapted to the subsequent multi-head cross attention module. Then, the position embedding operation is performed on the operated features. By adding position information to the feature map, the model can better capture the spatial structure and relative position information of the input features in the global scope, thereby enhancing the network's perception of local and global features of the image. Next, the input data enters a layer normalization module to stabilize the training and reduce the gradient disappearance phenomenon. The feature vector after layer normalization is represented as f2 ′ D , f c ′ ol ; In the multi-head cross attention module, the two-dimensional feature f2 ′ D Generate the query matrix Q through linear transformation, and the coronal feature f c ′ ol The key matrix K and the value matrix V are generated through linear transformation. Then, cross-attention calculation is performed. Multiple attention heads are processed in parallel. The results of each head are merged through splicing operation. The number of output channels remains the same. The formula is as follows:

[0112]

[0113]

[0114] MCA=Concat(head 1 ,…,head N )W O ;

[0115] in, are the weights of query, key and value respectively; d k is the key matrix K i Dimension.

[0116] The output of the multi-head cross attention module is normalized and added to the original input (residual connection); next, the feature enters a multi-layer perceptron (MLP) module, and then the residual connection and layer normalization are performed again. After repeated stacking of multiple layers of cross attention and MLP modules, the target feature f is finally output. cor_2D .

[0117] After fusion, the feature f fuse In the above example, the ResUnet network structure is used for decoding. The decoding process gradually restores the spatial resolution of the features through multiple upsampling operations, and uses skip connections to directly pass the features of the corresponding scales of each layer of the encoder to the decoder to help restore more detailed information. In this embodiment, the encoder features used by the skip connection are the output features of the two-dimensional feature extractor. And the middle layer features and

[0118] Finally, it passes through a segmentation head, which includes a two-dimensional convolutional layer with a size of 3×3 and an output channel number equal to the number of segmentation target categories, and a softmax layer. The two-dimensional convolutional layer is used to map the feature map generated by the decoder to the number of channels of the target category, ensuring that the final output dimension matches the required number of segmentation categories. The softmax layer is used to map the decoded features to the segmentation result of the target to obtain the final pixel-level segmentation result. In a preferred embodiment, the image segmentation model provided in the embodiment of the present application is used to perform image segmentation on three processed X-ray images, and the image segmentation results obtained are as follows: Figure 8 shown.

[0119] In a possible implementation, the segmentation model is constructed based on a neural network model and trained according to a number of historical CT images and a number of corresponding DRR images, including:

[0120] Acquire a number of historical CT images and a number of corresponding DRR images at a preset position;

[0121] Constructing corresponding real segmentation maps according to each of the DRR images;

[0122] Combining each of the historical CT images, each of the DRR images, and each of the real segmentation maps to construct a training data set;

[0123] Constructing an initial image segmentation model based on a neural network model, wherein the initial image segmentation model includes a two-dimensional feature extraction module, a three-dimensional feature extraction module, a feature fusion module, a feature decoder and a segmentation head;

[0124] The training data set is input into the initial image segmentation model, and each model parameter in the initial image segmentation model is updated through a preset loss function based on supervised learning to obtain the segmentation model.

[0125] The embodiment of the present application limits the content of CT images and DRR images during the model training process, and only uses a number of historical CT images of preset positions and a number of corresponding DRR images to construct training data, so that the image segmentation model obtained through training can specialize in a certain human body position. For example, when the preset position is the ankle joint, the model can learn the knowledge of the unique bone morphology and soft tissue morphology of the ankle joint based on the X-ray image characteristics of the ankle joint. The image segmentation model obtained through training can be specifically used for image segmentation of the ankle joint position, further improving the segmentation accuracy of the model.

[0126] like Fig. 9 As shown, in a preferred embodiment, taking the preset position as the ankle joint as an example, an ankle joint X-ray image segmentation network (i.e., the image segmentation model) for performing image segmentation on the ankle joint X-ray image is constructed, including steps S301-S306:

[0127] Step S301, obtaining an ankle joint CT image and an ankle joint X-ray image.

[0128] Step S302: Perform DRR projection on the ankle CT image to obtain an ankle joint DRR image and construct a data set. The data set includes a training data set and an inference data set. The training data set consists of a CT image, a number of corresponding DRR images, and a true segmentation map; the inference data set consists of a CT image and a number of corresponding X-ray images.

[0129] Step S303, constructing an ankle joint X-ray image segmentation network fused with three-dimensional CT data. The ankle joint X-ray image segmentation network fused with three-dimensional CT data consists of a two-dimensional feature extraction module, a three-dimensional feature extraction module, a feature fusion module, a feature decoder and a segmentation head.

[0130] Step S304: Use the training data set and the loss function to train the ankle joint X-ray image segmentation network fused with the three-dimensional CT data so that the network can achieve good segmentation effect.

[0131] Step S305: perform a histogram normalization operation on the ankle joint X-ray image in the inference data set so that its histogram is substantially consistent with the histogram of the DRR image used during training.

[0132] Step S306: input the processed ankle X-ray image and its corresponding ankle CT image into the ankle X-ray image segmentation network fused with three-dimensional CT data, obtain the segmentation map through network prediction, simulate the actual application process of the model, and evaluate the image segmentation effect of the model.

[0133] like Fig.10 As shown, in step S302, the training data set includes ankle CT images, DRR images and true segmentation maps, which are expressed as: in represents the ith ankle CT image data, Reason Several DRR images obtained by DRR projection technology, for The corresponding true segmentation map. The inference dataset contains ankle CT images and X-ray images, specifically: in For the collection and The corresponding ankle X-ray. The CT images of the training data set and the inference data set do not completely overlap. The construction process of the true segmentation map is:

[0134] First, the ankle CT image is segmented (using 3D Slicer) to obtain three-dimensional image data of the tibia, fibula, talus and other bones. After applying the same change parameters to the CT image, tibia, fibula, talus and other bones, DRR projection is performed to obtain DRRs. The DRRs of the tibia, fibula, talus and other bones are binarized to obtain the segmentation map of the corresponding bones. Finally, these segmentation maps are merged to form the real segmentation map for model training.

[0135] After training in step S304, the model parameters are obtained. In the inference stage, the inference dataset Dataset is firstly inf The X-ray images in the dataset are subjected to histogram matching before being input into the model for inference. Histogram matching can balance the intensity distribution differences between different image sources, thereby reducing the performance degradation of the model during the inference phase due to inconsistent image characteristics. By adjusting the pixel intensity of the X-ray image to match the histogram distribution of the DRR image during training and to have statistical characteristics similar to those of the training data, the model can perform segmentation or classification more accurately and improve the inference accuracy. Similarly, in the subsequent application phase of the model, the X-ray images to be segmented must also be subjected to histogram matching before being input into the model for image segmentation.

[0136] Furthermore, the loss function is constructed by combining the Dice loss function and the cross entropy loss function.

[0137] The embodiment of the present application adopts a loss function constructed by combining Dice loss and cross entropy loss during the training process, wherein Dice loss is used to measure the degree of overlap between the model prediction and the true label, and is mainly used in image segmentation tasks. The higher the Dice coefficient, the higher the match between the prediction result and the true segmentation result. Cross-entropy loss is mainly used for pixel-level classification tasks to ensure that the model can accurately predict the category to which it belongs at each pixel. These two loss functions are combined for model training, while taking into account the optimization of class imbalance and global accuracy, ensuring that the model performs well in both classification accuracy and segmentation area overlap.

[0138] In a preferred embodiment, Dice loss is used to measure the overlap between the model prediction and the true label, and is mainly used in image segmentation tasks. The higher the Dice coefficient, the higher the match between the prediction result and the true segmentation result. The formula is:

[0139]

[0140] L Dice =1-Dice;

[0141] Cross-Entropy Loss is mainly used for pixel-level classification tasks to ensure that the model can accurately predict the category of each pixel. Its formula is:

[0142]

[0143] Among them, p i is the pixel value predicted by the model, g i is the pixel value of the true label, is the probability that pixel i is predicted by the model to be of category c, is the true label that pixel i belongs to category c, N is the total number of pixels, and C is the total number of pixels.

[0144] In order to optimize both the Dice coefficient and the cross entropy, the final loss function combines the two in a weighted manner:

[0145] L=α·L Dice +β·L CE ;

[0146] Among them, α and β are adaptively adjusted according to the size of Dice loss and cross entropy loss to ensure that the two losses have a proper balance during training. The weight of the part with larger loss is smaller, and the weight of the part with smaller loss is larger, thus achieving an adaptive balance so that the two losses work together during training.

[0147]

[0148] Through this adaptive adjustment, the model can optimize Dice loss and cross entropy loss in a more balanced manner at different stages, ensuring good performance in both classification accuracy and segmentation area overlap.

[0149] In one possible implementation, performing DRR projection on the historical CT image to obtain the historical DRR image includes: simulating the process of performing X-ray projection on the historical CT image based on the Siddon algorithm and GPU acceleration technology, and then generating the corresponding historical DRR image according to the simulation result.

[0150] The principle of Siddon algorithm to generate DRR images is to simulate the projection process of X-rays through ray tracing. Specifically, the algorithm calculates the path of each ray emitted from the projection source through a three-dimensional volume (such as CT scan data), and accurately calculates the intersection of the ray with each voxel and the length of the ray. Then, based on the density value of each voxel, the attenuation effect of the light in the volume is accumulated, and finally these attenuation values ​​are projected onto a two-dimensional plane to generate a DRR image similar to the actual X-ray projection. This method uses the efficiency of the Siddon algorithm to quickly determine which voxels are traversed by the ray and the contribution of the ray in the voxel, thereby accelerating the generation of DRR.

[0151] In addition to the above technical solutions, the embodiments of the present application also have the following alternative solutions in some technical details:

[0152] 1. The DRR technology of the embodiment of the present application is implemented based on the Siddon algorithm, and the Ray-Casting algorithm, DRR generation based on deep learning and other methods may also be used.

[0153] 2. The backbone network of the two-dimensional feature extractor of the embodiment of the present application is implemented based on Swin-Unet, and other networks that combine the advantages of convolutional neural networks (CNN) and modern Transformers, such as ConvNeXt, CMT, Hybrid CNN-Transformer, etc., can also be replaced.

[0154] 3. When fusing two-dimensional and three-dimensional features, the embodiments of the present application use features from the highest level. This method focuses on extracting and fusing features from the deepest level of the encoder, and may ignore fine-grained local details. Therefore, the extracted multi-scale three-dimensional features can also be fused with the two-dimensional features at different levels to ensure that the fused features contain both local information and global context, but it will take up higher computing resources.

[0155] 4. The loss in the embodiment of the present application currently only considers the optimization of class imbalance and global accuracy, but does not pay enough attention to the accuracy of the boundary. In order to further improve the segmentation effect, a boundary-aware loss can be introduced, combined with the existing loss, and additional optimization of the edge area of ​​the target object can be performed to ensure that the model can obtain higher accuracy when processing complex boundaries, thereby improving the overall segmentation quality.

[0156] In summary, the beneficial effects of the embodiments of the present application are as follows:

[0157] 1. The present invention extracts two-dimensional X-ray image features and three-dimensional image features respectively, fully utilizes the local details and global spatial structure information of the image, improves the feature capture ability of the model under low contrast and fuzzy boundary conditions, ensures the high accuracy and robustness of segmentation or classification tasks, and performs well in ankle X-ray image segmentation.

[0158] 2. The present invention effectively solves the boundary blur problem caused by overlapping anatomical structures in ankle joint X-ray images by extracting three-dimensional features in different slicing directions and fusing them with two-dimensional features at multiple levels, thereby enhancing the model's comprehensive understanding of complex structures and improving the accuracy of bone segmentation.

[0159] 3. In the feature fusion stage, the present invention introduces a cross-attention mechanism, which realizes the precise integration of information by adaptively modeling the dependency between two-dimensional and three-dimensional features. In particular, it can effectively improve the performance of the model when dealing with the problem of small difference in bone and soft tissue density and unclear boundaries in ankle X-ray images.

[0160] 4. The present invention adopts a combination of Dice loss and cross entropy loss during the training process, which effectively addresses the problems of unclear bone boundaries and category imbalance in ankle joint X-ray images, ensures the precise positioning and classification accuracy of the target area in the segmentation task, and improves the optimization of the overlap of the fuzzy boundary area.

[0161] 5. In the inference stage, the present invention normalizes the input two-dimensional X-ray image into a histogram so that the intensity distribution of the inference data is consistent with the training data, reducing the performance fluctuation caused by the contrast difference in the imaging process, thereby enhancing the generalization ability of the model, especially the stability under different X-ray imaging conditions.

[0162] It should be noted that, although the description of the first embodiment repeatedly mentions the segmentation of ankle joint X-ray images, it is only a specific embodiment of the present application and does not limit the scope of application and protection of the present application. The image segmentation method provided in the embodiment of the present application can also be used to segment X-ray images of other parts.

[0163] Embodiment 2:

[0164] like Fig.11 As shown, accordingly, embodiment 2 provides an X-ray image segmentation system for fusion CT data, including an acquisition module 10, an image segmentation module 20 and a model building module 30;

[0165] The acquisition module 10 is used to acquire the X-ray image to be segmented and the CT image of the corresponding position;

[0166] The image segmentation module 20 is used to input the X-ray image and the CT image into a preset image segmentation model, so that the image segmentation model extracts two-dimensional features from the X-ray image and three-dimensional features from the CT image, and after fusing the two-dimensional features with the three-dimensional features, generates a corresponding segmentation map according to the fused features;

[0167] The model building module 30 is used to build the segmentation model based on a neural network model and train according to a number of historical CT images and a number of corresponding historical DRR images, and the historical DRR images are obtained by performing DRR projection on the historical CT images.

[0168] In a possible implementation, the image segmentation module 20 inputs the X-ray image and the CT image into a preset image segmentation model, so that the image segmentation model extracts two-dimensional features from the X-ray image and three-dimensional features from the CT image, and after fusing the two-dimensional features and the three-dimensional features, generates a corresponding segmentation map according to the fused features, including:

[0169] According to the historical DRR image, a histogram normalization operation is performed on the X-ray image to generate a corresponding processed X-ray image;

[0170] The processed X-ray image and the CT image are input into a preset image segmentation model so that the image segmentation model generates a corresponding segmentation map according to the processed X-ray image and the CT image.

[0171] Furthermore, the step of performing a histogram normalization operation on the X-ray image according to the historical DRR image to generate a corresponding processed X-ray image includes:

[0172] According to a preset pixel range, respectively calculating the histograms of the historical DRR image and the X-ray image to obtain corresponding DRR image histograms and X-ray image histograms;

[0173] Calculating the cumulative distribution functions of the DRR image histogram and the X-ray image histogram respectively to obtain corresponding DRR image cumulative distribution functions and X-ray image cumulative distribution functions;

[0174] Determining a pixel mapping function according to the DRR image cumulative distribution function and the X-ray image cumulative distribution function;

[0175] According to the pixel mapping function, the X-ray image is pixel mapped to generate the processed X-ray image.

[0176] Furthermore, the image segmentation model generates a corresponding segmentation map according to the processed X-ray image and the CT image, including:

[0177] Performing two-dimensional feature encoding on the processed X-ray image, extracting multi-scale semantic features, and generating a corresponding two-dimensional feature map;

[0178] Performing three-dimensional feature coding on the CT image, extracting three-dimensional spatial information, and obtaining three-dimensional coding features;

[0179] The three-dimensional coded features are subjected to three-dimensional feature separation, and three corresponding feature maps are generated after dimension order adjustment, dimension flipping, feature dimension reduction and convolution operations, including a coronal feature map, a sagittal feature map and an axial feature map;

[0180] Based on the cross attention mechanism, the two-dimensional feature map is cross-fused with the three feature maps respectively to obtain three corresponding fused feature maps;

[0181] After performing a splicing operation, a convolution operation, and a batch normalization operation on the three fused feature maps, a normalized fused feature map is obtained;

[0182] Decoding the normalized fused feature map, converting the features in the normalized fused feature map into a two-dimensional space, and obtaining a two-dimensional fused feature map;

[0183] Based on a preset number of segmentation categories, two-dimensional convolution and feature mapping are performed on the two-dimensional fusion feature map to generate the segmentation map.

[0184] In a possible implementation, the model building module 30 is constructed based on a neural network model and trained according to a number of historical CT images and a number of corresponding DRR images to obtain the segmentation model, including:

[0185] Acquire a number of historical CT images and a number of corresponding DRR images at a preset position;

[0186] Constructing corresponding real segmentation maps according to each of the DRR images;

[0187] Combining each of the historical CT images, each of the DRR images, and each of the real segmentation maps to construct a training data set;

[0188] Constructing an initial image segmentation model based on a neural network model, wherein the initial image segmentation model includes a two-dimensional feature extraction module, a three-dimensional feature extraction module, a feature fusion module, a feature decoder and a segmentation head;

[0189] The training data set is input into the initial image segmentation model, and each model parameter in the initial image segmentation model is updated through a preset loss function based on supervised learning to obtain the segmentation model.

[0190] Furthermore, the loss function is constructed by combining the Dice loss function and the cross entropy loss function.

[0191] In one possible implementation, performing DRR projection on the historical CT image to obtain the historical DRR image includes: simulating the process of performing X-ray projection on the historical CT image based on the Siddon algorithm and GPU acceleration technology, and then generating the corresponding historical DRR image according to the simulation result.

[0192] The embodiment of the present application provides an X-ray image segmentation system that integrates CT data, and realizes the automatic segmentation of X-ray images through a preset image segmentation model. In the image segmentation process, the embodiment of the present application extracts the two-dimensional features of the X-ray image and the three-dimensional features of the CT image respectively and performs feature fusion, so that the image segmentation model can obtain more reference information related to the X-ray image, and helps the image segmentation model to more accurately segment the X-ray image. In addition, during the training process of the image segmentation model, several historical CT images are processed by DRR projection to obtain several corresponding historical DDR images. Through this technical means, the effective information of high-dimensional data is fully utilized, and two-dimensional DDR image data is generated based on three-dimensional CT image data, so that the corresponding X-ray image data is not required during the model training process. While ensuring the training effect, the demand for training data is reduced and the utilization rate of CT image data is improved. Compared with the prior art, since the embodiment of the present application introduces high-dimensional CT image data in both the model training and image segmentation processes, it fully utilizes the local details and global spatial structure information of the image, improves the model's feature capture capability under low contrast and fuzzy boundary conditions, ensures high precision and robustness of segmentation or classification tasks, effectively solves the problem of blurred boundaries of tissue parts in X-ray images, and improves the accuracy of automatic segmentation of various tissue structures in X-ray images.

[0193] The more detailed working principle and step flow of this embodiment can refer to, but are not limited to, the relevant records of Embodiment 1.

[0194] The specific embodiments described above further describe the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only a specific embodiment of the present application and is not intended to limit the scope of protection of the present application. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of protection of the present application.

Claims

1. A method for segmenting an X-ray image by fusing CT data, characterized in that: include: Acquire the X-ray image to be segmented and the CT image of the corresponding position; Inputting the X-ray image and the CT image into a preset image segmentation model, so that the image segmentation model extracts two-dimensional features from the X-ray image and three-dimensional features from the CT image, and after fusing the two-dimensional features with the three-dimensional features, generates a corresponding segmentation map according to the fused features; The segmentation model is constructed based on a neural network model and is obtained by training according to a number of historical CT images and a number of corresponding historical DRR images, and the historical DRR images are obtained by performing DRR projection on the historical CT images.

2. The X-ray image segmentation method for fused CT data according to claim 1, characterized in that: The step of inputting the X-ray image and the CT image into a preset image segmentation model so that the image segmentation model extracts two-dimensional features from the X-ray image and three-dimensional features from the CT image, and after fusing the two-dimensional features with the three-dimensional features, generating a corresponding segmentation map according to the fused features includes: According to the historical DRR image, a histogram normalization operation is performed on the X-ray image to generate a corresponding processed X-ray image; The processed X-ray image and the CT image are input into a preset image segmentation model so that the image segmentation model generates a corresponding segmentation map according to the processed X-ray image and the CT image.

3. The X-ray image segmentation method for fused CT data according to claim 2, characterized in that: The step of performing a histogram normalization operation on the X-ray image according to the historical DRR image to generate a corresponding processed X-ray image includes: According to a preset pixel range, respectively calculating the histograms of the historical DRR image and the X-ray image to obtain corresponding DRR image histograms and X-ray image histograms; Calculating the cumulative distribution functions of the DRR image histogram and the X-ray image histogram respectively to obtain corresponding DRR image cumulative distribution functions and X-ray image cumulative distribution functions; Determining a pixel mapping function according to the DRR image cumulative distribution function and the X-ray image cumulative distribution function; According to the pixel mapping function, the X-ray image is pixel mapped to generate the processed X-ray image.

4. The X-ray image segmentation method for fused CT data according to claim 2, characterized in that: The image segmentation model generates a corresponding segmentation map according to the processed X-ray image and the CT image, including: Performing two-dimensional feature encoding on the processed X-ray image, extracting multi-scale semantic features, and generating a corresponding two-dimensional feature map; Performing three-dimensional feature coding on the CT image, extracting three-dimensional spatial information, and obtaining three-dimensional coding features; The three-dimensional coded features are subjected to three-dimensional feature separation, and three corresponding feature maps are generated after dimension order adjustment, dimension flipping, feature dimension reduction and convolution operations, including a coronal feature map, a sagittal feature map and an axial feature map; Based on the cross attention mechanism, the two-dimensional feature map is cross-fused with the three feature maps respectively to obtain three corresponding fused feature maps; After performing a splicing operation, a convolution operation, and a batch normalization operation on the three fused feature maps, a normalized fused feature map is obtained; Decoding the normalized fused feature map, converting the features in the normalized fused feature map into a two-dimensional space, and obtaining a two-dimensional fused feature map; Based on a preset number of segmentation categories, two-dimensional convolution and feature mapping are performed on the two-dimensional fusion feature map to generate the segmentation map.

5. The X-ray image segmentation method for fused CT data according to claim 1, characterized in that: The segmentation model is constructed based on the neural network model and trained according to a number of historical CT images and a number of corresponding DRR images, including: Acquire a number of historical CT images and a number of corresponding DRR images at a preset position; Constructing corresponding real segmentation maps according to each of the DRR images; Combining each of the historical CT images, each of the DRR images, and each of the real segmentation maps to construct a training data set; Constructing an initial image segmentation model based on a neural network model, wherein the initial image segmentation model includes a two-dimensional feature extraction module, a three-dimensional feature extraction module, a feature fusion module, a feature decoder and a segmentation head; The training data set is input into the initial image segmentation model, and each model parameter in the initial image segmentation model is updated through a preset loss function based on supervised learning to obtain the segmentation model.

6. The X-ray image segmentation method for fused CT data according to claim 5, characterized in that: The loss function is constructed by combining the Dice loss function and the cross entropy loss function.

7. The X-ray image segmentation method for fused CT data according to claim 1, characterized in that: The performing DRR projection on the historical CT image to obtain the historical DRR image includes: simulating the process of performing X-ray projection on the historical CT image based on Siddon algorithm and GPU acceleration technology, and then generating the corresponding historical DRR image according to the simulation result.

8. An X-ray image segmentation system integrating CT data, characterized in that: It includes an acquisition module, an image segmentation module and a model building module; Wherein, the acquisition module is used to acquire the X-ray image to be segmented and the CT image of the corresponding position; The image segmentation module is used to input the X-ray image and the CT image into a preset image segmentation model, so that the image segmentation model extracts two-dimensional features from the X-ray image and three-dimensional features from the CT image, and after fusing the two-dimensional features and the three-dimensional features, generates a corresponding segmentation map according to the fused features; The model building module is used to build the segmentation model based on a neural network model and train according to several historical CT images and corresponding several historical DRR images, and the historical DRR images are obtained by performing DRR projection on the historical CT images.

9. The X-ray image segmentation system for fusion of CT data according to claim 8, characterized in that: The image segmentation module inputs the X-ray image and the CT image into a preset image segmentation model, so that the image segmentation model extracts two-dimensional features from the X-ray image and three-dimensional features from the CT image, and after fusing the two-dimensional features with the three-dimensional features, generates a corresponding segmentation map according to the fused features, including: According to the historical DRR image, a histogram normalization operation is performed on the X-ray image to generate a corresponding processed X-ray image; The processed X-ray image and the CT image are input into a preset image segmentation model so that the image segmentation model generates a corresponding segmentation map according to the processed X-ray image and the CT image.

10. The X-ray image segmentation system for fusion of CT data according to claim 9, characterized in that: The image segmentation model generates a corresponding segmentation map according to the processed X-ray image and the CT image, including: Performing two-dimensional feature encoding on the processed X-ray image, extracting multi-scale semantic features, and generating a corresponding two-dimensional feature map; Performing three-dimensional feature coding on the CT image, extracting three-dimensional spatial information, and obtaining three-dimensional coding features; The three-dimensional coded features are subjected to three-dimensional feature separation, and three corresponding feature maps are generated after dimension order adjustment, dimension flipping, feature dimension reduction and convolution operations, including a coronal feature map, a sagittal feature map and an axial feature map; Based on the cross attention mechanism, the two-dimensional feature map is cross-fused with the three feature maps respectively to obtain three corresponding fused feature maps; After performing a splicing operation, a convolution operation, and a batch normalization operation on the three fused feature maps, a normalized fused feature map is obtained; Decoding the normalized fused feature map, converting the features in the normalized fused feature map into a two-dimensional space, and obtaining a two-dimensional fused feature map; Based on a preset number of segmentation categories, two-dimensional convolution and feature mapping are performed on the two-dimensional fusion feature map to generate the segmentation map.

Images