2D to 3D image reconstruction method and device, electronic equipment and storage medium

Abstract

The invention provides a 2D-to-3D image reconstruction method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining an original 3D bone segmentation image corresponding to a 3D CT image, calculating and aligning the 3D bone segmentation image according to the original 3D bone segmentation image, a real projection parameter and a fixed projection parameter, and obtaining a 3D-to-3D image; the original 3D bone segmentation image is mapped through the real projection parameter to obtain a 2D real projection positive and lateral position X-ray image, and the 2D real projection positive and lateral position X-ray image is mapped through the fixed projection parameter to obtain a 2D fixed projection positive and lateral position X-ray image, and the 2D fixed projection positive and lateral position X-ray image is mapped through the fixed projection parameter to obtain the 3D bone segmentation image. Therefore, the change of the real projection parameters is converted into the change of the pose and the form of the shot object under the fixed projection parameters; and inputting the 2D real projection positive and lateral X-ray image into the trained three-dimensional reconstruction model to obtain a predicted 3D aligned bone segmentation image, and calculating a real 3D bone segmentation image according to the predicted 3D aligned bone segmentation image, thereby improving the accuracy of model prediction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a 2D to 3D image reconstruction method and device, electronic equipment and storage medium. BACKGROUND

[0002] X-ray generated 3D segmentation refers to inputting information including frontal and lateral X-ray images and X-ray parameter information (such as resolution, ISO center, detector distance, tube distance and the like), and then outputting a 3D segmentation result through an algorithm. At present, there are mainly two strategies, one is based on a network, and the other is a traditional method such as SSM (statistical shape models) and MPPM (model deformation moving least squares).

[0003] However, the current traditional method and deep learning method do not take into account the change of the isocenter and the X-ray tube position and the detector distance (such as shown in Figure 1 Especially for larger structures such as the pelvis, this deformation is more obvious. The prior art discusses a similar problem of a human face, that is, the influence of the camera distance on the human face. The prior art also notices a similar problem of X-rays. However, both of them try to solve the problem in a 2D to 2D mapping manner.

[0004] Figure 2 (a) and (b) are X-ray images generated by the Drr algorithm using different isocenter parameters for the same CT sample, Figure 2 There is a subtle difference between the (a) and (b) images, and this difference also exists in actual X-ray imaging. This problem leads to non-unique projection from 3D to 2D, and there is a case of information loss in the restoration from 2D to 3D. This makes the algorithm training process ambiguous and easily falls into a local minimum. These problems significantly reduce the generalization ability of the algorithm.

[0005] In the prior art, the parameters of the X-ray (X-ray) shooting device are assumed to remain unchanged, and a digital reconstructed image (DRR) is generated on this basis, without fully considering the potential impact of the change of the X-ray shooting device parameters on three-dimensional image segmentation. In other words, if there is a significant difference between the X-ray parameters in the training set and the X-ray parameters of the image to be processed, the accuracy of the prediction result will inevitably be affected, resulting in a poor prediction result. SUMMARY

[0006] The present application aims to overcome the above technical deficiencies, and provide a 3D segmented image generation method and device, electronic equipment and storage medium, to solve the problem of the influence of X-ray shooting parameter changes on three-dimensional image segmentation accuracy in related technologies.

[0007] To achieve the above technical purpose, the present application adopts the following technical scheme:

[0008] According to a first aspect of the present application, a 2D to 3D image reconstruction method is provided, which comprises:

[0009] Standardizing the real projection parameters of the 3D CT image to reduce the degrees of freedom of the mapping relationship between the 3D position information and the 2D position information;

[0010] According to the real projection parameters, performing DRR projection on the 3D CT image to obtain a 2D real projection anteroposterior X-ray image;

[0011] Position encoding is performed on the 2D real projection anteroposterior X-ray image to retain the global position information of the 2D real projection anteroposterior X-ray image;

[0012] According to a random mask method, a 3D CT reserved region image and a 2D local mapping image of the 2D real projection anteroposterior X-ray image are obtained to improve the diversity of image data;

[0013] An original 3D bone segmentation image corresponding to the 3D CT image is obtained, and an aligned 3D bone segmentation image is calculated according to the original 3D bone segmentation image, the real projection parameters and the fixed projection parameters, so that the 2D real projection anteroposterior X-ray image obtained by mapping the original 3D bone segmentation image through the real projection parameters is the same as the 2D fixed projection anteroposterior X-ray image obtained by mapping the aligned 3D bone segmentation image through the fixed projection parameters;

[0014] In the training stage, the 2D real projection anteroposterior X-ray image is taken as the input, the aligned 3D bone segmentation image is taken as the true value, and / or the 2D local mapping image is taken as the input, and the 3D CT reserved region image is taken as the true value, a pre-constructed neural network model is trained, the neural network model learns the corresponding relationship between the 2D real projection anteroposterior X-ray image and the aligned 3D bone segmentation image, and / or the neural network model learns the corresponding relationship between the 2D local mapping image and the 3D CT reserved region image, until the model converges, and a three-dimensional reconstruction model is obtained;

[0015] In the prediction stage, the 2D real projection anteroposterior X-ray image is input into the three-dimensional reconstruction model to obtain a predicted 3D aligned bone segmentation image, and a real 3D bone segmentation image is obtained by calculating the inverse of a matrix composed of real projection parameters and fixed projection parameters according to the predicted 3D aligned bone segmentation image.

[0016] Preferably, the method for position coding the 2D real projection anteroposterior X-ray image comprises:

[0017] Position feature information embedding is performed on the 2D real projection anteroposterior X-ray image.

[0018] The 2D real projection anteroposterior X-ray image after position feature information embedding is subjected to spatial information cropping, and a target region is extracted from a non-blank region of the 2D real projection anteroposterior X-ray image.

[0019] Preferably, the method for extracting the target region comprises:

[0020] According to the global spatial information, position embedding is performed on the 2D real projection anteroposterior X-ray image, and the 2D real projection anteroposterior X-ray image is cropped according to the target region, so as to avoid destroying the mapping relationship between the 2D position information and the 3D position information when directly cropping, and to reduce the input image size, thereby reducing the calculation amount.

[0021] Preferably, the random mask method comprises:

[0022] According to the random cube mask and projection method, the 2D real projection anteroposterior X-ray image is enhanced and augmented to improve the robustness of the neural network model.

[0023] Preferably, the random cube mask and projection method comprises:

[0024] According to a preset region size parameter, a cube region is randomly selected in the 3D CT image, and the pixel values of the 3D CT image except the cube region are set to zero to obtain a 3D CT reserved region image;

[0025] The 3D CT reserved region image is projected onto a two-dimensional plane to obtain a 2D local mapping image.

[0026] Preferably, the method for calculating the aligned 3D bone segmentation image comprises:

[0027] The mapping relationship between the 2D real projection anteroposterior X-ray image and the original 3D bone segmentation image is represented by a real affine matrix, and the real affine matrix is calculated according to real projection parameters;

[0028] The mapping relationship between the 2D fixed projection frontal-lateral X-ray image and the aligned 3D bone segmentation image is represented by a fixed affine matrix, which is calculated according to fixed projection parameters.

[0029] According to the real affine matrix and the fixed affine matrix, the mapping relationship between the original 3D bone segmentation image and the aligned 3D bone segmentation image is constructed, so that the change of the real projection parameters is converted into the change of the pose and morphology of the photographed object under the fixed projection parameters.

[0030] Preferably, the method of DRR projection comprises:

[0031] The real projection parameters of different combinations are sampled to obtain 2D real projection frontal-lateral X-ray images.

[0032] According to a second aspect of the present application, a 2D to 3D image reconstruction device is provided, which comprises:

[0033] A normalization module is configured to normalize the real projection parameters of the 3D CT image to reduce the degrees of freedom of the mapping relationship between the 3D position information and the 2D position information.

[0034] A projection module is configured to perform DRR projection on the 3D CT image according to the real projection parameters to obtain 2D real projection frontal-lateral X-ray images.

[0035] A position encoding module is configured to perform position encoding on the 2D real projection frontal-lateral X-ray images to retain the global position information of the 2D real projection frontal-lateral X-ray images.

[0036] An enhancement module is configured to obtain 3D CT reserved region images and 2D local mapping images of the 2D real projection frontal-lateral X-ray images according to a random mask method to improve the diversity of image data.

[0037] A calculation module is configured to obtain an original 3D bone segmentation image corresponding to the 3D CT image, and calculate an aligned 3D bone segmentation image according to the original 3D bone segmentation image, the real projection parameters and the fixed projection parameters, so that the 2D real projection frontal-lateral X-ray image obtained by mapping the original 3D bone segmentation image through the real projection parameters is the same as the 2D fixed projection frontal-lateral X-ray image obtained by mapping the aligned 3D bone segmentation image through the fixed projection parameters.

[0038] a training module, configured to, in a training phase, input the 2D real projection anteroposterior X-ray image into a pre-constructed neural network model, input the 3D aligned bone segmentation image as a true value, and / or input the 2D local mapping image into the pre-constructed neural network model, input the 3D CT reserved region image as a true value, train the pre-constructed neural network model, make the neural network model learn the corresponding relationship between the 2D real projection anteroposterior X-ray image and the 3D aligned bone segmentation image, and / or make the neural network model learn the corresponding relationship between the 2D local mapping image and the 3D CT reserved region image, until the model converges, and obtain a three-dimensional reconstruction model.

[0039] a prediction module, configured to, in a prediction phase, input the 2D real projection anteroposterior X-ray image into the three-dimensional reconstruction model, obtain a predicted 3D aligned bone segmentation image, and obtain a real 3D bone segmentation image by calculating the inverse of a matrix composed of real projection parameters and fixed projection parameters according to the predicted 3D aligned bone segmentation image.

[0040] According to a third aspect of the present application, an electronic device is provided, comprising:

[0041] a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete communication with each other through the communication bus;

[0042] a memory, configured to store a computer program;

[0043] a processor, configured to execute the program stored on the memory, and realize the method.

[0044] According to a fourth aspect of the present application, a non-transitory computer readable storage medium storing computer instructions is provided, the computer instructions are used to make a computer execute the method.

[0045] The technical scheme provided by the embodiments of the present application can have the following beneficial effects:

[0046] By standardizing the real projection parameters of the 3D CT image, the degrees of freedom of the mapping relationship between the 3D position information and the 2D position information are reduced, the sampling data amount of the 2D real projection anteroposterior X-ray image is reduced, and the accuracy of the 2D real projection anteroposterior X-ray image is improved; the position of the 2D real projection anteroposterior X-ray image is encoded, the global position information of the 2D real projection anteroposterior X-ray image is retained, and the target region in the 2D real projection anteroposterior X-ray image is extracted; according to the random mask method, the 3D CT retained region image and the 2D local mapping image of the 2D real projection anteroposterior X-ray image are obtained, the image data diversity is improved, and the model fails to learn the local mapping relationship from 2D to 3D is avoided; the original 3D bone segmentation image corresponding to the 3D CT image is obtained, and the aligned 3D bone segmentation image is calculated according to the original 3D bone segmentation image, the real projection parameters and the fixed projection parameters, so that the 2D real projection anteroposterior X-ray image obtained by mapping the original 3D bone segmentation image through the real projection parameters is the same as the 2D fixed projection anteroposterior X-ray image obtained by mapping the aligned 3D bone segmentation image through the fixed projection parameters, thereby converting the change of the real projection parameters into the change of the pose and morphology of the photographed object under the fixed projection parameters; in the training stage, the 2D real projection anteroposterior X-ray image is taken as the input, the aligned 3D bone segmentation image is taken as the true value, and / or, the 2D local mapping image is taken as the input, and the 3D CT retained region image is taken as the true value, the pre-constructed neural network model is trained, the corresponding relationship between the 2D real projection anteroposterior X-ray image and the aligned 3D bone segmentation image is learned by the neural network model, and / or, the corresponding relationship between the 2D local mapping image and the 3D CT retained region image is learned by the neural network model, until the model converges, and a three-dimensional reconstruction model is obtained; in the prediction stage, the 2D real projection anteroposterior X-ray image is input into the three-dimensional reconstruction model to obtain a predicted 3D aligned bone segmentation image, and a real 3D bone segmentation image is obtained by calculating the inverse of the matrix composed of the real projection parameters and the fixed projection parameters according to the predicted 3D aligned bone segmentation image, thereby improving the accuracy of model prediction. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 is a structural schematic diagram of an X-ray imaging device according to an exemplary embodiment;

[0048] Figure 2 is a 2D anteroposterior X-ray image obtained according to different real projection parameters according to another exemplary embodiment;

[0049] Figure 3 is a flowchart of a 2D to 3D image reconstruction method according to another exemplary embodiment;

[0050] Figure 4 is a schematic diagram of the effect of a random mask and projection method according to another example embodiment;

[0051] Figure 5 is a flowchart of a VAE network model training method according to another example embodiment;

[0052] Figure 6 is a flowchart of a VAE network model generating a 3D segmentation result according to another example embodiment;

[0053] Figure 7 is a 2D real projection frontal-lateral X-ray image according to another example embodiment;

[0054] Figure 8 is a schematic diagram of the effect of a method of calculating an aligned 3D bone segmentation image according to another example embodiment;

[0055] Figure 9 is a structural schematic diagram of a 2D to 3D image reconstruction device according to another example embodiment;

[0056] Figure 10 is a structural schematic diagram of an electronic device according to another example embodiment. DETAILED DESCRIPTION

[0057] In order to enable persons skilled in the art to better understand the scheme of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative labor should fall within the scope of protection of the present application.

[0058] Embodiment One

[0059] Figure 3 is a flowchart of a 2D to 3D image reconstruction method according to an example embodiment, as shown in Figure 3 the 2D to 3D image reconstruction method comprises:

[0060] Step S11, standardizing real projection parameters of a 3D CT image;

[0061] Step S12, performing DRR projection on the 3D CT image according to the real projection parameters to obtain a 2D real projection frontal-lateral X-ray image;

[0062] Step S13, position encoding is performed on the 2D real projection anteroposterior X-ray image;

[0063] Step S14, according to a random mask method, a 3D CT reserved region image and a 2D local mapping image of the 2D real projection anteroposterior X-ray image are obtained;

[0064] Step S15, an original 3D bone segmentation image corresponding to the 3D CT image is obtained, and an aligned 3D bone segmentation image is calculated according to the original 3D bone segmentation image, the real projection parameter and the fixed projection parameter, so that a 2D real projection anteroposterior X-ray image obtained by mapping the original 3D bone segmentation image through the real projection parameter is the same as a 2D fixed projection anteroposterior X-ray image obtained by mapping the aligned 3D bone segmentation image through the fixed projection parameter;

[0065] Step S16, in the training stage, the 2D real projection anteroposterior X-ray image is taken as input, the aligned 3D bone segmentation image is taken as true value, and / or, the 2D local mapping image is taken as input, and the 3D CT reserved region image is taken as true value, a pre-constructed neural network model is trained, the neural network model learns the corresponding relationship between the 2D real projection anteroposterior X-ray image and the aligned 3D bone segmentation image, and / or, the neural network model learns the corresponding relationship between the 2D local mapping image and the 3D CT reserved region image, until the model converges, and a three-dimensional reconstruction model is obtained;

[0066] Step S17, in the prediction stage, the 2D real projection anteroposterior X-ray image is input into the three-dimensional reconstruction model to obtain a predicted 3D aligned bone segmentation image, and a real 3D bone segmentation image is obtained by calculating the inverse of a matrix composed of the real projection parameter and the fixed projection parameter according to the predicted 3D aligned bone segmentation image.

[0067] It should be noted that the X-ray shooting parameters refer to various parameters that need to be set when X-ray shooting is performed, for example: isocenter, detector distance, sensor and tube distance, pixel resolution, origin and pitch, direction, etc. In the actual projection process, an affine matrix can be calculated according to the X-ray shooting parameters, and the affine matrix is used for mutual conversion between 2D images and 3D images. Before calculating the affine matrix, the X-ray shooting parameters need to be sampled, and reasonable sampling parameters can obtain a more accurate affine matrix, so that the conversion between 2D images and 3D images is more accurate. If the affine matrix is a 4x6 matrix, and the X-ray shooting parameters are 8, which are isocenter, detector distance, sensor and tube distance, pixel resolution, origin and pitch, direction, if each parameter is sampled 8 times, a total of 824 Sampling multiple times will result in an excessively large amount of calculation, so the sampling parameters need to be reduced. For example, the reduction method is to require different patients to take the same standing position for X-ray shooting so that the direction can be consistent, thereby effectively reducing the sampling of the X-ray shooting parameter of direction.

[0068] It should be noted that because the 2D real-projection anteroposterior and lateral X-ray images are obtained by projecting the 3D CT images according to the real projection parameters, it is generally necessary to input the 2D real-projection anteroposterior and lateral X-ray images and the real projection parameters into the neural network model training, and use the aligned 3D bone segmentation images as the true value to enable the neural network model to learn the correspondence between the 2D real-projection anteroposterior and lateral X-ray images and the aligned 3D bone segmentation images. However, after using the aligned 3D bone segmentation images as the true value, the X-ray shooting parameters are actually fixed. Therefore, only the 2D anteroposterior and lateral X-ray images need to be input into the neural network model to establish the correspondence between the 2D anteroposterior and lateral X-ray images and the aligned 3D bone segmentation images, without the need to input the real projection parameters. This greatly reduces the amount of computation required during the model training process. Moreover, due to the calculation process of the aligned 3D bone segmentation images, by simulating the adjustment of the patient's shooting posture, the problem of non-unique 3D to 2D projection caused by changes in X-ray shooting parameters is avoided.

[0069] like Figure 4 As shown, during the training phase, 2D reserved area projection images and / or 2D real projection anteroposterior and lateral X-ray images are input into the neural network model, so that the neural network model can perform accurate 3D segmentation on various 2D anteroposterior and lateral X-ray images, avoiding the overfitting problem and making the model more robust.

[0070] like Figure 5 As shown in the figure, the neural network model can use the VAE network model. The VAE network consists of two parts: the encoding network (which can be implemented by a convolutional neural network or transformer, used to abstract and downsample information) and the decoding network (which can be implemented by a convolutional neural network or transformer, used to gradually concretize abstract information). The training process of the VAE network model is as follows:

[0071] Inputting pre-processed 2D anteroposterior and lateral X-ray images, and / or 2D reserved area projection images;

[0072] The input data is [256, 320, 16], and then downsampling is used to obtain two 2D features of [8, 10, 256] in the front and side positions, where 8 and 10 represent 2D spatial information and 256 represents feature information;

[0073] Then use transformer to output the features of [5,8,8,128], where [5,8,8] represents 3D spatial information and 128 represents feature information. It is then passed to the decoding network (decode) to finally generate the segmentation result of [160,256,256,5], where [160,256,256] represents spatial information and 5 represents the number of output categories.

[0074] Original 3D bone segmentation image S org , through the above formula S aligned [P aligned ]=S org [P org ], calculate the aligned 3D bone segmentation image S aligned , and S aligned Perform corresponding random mask processing;

[0075] The decoding network generates the predicted aligned 3D bone segmentation image Y aligned and compare it with S aligned Compare and then calculate the loss function value (loss) and perform backpropagation to update the hyperparameters in the VAE network model until the model converges and the training of the VAE network model is completed.

[0076] like Figure 6 As shown, the prediction process of the VAE network model is as follows:

[0077] Input 2D real mapping anteroposterior and lateral X-ray images;

[0078] The VAE network model generates predicted aligned 3D bone segmentation images Y aligned , change Y aligned Substitute into the above formula S aligned [P aligned ]=S org [P org ](Due to Y aligned and S aligned They are all aligned 3D bone segmentation images and also satisfy the formula), and we get Y aligned [P aligned ]=Y org [P org ], thereby calculating the real 3D bone segmentation image Y org .

[0079] The technical scheme provided by the embodiment reduces the degrees of freedom of the mapping relationship between 3D position information and 2D position information by standardizing the real projection parameters of the 3D CT image, reduces the sampling data amount of the 2D real projection anteroposterior X-ray image, and improves the accuracy of the 2D real projection anteroposterior X-ray image; the position encoding of the 2D real projection anteroposterior X-ray image retains the global position information of the 2D real projection anteroposterior X-ray image and extracts a target region in the 2D real projection anteroposterior X-ray image; the 3D CT reserved region image and the 2D local mapping image of the 2D real projection anteroposterior X-ray image are obtained according to the random mask method, the image data diversity is improved, and the model is unable to learn the local mapping relationship from 2D to 3D is avoided; the original 3D bone segmentation image corresponding to the 3D CT image is obtained, and the aligned 3D bone segmentation image is calculated according to the original 3D bone segmentation image, the real projection parameters and the fixed projection parameters, so that the 2D real projection anteroposterior X-ray image obtained by mapping the original 3D bone segmentation image through the real projection parameters is the same as the 2D fixed projection anteroposterior X-ray image obtained by mapping the aligned 3D bone segmentation image through the fixed projection parameters, thereby converting the change of the real projection parameters into the change of the pose and morphology of the photographed object under the fixed projection parameters; in the training stage, the 2D real projection anteroposterior X-ray image is taken as the input, the aligned 3D bone segmentation image is taken as the true value, and / or, the 2D local mapping image is taken as the input, and the 3D CT reserved region image is taken as the true value, the pre-constructed neural network model is trained, the corresponding relationship between the 2D real projection anteroposterior X-ray image and the aligned 3D bone segmentation image is learned by the neural network model, and / or, the corresponding relationship between the 2D local mapping image and the 3D CT reserved region image is learned by the neural network model, until the model converges, and a three-dimensional reconstruction model is obtained; in the prediction stage, the 2D real projection anteroposterior X-ray image is input into the three-dimensional reconstruction model to obtain a predicted 3D aligned bone segmentation image, and the real 3D bone segmentation image is obtained by calculating the inverse of the matrix composed of the real projection parameters and the fixed projection parameters according to the predicted 3D aligned bone segmentation image, thereby improving the accuracy of model prediction.

[0080] In specific practice, the method of encoding the 2D real projection anteroposterior X-ray image includes:

[0081] Position feature information embedding is performed on the 2D real projection anteroposterior X-ray image.

[0082] The 2D real projection anteroposterior X-ray image after position feature information embedding is subjected to spatial information cropping, and a target region is extracted from a non-blank region of the 2D real projection anteroposterior X-ray image.

[0083] In specific practice, the method of extracting the target region comprises:

[0084] According to the global spatial information, the 2D real projection anteroposterior X-ray image is subjected to position embedding, and the 2D real projection anteroposterior X-ray image is cropped according to the target region, so as to avoid destroying the mapping relationship between the 2D position information and the 3D position information and reducing the input image size, thereby reducing the calculation amount.

[0085] As shown in the formula: Figure 7 The 2D anteroposterior X-ray image obtained after projection of the 3D CT image has a large size (such as 2000x2000 pixels), and due to the difference in real projection parameters, the target region (ROI) is significantly different, and part of the blank area appears in the 2D real projection anteroposterior X-ray image. In order to meet the GPU memory limit and retain the effective area in the image, the image generally needs to be cropped first, but such operation will destroy the corresponding relationship between the 2D image and the 3D image, so the 2D real projection anteroposterior X-ray image is subjected to position embedding first and then cropping.

[0086] The position embedding feature is calculated according to the following formula:

[0087]

[0088]

[0089]

[0090]

[0091] Wherein, PE represents the position embedding feature; d model represents the feature dimension, d model here is 4; the position encoding feature dimension here is set to 4xd model , that is, a 16-dimensional feature; h represents the row index, the value range of which is [0, H-1], and H represents the image height, which is set to 500 here; w represents the column index, the value range of which is [0, W-1], and W represents the image width, which is set to 500 here; k represents the dimension index of the position encoding, the value range of which is [0, d model -1].

[0092] The block processing of the image is realized according to the following formula:

[0093] X = rearrange (im, '(h x b1) (w x b2) → (h x w) (b1 x b2) ') + PE

[0094] In the above formula, im represents an image of [2000, 2000]; X is an image of [500, 500, 16] obtained after partitioning, wherein [500, 500] represents spatial information, and 16 represents feature information (16 small blocks are divided out); b1 and b2 represent the size of the partitioning.

[0095] The position embedding feature PE is added to the above partitioned data X to obtain an intermediate processing image of [500, 500, 16], but 2 / 3 of the intermediate processing image is a blank area, and the ROI image feature of [256, 320, 16] is selected according to the foreground non-blank area to obtain a 2D real projection frontal and lateral X-ray image of [256, 320, 16] as the input of the neural network model, thereby reducing the size of the 2D real projection frontal and lateral X-ray image and retaining the effective area in the 2D real projection frontal and lateral X-ray image, and reducing the calculation amount.

[0096] In specific implementation, the random mask method comprises:

[0097] According to the random cube mask and projection method, the 2D real projection frontal and lateral X-ray image is enhanced and augmented to improve the robustness of the neural network model.

[0098] In specific implementation, the random cube mask and projection method comprises:

[0099] According to the preset region size parameter, a cube region is randomly selected in the 3D CT image, and the pixel values outside the cube region in the 3D CT image are set to zero to obtain a 3D CT reserved region image;

[0100] The 3D CT reserved region image is projected onto a two-dimensional plane to obtain a 2D local mapping image.

[0101] It should be noted that if only the 2D real projection frontal and lateral X-ray image is used as the input of the neural network model, then the trained neural network model can only generate a 3D segmentation image according to the complete 2D real projection frontal and lateral X-ray image. Once a part of the region in the 2D frontal and lateral X-ray image is randomly selected and input into the neural network model, the output 3D segmentation result will lose accuracy and cause overfitting problem, so it is necessary to enhance and augment the input data through the random cube mask and projection method in the batch processing process of neural network training, and the specific implementation process is as follows:

[0102] A cube region P of [32, 64, 128, 256, 512] is randomly selected in the 3D CT image cube , and it is assumed that P cubeFor an 8x4 matrix, 8 represents the 8 vertex positions of the cubical region, and 4 represents the 3D coordinate information of each row as [x, y, z, 1]. The pixel values of the regions outside the cubical region are all set to zero, ensuring that only the information in the selected cubical region is retained, and the information of other regions is discarded, obtaining a 3D CT retained region image.

[0103] The selected cubical region P cube is projected to the corresponding frontal and lateral views using the frontal-lateral affine matrix A, where A is a 4x6 matrix, meaning that a random cubical region in three-dimensional space is projected onto a 2D plane, and the calculation process of the projection is implemented according to the following formula:

[0104] P rect = A·P cube

[0105] P cor_min = P rect [all, 0, 1].min()

[0106] P cor_max = P rect [all, 0, 1].max()

[0107] P sag_min = P rect [all, 3, 4].min()

[0108] P sag_max = P rect [all, 3, 4].max()

[0109] In the above formula, P rect is the projection of the 3D vertex coordinates on the frontal and lateral views, where each row is [P cor_x , P cor_y , 1, P sag_x , P sag_y ], P cor_x and P cor_y represent the xy coordinates of the frontal 2D points, and P sag_x and P sag_y represent the xy coordinates of the lateral 2D points; P cor_min is [w min , h min ] representing the upper left corner of the frontal projection, P cor_max is [w max , h max ] representing the right lower corner of the frontal projection, P sag_min is [w min , h min ] representing the upper left corner of the lateral projection, and P sag_max is [w max , hmax ] represents the lower right corner of the lateral projection, based on P cor_min and P cor_max Get the positive projection rectangle based on P sag_min and P sag_max Get the side projection rectangle and obtain the 2D reserved area image.

[0110] In specific practice, the method for calculating and aligning 3D bone segmentation images includes:

[0111] A mapping relationship between the 2D real projection anteroposterior and lateral X-ray images and the original 3D bone segmentation image is represented by a real affine matrix, wherein the real affine matrix is ​​calculated based on the real projection parameters;

[0112] A mapping relationship between the 2D fixed-projection anteroposterior and lateral X-ray images and the aligned 3D bone segmentation images is represented by a fixed affine matrix, wherein the fixed affine matrix is ​​calculated based on fixed projection parameters;

[0113] According to the real affine matrix and the fixed affine matrix, a mapping relationship between the original 3D bone segmentation image and the aligned 3D bone segmentation image is constructed, thereby converting the change of the real projection parameters into the change of the pose and morphology of the photographed object under the fixed projection parameters.

[0114] The calculation process of the above-mentioned aligned 3D bone segmentation image is as follows:

[0115] Let the coordinates P of the 3D point set in the original 3D bone segmentation image be org It is an n×4 matrix, where n represents the number of 3D points and each row of the matrix contains the coordinates of a 3D point [P x ,P y ,P z ,1], the real affine matrix is ​​A α (A α It is a 4×6 matrix that contains information about the distance between the sensor and the tube, the distance between the sensor and the patient, the center of focus, and pixel resolution.

[0116] According to the formula P I =A α ·P org , calculate P I (P I is a 3D point set through A α Projection, the coordinates of the 2D point set in the obtained 2D frontal and lateral projection images), according to the matrix operation rules, P I is an n×6 matrix;

[0117] Let the fixed affine matrix be A β , according to P I and A β It can be obtained from the unknown 3D point set Paligned The aligned 3D bone segmentation image is obtained, that is, the formula P I = A β ·P aligned (indicating that the 2D projection images of the aligned 3D bone segmentation image and the original 3D bone segmentation image are the same), and further derivation can obtain, according to the formula The coordinates P aligned of the unknown 3D point set can be obtained.

[0118] After P aligned is obtained, P org can also be obtained according to the formula The inverses of the above matrices are pseudo inverses;

[0119] The transformation formula of the segmentation image is S aligned [P aligned ] = S org [P org ], wherein S org is the original 3D bone segmentation image, and S aligned is the aligned 3D bone segmentation image.

[0120] The above calculation process actually converts the calculation of the 3D point set P org and the real affine matrix into the calculation of the fixed affine matrix and the unknown 3D point set P aligned .

[0121] In order to more intuitively represent the effect of the above calculation process, Figure 8 (a) contains the original 3D bone segmentation image and the aligned 3D bone segmentation image of the same 3D CT image, and the original 3D bone segmentation image in (a) is projected onto a 2D plane by a real affine matrix to obtain a 2D frontal-lateral projection Figure 8 1, and the aligned 3D bone segmentation image in (a) is projected onto a 2D plane by a fixed affine matrix to obtain a 2D frontal-lateral projection Figure X 2, X1 and X2 are the same, as shown in Figure 8 (b) and Figure X (c), that is, the change of the X-ray shooting parameter is converted into the adjustment of the shooting posture of the simulated photographed patient, more specifically, the X-ray shooting parameter is fixed, and the corresponding relationship between the 2D projection image and the aligned 3D bone segmentation image is directly established, and this conversion will play a key role in the subsequent training of the model. Figure 8 Figure 8 In specific practice, the method of the DRR projection comprises:

[0122] Different combinations of real projection parameters are sampled to obtain 2D real projection frontal-lateral X-ray images.

[0123] Different combinations of real projection parameters are sampled to obtain 2D real projection frontal-lateral X-ray images. ​

[0124] The process of projecting the 3D CT image according to the X-ray shooting parameters is as follows:

[0125] The three degrees of freedom of the isocenter and one degree of freedom of the detector distance are respectively parameterized;

[0126] The grid method (a commonly used sampling technique) is used to sample each degree of freedom 8 times, obtaining 84 parameter combinations;

[0127] For the same 3D CT image, 1294 groups of 2D frontal and lateral X-ray images are obtained by mapping according to the 84 parameter combinations, wherein each group of 2D frontal and lateral X-ray images includes one 2D frontal X-ray image and one 2D lateral X-ray image (the two images can show the morphological characteristics of the lesion at different angles).

[0128] Embodiment Two

[0129] Figure 9 is a structural schematic diagram of a 2D-to-3D image reconstruction device according to another exemplary embodiment, as shown in Figure 9 The 2D-to-3D image reconstruction device includes:

[0130] a standardization module configured to standardize the real projection parameters of the 3D CT image to reduce the degrees of freedom of the mapping relationship between the 3D position information and the 2D position information;

[0131] a projection module configured to perform DRR projection on the 3D CT image according to the real projection parameters to obtain 2D real projection frontal and lateral X-ray images;

[0132] a position encoding module configured to perform position encoding on the 2D real projection frontal and lateral X-ray images to retain the global position information of the 2D real projection frontal and lateral X-ray images;

[0133] an enhancement module configured to obtain 3D CT reserved region images and 2D local mapping images of the 2D real projection frontal and lateral X-ray images according to a random mask method to improve the diversity of image data;

[0134] The computing module is configured to acquire an original 3D bone segmentation image corresponding to the 3D CT image, and calculate an aligned 3D bone segmentation image according to the original 3D bone segmentation image, the real projection parameter and the fixed projection parameter, so that a 2D real projection anteroposterior X-ray image obtained by mapping the original 3D bone segmentation image through the real projection parameter is the same as a 2D fixed projection anteroposterior X-ray image obtained by mapping the aligned 3D bone segmentation image through the fixed projection parameter.

[0135] The training module is configured to, in a training stage, train a pre-constructed neural network model by taking the 2D real projection anteroposterior X-ray image as input, taking the aligned 3D bone segmentation image as true value, and / or taking the 2D local mapping image as input and taking the 3D CT reserved region image as true value, so that the neural network model learns the corresponding relationship between the 2D real projection anteroposterior X-ray image and the aligned 3D bone segmentation image, and / or so that the neural network model learns the corresponding relationship between the 2D local mapping image and the 3D CT reserved region image, until the model converges, to obtain a three-dimensional reconstruction model.

[0136] The prediction module is configured to, in a prediction stage, input the 2D real projection anteroposterior X-ray image into the three-dimensional reconstruction model to obtain a predicted 3D aligned bone segmentation image, and obtain a real 3D bone segmentation image by calculating an inverse of a matrix composed of the real projection parameter and the fixed projection parameter according to the predicted 3D aligned bone segmentation image.

[0137] The technical scheme provided by the embodiment is characterized in that the acquisition module, the sampling module, the computing module, the training module and the prediction module are arranged in the 3D image reconstruction device, the aligned 3D bone segmentation image is calculated according to the original 3D bone segmentation image and the real projection parameter, so that the 2D fixed parameter projection image of the aligned 3D bone segmentation image is the same as the real projection X-ray image, the 2D anteroposterior X-ray image is taken as input and the aligned 3D bone segmentation image is taken as true value, the pre-constructed neural network model is trained, so that the neural network model learns the corresponding relationship between the 2D anteroposterior X-ray image and the aligned 3D bone segmentation image, thereby converting the change of the X-ray shooting parameter into the change of the object pose, avoiding the influence of the change of the X-ray shooting parameter on the 3D image segmentation effect, improving the accuracy of the generated 3D segmentation image, and overcoming the problem that the model training needs to input the X-ray shooting parameter, so as to improve the generation accuracy.

[0138] Embodiment three

[0139] Figure 10 is a structural schematic diagram of an electronic device according to another exemplary embodiment, as shown in Figure 10 The electronic device includes:

[0140] a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface, the memory and the communication bus are capable of communicating with each other through the communication bus;

[0141] a memory for storing a computer program;

[0142] a processor for executing the program stored in the memory to implement the method.

[0143] The technical scheme provided by the embodiment reduces the degrees of freedom of the mapping relationship between the 3D position information and the 2D position information by standardizing the real projection parameters of the 3D CT image, reduces the sampling data amount of the 2D real projection anteroposterior X-ray image, and improves the accuracy of the 2D real projection anteroposterior X-ray image. The global position information of the 2D real projection anteroposterior X-ray image is retained by position encoding the 2D real projection anteroposterior X-ray image, and the target region in the 2D real projection anteroposterior X-ray image is extracted. The 3D CT reserved region image and the 2D local mapping image of the 2D real projection anteroposterior X-ray image are obtained according to the random mask method, the image data diversity is improved, and the model is unable to learn the local mapping relationship from 2D to 3D. The original 3D bone segmentation image corresponding to the 3D CT image is obtained, and the aligned 3D bone segmentation image is calculated according to the original 3D bone segmentation image, the real projection parameters and the fixed projection parameters, so that the 2D real projection anteroposterior X-ray image obtained by mapping the original 3D bone segmentation image through the real projection parameters is the same as the 2D fixed projection anteroposterior X-ray image obtained by mapping the aligned 3D bone segmentation image through the fixed projection parameters, so that the change of the real projection parameters is converted into the change of the pose and morphology of the photographed object under the fixed projection parameters. In the training stage, the 2D real projection anteroposterior X-ray image is taken as the input, the aligned 3D bone segmentation image is taken as the true value, and / or the 2D local mapping image is taken as the input, and the 3D CT reserved region image is taken as the true value, and the pre-constructed neural network model is trained, so that the neural network model learns the corresponding relationship between the 2D real projection anteroposterior X-ray image and the aligned 3D bone segmentation image, and / or the neural network model learns the corresponding relationship between the 2D local mapping image and the 3D CT reserved region image, until the model converges, and a three-dimensional reconstruction model is obtained. In the prediction stage, the 2D real projection anteroposterior X-ray image is input into the three-dimensional reconstruction model to obtain a predicted 3D aligned bone segmentation image, and the real 3D bone segmentation image is obtained by calculating the inverse of the matrix composed of the real projection parameters and the fixed projection parameters according to the predicted 3D aligned bone segmentation image, thereby improving the accuracy of model prediction.

[0144] Embodiment Four

[0145] According to another exemplary embodiment, a non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute the method is shown.

[0146] The technical scheme provided by the embodiment reduces the degree of freedom of the mapping relationship between the 3D position information and the 2D position information by standardizing the real projection parameters of the 3D CT image, reduces the sampling data amount of the 2D real projection anteroposterior X-ray image, and improves the accuracy of the 2D real projection anteroposterior X-ray image. The global position information of the 2D real projection anteroposterior X-ray image is retained by position encoding the 2D real projection anteroposterior X-ray image, and the target region in the 2D real projection anteroposterior X-ray image is extracted. The 3D CT reserved region image and the 2D local mapping image of the 2D real projection anteroposterior X-ray image are obtained according to the random mask method, the image data diversity is improved, and the model fails to learn the local mapping relationship from 2D to 3D. The original 3D bone segmentation image corresponding to the 3D CT image is obtained, and the aligned 3D bone segmentation image is calculated according to the original 3D bone segmentation image, the real projection parameters, and the fixed projection parameters, so that the 2D real projection anteroposterior X-ray image obtained by mapping the original 3D bone segmentation image through the real projection parameters is the same as the 2D fixed projection anteroposterior X-ray image obtained by mapping the aligned 3D bone segmentation image through the fixed projection parameters, thereby converting the change of the real projection parameters into the change of the pose and morphology of the photographed object under the fixed projection parameters. In the training stage, the 2D real projection anteroposterior X-ray image is taken as the input, the aligned 3D bone segmentation image is taken as the true value, and / or the 2D local mapping image is taken as the input, and the 3D CT reserved region image is taken as the true value, and a pre-constructed neural network model is trained, so that the neural network model learns the corresponding relationship between the 2D real projection anteroposterior X-ray image and the aligned 3D bone segmentation image, and / or the neural network model learns the corresponding relationship between the 2D local mapping image and the 3D CT reserved region image, until the model converges, and a three-dimensional reconstruction model is obtained. In the prediction stage, the 2D real projection anteroposterior X-ray image is input into the three-dimensional reconstruction model to obtain a predicted 3D aligned bone segmentation image, and the real 3D bone segmentation image is obtained by calculating the inverse of the matrix composed of the real projection parameters and the fixed projection parameters according to the predicted 3D aligned bone segmentation image, thereby improving the accuracy of model prediction.

[0147] The above sequence numbers of the embodiments of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0148] The integrated units in the above embodiments, if implemented in the form of software function units and sold or used as independent products, can be stored in the above computer-readable storage medium. Based on such understanding, the technical solutions of the present application essentially or say the parts that make contributions to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing one or more computer devices (which can be personal computers, servers or network devices, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application.

[0149] In the above embodiments of the present application, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0150] In the several embodiments provided by the present application, it should be understood that the disclosed client can be implemented in other ways. Of course, the above device embodiment is only illustrative, and for example, the division of the units is only a logical function division, and there can be another division manner in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, unit or module, and can be electrical or other forms.

[0151] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., they can be located in one place or distributed on multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiment according to actual needs.

[0152] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The above integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0153] The above is only the preferred embodiment of the present application, and it should be pointed out that for those skilled in the art, without departing from the principle of the present application, a number of improvements and refinements can be made, which should be considered as the protection scope of the present application.

Claims

1. A 2D to 3D image reconstruction method, characterized in that: The 2D to 3D image reconstruction method comprises: Normalize the real projection parameters of 3D CT images; Performing DRR projection on the 3D CT image according to the real projection parameters to obtain a 2D real projection frontal and lateral X-ray image; Position encoding is performed on the 2D real projection anteroposterior and lateral X-ray images; Acquire a 3D CT-reserved region image and a 2D local mapping image of the 2D true projection anteroposterior and lateral X-ray images according to a random mask method; Obtaining an original 3D bone segmentation image corresponding to the 3D CT image, and calculating an aligned 3D bone segmentation image based on the original 3D bone segmentation image, the real projection parameters, and the fixed projection parameters, so that a 2D real projection anteroposterior and lateral X-ray image obtained by mapping the original 3D bone segmentation image through the real projection parameters is identical to a 2D fixed projection anteroposterior and lateral X-ray image obtained by mapping the aligned 3D bone segmentation image through the fixed projection parameters; In the training stage, the 2D real projected anteroposterior and lateral X-ray images are used as input, the aligned 3D bone segmentation images are used as true values, and / or the 2D local mapping images are used as input, and the 3D CT reserved area images are used as true values, and a pre-constructed neural network model is trained, so that the neural network model learns the correspondence between the 2D real projected anteroposterior and lateral X-ray images and the aligned 3D bone segmentation images, and / or the neural network model learns the correspondence between the 2D local mapping images and the 3D CT reserved area images, until the model converges to obtain a three-dimensional reconstructed model; In the prediction stage, the 2D real projection frontal and lateral X-ray images are input into the three-dimensional reconstruction model to obtain a predicted 3D aligned bone segmentation image. Based on the predicted 3D aligned bone segmentation image, the real 3D bone segmentation image is obtained by calculating the inverse of the matrix composed of the real projection parameters and the fixed projection parameters.

2. The 2D to 3D image reconstruction method according to claim 1, characterized in that: The method for position encoding the 2D real projected frontal and lateral X-ray images comprises: Embed position feature information into 2D real projection frontal and lateral X-ray images; The spatial information of the 2D real projection anteroposterior and lateral X-ray images in which the position feature information is embedded is cropped, and the target area is extracted from the non-blank area of ​​the 2D real projection anteroposterior and lateral X-ray images.

3. The 2D to 3D image reconstruction method according to claim 2, characterized in that: The method for extracting the target area includes: According to the global spatial information, the position of the 2D real-projection frontal and lateral X-ray images is embedded, and then the 2D real-projection frontal and lateral X-ray images are cropped according to the target area to avoid direct cropping that will destroy the mapping relationship between the 2D position information and the 3D position information, and to reduce the input image size, thereby reducing the amount of calculation.

4. The 2D to 3D image reconstruction method according to claim 1, characterized in that: The random masking method includes: According to the random cube mask and projection method, the 2D real projection frontal and lateral X-ray images are enhanced and amplified to improve the robustness of the neural network model.

5. The 2D to 3D image reconstruction method according to claim 4, characterized in that: The random cube masking and projection method includes: randomly selecting a cubic region in the 3D CT image according to a preset region size parameter, and setting pixel values ​​in the 3D CT image except for the cubic region to zero, to obtain a 3D CT reserved region image; The 3D CT preserved area image is projected onto a two-dimensional plane to obtain a 2D local mapping image.

6. The 2D to 3D image reconstruction method according to claim 1, characterized in that: The method for calculating and aligning 3D bone segmentation images comprises: The mapping relationship between the 2D real projection frontal and lateral X-ray images and the original 3D bone segmentation image is represented by a real affine matrix, wherein the real affine matrix is ​​calculated according to the real projection parameters; The mapping relationship between the 2D fixed projection anteroposterior and lateral X-ray images and the aligned 3D bone segmentation images is represented by a fixed affine matrix, wherein the fixed affine matrix is ​​calculated based on fixed projection parameters; According to the real affine matrix and the fixed affine matrix, a mapping relationship between the original 3D bone segmentation image and the aligned 3D bone segmentation image is constructed, thereby converting the change of the real projection parameters into the change of the pose and morphology of the photographed object under the fixed projection parameters.

7. The 2D to 3D image reconstruction method according to claim 1, characterized in that: The DRR projection method includes: The real projection parameters of different combinations are sampled to obtain 2D real projection frontal and lateral X-ray images.

8. A 2D to 3D image reconstruction apparatus, comprising: A normalization module, for normalizing the real projection parameters of the 3D CT image to reduce the degree of freedom of the mapping relationship between the 3D position information and the 2D position information; A projection module, configured to perform DRR projection on the 3D CT image according to the real projection parameters to obtain a 2D real projection frontal and lateral X-ray image; a position encoding module, configured to perform position encoding on the 2D real projection anteroposterior and lateral X-ray images so as to retain global position information of the 2D real projection anteroposterior and lateral X-ray images; an enhancement module, configured to obtain a 3D CT reserved region image and a 2D local mapping image of the 2D real projection anteroposterior and lateral X-ray images according to a random mask method, so as to improve image data diversity; a calculation module, configured to obtain an original 3D bone segmentation image corresponding to the 3D CT image, and calculate an aligned 3D bone segmentation image based on the original 3D bone segmentation image, real projection parameters, and fixed projection parameters, so that a 2D real projection anteroposterior and lateral X-ray image obtained by mapping the original 3D bone segmentation image through the real projection parameters is identical to a 2D fixed projection anteroposterior and lateral X-ray image obtained by mapping the aligned 3D bone segmentation image through the fixed projection parameters; A training module is configured to, during a training phase, use the 2D real-projection anteroposterior and lateral X-ray images as input, the aligned 3D bone segmentation images as true values, and / or use the 2D local mapping images as input, the 3D CT reserved area images as true values, to train a pre-constructed neural network model, so that the neural network model learns the correspondence between the 2D real-projection anteroposterior and lateral X-ray images and the aligned 3D bone segmentation images, and / or the correspondence between the 2D local mapping images and the 3D CT reserved area images, until the model converges to obtain a three-dimensional reconstructed model; The prediction module is used to input the 2D real projection frontal and lateral X-ray images into the three-dimensional reconstruction model in the prediction stage to obtain a predicted 3D aligned bone segmentation image, and obtain a real 3D bone segmentation image based on the predicted 3D aligned bone segmentation image by calculating the inverse of the matrix composed of real projection parameters and fixed projection parameters.

9. An electronic device, characterized in that: include: A processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus; Memory for storing computer programs; A processor, configured to implement the method according to any one of claims 1 to 7 when executing a program stored in a memory.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 7.

Images