Three-dimensional reconstruction method and system based on multi-modal two-dimensional image

Abstract

The invention discloses a three-dimensional reconstruction method and system based on a multi-modal two-dimensional image, and the method comprises the steps: obtaining imaging system parameters of CT equipment and original CT three-dimensional data, and constructing an X-ray projection multi-modal data set; the data set is preprocessed, for each view angle, the position of a pixel sampling point is calculated through a double-domain collaborative sampling strategy, and Np pixel positions are sampled in total and correspond to Np sampling rays; constructing a segmented adaptive neural radiation field model MNeRF based on a probability feature coupler; training the model constructed in the step S3; and performing three-dimensional reconstruction on the new sampling view angle by using the trained model. According to the method, the deficiency of context information caused by random uniform sampling is avoided, meanwhile, the calculation amount can be reduced and the calculation efficiency can be improved through the segmented self-adaptive strategy, and three-dimensional reconstruction of a scene can be carried out by utilizing the two-dimensional X-ray image under the constraint of a sparse view angle, so that sufficient three-dimensional information can be obtained under low-dose X-rays.

Description

Technical Field

[0001] The present invention relates to the field of computer vision three-dimensional reconstruction, and in particular to a three-dimensional reconstruction method and system based on multimodal two-dimensional images. Background Art

[0002] With the rapid development of medical imaging technology, three-dimensional image reconstruction plays a vital role in clinical diagnosis, disease treatment planning, and surgical navigation. In traditional medical imaging, two-dimensional X-ray images are widely used in various examinations, such as chest radiographs, dental X-rays, and bone imaging. However, two-dimensional X-ray images themselves can only provide limited spatial information and cannot fully describe the three-dimensional morphology of the patient's internal structure. At present, three-dimensional image reconstruction methods mainly rely on technologies such as CT (computed tomography) and MRI (magnetic resonance imaging). These methods can provide high-quality three-dimensional images, but due to high imaging costs, high radiation doses, and complex equipment maintenance and operation, their widespread application in certain specific application scenarios is limited.

[0003] Existing X-ray 3D reconstruction methods can be divided into three categories: analytical methods, iterative methods, and learning-based methods. Analytical methods use mathematical inversion techniques to convert projected multimodal data into reconstructed images. They are suitable for situations with sufficient data, but perform poorly for sparse or limited-angle data. Iterative methods iteratively solve the error between projected multimodal data and reconstructed images, gradually approaching the true image. They perform well when dealing with ill-conditioned or poorly defined problems, and can suppress noise and artifacts through regularization to improve the quality of reconstructed images. However, iterative methods are usually computationally intensive, time-consuming, and have high memory requirements. Learning-based methods use deep neural networks to predict and extrapolate projection or regression target attenuation coefficients, reducing computational complexity, integrating the optimization process into a deep learning framework, and simultaneously learning the optimization strategies for projection and reconstruction in end-to-end training to improve reconstruction stability and image quality. However, the reconstruction results of existing methods still face challenges such as visual degradation, severe streak artifacts, and low peak signal-to-noise ratio.

[0004] Therefore, in order to solve the above technical problems, providing a method and system with better imaging quality, which can retain better detail information and obtain more sufficient three-dimensional information under low-dose X-rays is an urgent problem to be solved in the current medical imaging field. Summary of the Invention

[0005] In view of this, the present invention provides a three-dimensional reconstruction method and system based on multimodal two-dimensional images.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0007] A three-dimensional reconstruction method and system based on multimodal two-dimensional images includes the following steps:

[0008] Step S1: Obtain imaging system parameters of the CT device and original CT three-dimensional data, and construct an X-ray projection multimodal data set;

[0009] Step S2: Preprocess the data set. For each viewing angle, calculate the pixel sampling point position through the dual-domain collaborative sampling strategy, and sample N pixels in total. p pixel positions, corresponding to N p Sampling rays;

[0010] Step S3: Construct a piecewise adaptive neural radiation field model M based on probabilistic characteristic coupler NeRF ;

[0011] Step S4: using the acquired CT device imaging system parameters and projection multimodal data set as training data to train the model constructed in step S3;

[0012] Step S5: Using the trained model M NeRF Perform 3D reconstruction of the new sampled viewpoint.

[0013] Preferably, step S1 includes downloading a CT dataset from an open source dataset official website to obtain original CT three-dimensional data, i.e., two-dimensional projection multimodal data at equiangular ray intervals;

[0014] Preferably, for the N sampled in step S2 p rays, sampling N on each ray s The radiation density corresponding to each sampling position is predicted and output, and the reconstructed image under the corresponding perspective is obtained by integral rendering calculation along the ray direction. The loss is calculated by comparing with the real image, and the gradient is calculated by backpropagation to optimize the weight parameters of the updated model.

[0015] Preferably, step S2 includes the following steps:

[0016] Step S2.1: First, the two-dimensional projection image at each viewing angle According to the threshold Perform foreground and background segmentation to obtain segmentation mask That is, M = 1 I>T ;

[0017] Step S2.2: Evenly divide the segmentation mask M into non-overlapping blocks of size S×S. blocks, and the set of blocks is represented as Among them, Q f Represents a block containing only foreground information, i.e. Q f ={q∈Q|q=1 S×S};

[0018] Step S2.3: Perform dual-domain collaborative sampling, first perform block domain local sampling, from Q f Randomly select N l blocks, forming a set Then N l The X-ray set corresponding to each block is Among them, Ray(p) represents mapping the pixel p of the projection image to the corresponding X-ray, and the corresponding sampling point set is P l ={p∈W l |p=1};

[0019] Step S2.4: Perform full-domain sampling, starting from Q l Randomly extract N foreground areas outside g pixel positions, and obtain the pixel set P g ={p∈(QQ l )|p=1}, and its corresponding X-ray set is

[0020] Step S2.5: Merge the X-ray sets of block-domain local sampling and global sampling R = R l ∪R g , get the total sampled ray set R and the sampling pixel point set P corresponding to the ray set = P l ∪P g , the total number of sampling pixels is N=N l ∪N g .

[0021] Preferably, step S3 includes the following steps:

[0022] Step S3.1: Construct a multi-scale feature aggregation unit. For the N sampling points obtained in step S2, input their spatial position coordinates. By mapping the multi-scale hash function to a high-dimensional space, feature encoding at multiple scales is obtained. C represents the feature dimension of the encoding;

[0023] Step S3.2: First, construct the probabilistic feature coupler M C , including feature coupler E, latent variable layer H, and probability generator D. Its training process is: the feature coupler E encodes the multi-scale feature F obtained in step S3.1 to obtain F E , F E =E(F), the latent variable layer H encodes the features obtained by the feature coupler E into latent variables that conform to the standard Gaussian distribution, and the probability generator D reconstructs the input feature data from the latent variables. The parameters of the probability feature coupler are updated through the reconstruction loss training to complete M Ctraining;

[0024] Step S3.3: Then construct the segmented adaptive heterogeneous attention module M Atten , fix the weight parameters of the characteristic coupler E, and set M C The characteristic of the output of coupler E is F E Spliced ​​to the segmented adaptive attention module M Atten The jump connection layer in the probabilistic feature coupler M C The feature coupler E and the segmentation-adaptive heterogeneous attention module M Atten Together they form a piecewise adaptive neural radiation field model M based on probabilistic characteristic coupler. NeRF .

[0025] Preferably, step S4 includes the following steps:

[0026] Step S4.1: For the N samples sampled in step S2 p pixel position, the corresponding ray is o is the origin, d is the direction, and N samples are taken on each ray. s There are N spatial position points in total. p ×N s The spatial position sampling points are input into the position features of the obtained sampling points into the M constructed in step S3. NeRF , predict and output the radiation density ρ corresponding to each position;

[0027] Step S4.2: According to the rendering formula The radiation intensity under the projection of the viewing angle predicted by the model is obtained, where I o represents the initial strength, ρ i =||P i+1 -P i || represents the distance between adjacent points;

[0028] Step S4.3: Compare the predicted radiation intensity under the projection angle with the actual projection intensity I gt (r) Compare and calculate the mean square error loss in, Represents a batch of rays sampled during training, and uses the mean square error loss to iteratively optimize the weight parameters of the model using the gradient descent method.

[0029] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the three-dimensional reconstruction method and system based on multimodal two-dimensional images.

[0030] A computer program product includes a computer program, which, when executed by a processor, implements the three-dimensional reconstruction method and system based on multimodal two-dimensional images.

[0031] Compared with the prior art, the present invention has achieved the following technical effects:

[0032] (1) The present invention proposes a dual-domain collaborative sampling strategy to calculate the pixel sampling point position, which can make the sampling process more effective in focusing on the geometric and contextual information of the object, avoiding the loss of contextual information in random uniform sampling;

[0033] (2) The present invention uses a multi-scale feature aggregation unit to encode input position features, mapping spatial coordinates to the feature representation of the neural network. By encoding these coordinates at multiple scale levels, the model can efficiently represent details while preserving the overall scene structure;

[0034] (3) The present invention uses a segmented adaptive neural radiation field model based on a probabilistic feature coupler for feature extraction, and predicts and outputs density information under the corresponding viewing angle. Compared with the original attention structure, the segmented adaptive heterogeneous attention module can reduce the amount of calculation and improve the calculation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is a flow chart of the method of the present invention;

[0036] Figure 2 It is a structural diagram of the present invention;

[0037] Figure 3 Schematic diagram of the sampling strategy of the present invention. DETAILED DESCRIPTION

[0038] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0039] like Figure 1 As shown, the present invention discloses a three-dimensional reconstruction method and system based on multimodal two-dimensional images;

[0040] The following steps are involved:

[0041] S1: Obtain the imaging system parameters and original CT 3D data of the CT equipment and construct an X-ray projection multimodal dataset;

[0042] S2: Preprocess the data. For each viewing angle, calculate the pixel sampling point position through the dual-domain collaborative sampling strategy, and sample N pixels in total. p pixel positions, corresponding to N p Sampling rays;

[0043] Step S2 is specifically as follows:

[0044] Step S2.1: First, the two-dimensional projection image at each viewing angle According to the threshold Perform foreground and background segmentation to obtain segmentation mask That is, M = 1 I>T .

[0045] Step S2.2: Evenly divide the segmentation mask M into non-overlapping blocks of size S×S. blocks, and the set of blocks is represented as Among them, Q f Represents a block containing only foreground information, i.e. Q f ={q∈Q|q=1 s×s}.

[0046] Step S2.3: Perform dual-domain collaborative sampling, first perform block domain local sampling, from Q f Randomly select N l blocks, forming a set Q l ={q1,q,…q}, then N l The X-ray set corresponding to each block is Ray(p) represents mapping the pixel p of the projection image to the corresponding X-ray, and the corresponding sampling point set is P l ={p∈W l |p=1}.

[0047] Step S2.4: Perform full-domain sampling, starting from Q l Randomly extract N foreground areas outside g pixel positions, and obtain the pixel set P g ={p∈(MW l )|p=1}, and its corresponding X-ray set is

[0048] Step S2.5: Merge the X-ray sets of block-domain sampling and global sampling R = R l ∪R g , get the total sampled ray set R and the sampling pixel point set P corresponding to the ray set = P l ∪P g , the total number of sampling pixels is N=N l ∪N g.

[0049] S3. Construct a segmented adaptive neural radiation field model M based on probabilistic characteristic coupler NeRF ;

[0050] Step S3 is specifically as follows:

[0051] Step S3.1: Construct a multi-scale feature aggregation unit. For the N sampling points obtained in step S2, input their spatial position coordinates. By mapping the multi-scale hash function to a high-dimensional space, feature encoding at multiple scales is obtained. C represents the feature dimension of the encoding.

[0052] Step S3.2: First construct the probabilistic feature coupler M C , including feature coupler E, latent variable layer H, and probability generator D. Its training process is: the feature coupler E encodes the multi-scale feature F obtained in step S3.1 to obtain F E , F E =E(F), the latent variable layer H encodes the encoded features obtained by the feature coupler E into latent variables that conform to the Gaussian distribution, and the probability generator D reconstructs the input feature data from the latent variables. The parameters of the probability feature coupler are updated through the reconstruction loss training to complete M C training.

[0053] Step S3.3: Then construct the segmented adaptive heterogeneous attention module M Atten , fix the weight parameters of the characteristic coupler E, and set M C The characteristic of the output of coupler E is F E Spliced ​​to the segmented adaptive heterogeneous attention module M Atten The jump connection layer in the probabilistic feature coupler M C The feature coupler E and the segmentation-adaptive heterogeneous attention module M Atten Together they form a piecewise adaptive neural radiation field model M based on probabilistic characteristic coupler. NeRF .

[0054] S4, using the acquired CT equipment imaging system parameters and projection multimodal data set as training data, to train the model constructed in step S3. p rays, sampling N on each ray s The radiation density corresponding to each sampling position is predicted and output, and the reconstructed image under the corresponding perspective is obtained by integral rendering calculation along the ray direction. The loss is calculated by comparing with the real image, and the gradient is calculated by backpropagation to optimize the weight parameters of the updated model.

[0055] Step S4 is specifically as follows:

[0056] Step S4.1: For the N sampled in step S2 p pixel position, the corresponding ray is o is the origin, d is the direction, and N samples are taken on each ray. s There are N spatial position points in total. p ×N s The spatial position sampling points are input into the obtained position features of the sampling points into the segmented adaptive neural radiation field model M based on the probability feature coupler constructed in step S3. NeRF , predict and output the radiation density ρ corresponding to each position.

[0057] Step S4.2: According to the rendering formula The radiation intensity under the projection of the viewing angle predicted by the model is obtained, where I o represents the initial strength, ρ i =||P i+1 -P i || represents the distance between adjacent points.

[0058] Step S4.3: Compare the predicted radiation intensity under the projection angle with the actual projection intensity I gt (r) Compare and calculate the mean square error loss in Represents a batch of rays sampled during training, and uses the mean square error loss to iteratively optimize the weight parameters of the model using the gradient descent method.

[0059] S5. Use the trained segmented adaptive neural radiation field model based on probabilistic feature coupler to perform three-dimensional reconstruction of the new sampling perspective.

[0060] An embodiment of the present invention further provides a storage medium for storing a computer program, which at least performs the above method when executed.

[0061] An embodiment of the present invention further provides a control device, comprising a processor and a storage medium for storing a computer program; wherein the processor is configured to execute at least the method described above when executing the computer program.

[0062] An embodiment of the present invention further provides a processor, which executes a computer program and at least performs the above method.

[0063] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a magnetic disk memory or a magnetic tape memory. The storage medium described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0064] In the implementation examples provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0065] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0066] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0067] Those skilled in the art will understand that all or part of the steps of the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc. Various media that can store program codes.

[0068] The methods disclosed in the several method embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0069] The features disclosed in several product embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0070] The features disclosed in several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0071] The above description is merely a preferred embodiment of the present invention and does not limit the technical scope of the present invention. Therefore, any minor modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A three-dimensional reconstruction method and system based on multimodal two-dimensional images, characterized in that: The steps include: Step S1: Obtain imaging system parameters of the CT device and original CT three-dimensional data, and construct an X-ray projection multimodal data set; Step S2: Preprocess the multimodal dataset. For each viewing angle, calculate the pixel sampling point position through the dual-domain collaborative sampling strategy, and sample N pixels in total. p pixel positions, corresponding to N p Sampling rays; Step S3: Construct a piecewise adaptive neural radiation field model M based on probabilistic characteristic coupler NeRF ; Step S4: Using the acquired CT imaging system parameters and projection multimodal dataset as training data, train the model M constructed in step S3. NeRF ; Step S5: Using the trained model M NeRF Perform 3D reconstruction of the new sampled viewpoint.

2. The method and system for 3D reconstruction based on multimodal 2D images according to claim 1, characterized in that: The step S1 includes downloading a CT dataset from an open source dataset official website to obtain original CT three-dimensional data, namely, two-dimensional projection multimodal data at equiangular ray intervals.

3. The method and system for 3D reconstruction based on multimodal 2D images according to claim 1, characterized in that: For the N sampled in step S2 p rays, sampling N on each ray s The radiation density corresponding to each sampling position is predicted and output, and the reconstructed image under the corresponding perspective is obtained by integral rendering calculation along the ray direction. The loss is calculated by comparing with the real image, and the gradient is calculated by backpropagation to optimize and update the weight parameters of the model.

4. The method and system for 3D reconstruction based on multimodal 2D images according to claim 1, characterized in that: The step S2 comprises the following steps: Step S2.1: First, the two-dimensional projection image at each viewing angle According to the threshold Perform foreground and background segmentation to obtain segmentation mask That is, M = 1 I>T ; Step S2.2: Evenly divide the segmentation mask M into non-overlapping blocks of size S×S. blocks, and the set of blocks is represented as Among them, Q f Represents a block containing only foreground information, i.e. Q f ={q∈Q|q=1 s×s }; Step S2.3: Perform dual-domain collaborative sampling, first perform block domain local sampling, from Q f Randomly select N l blocks, forming a set Then N l The X-ray set corresponding to each block is Among them, Ray(p) represents mapping the pixel p of the projection image to the corresponding X-ray, and the corresponding sampling point set is P l ={p∈Q l |p=1}; Step S2.4: Perform full-domain sampling, starting from Q l Randomly extract N foreground areas outside g pixel positions, and obtain the pixel set P g ={p∈(QQ l )|p=1}, and its corresponding X-ray set is Step S2.5: Merge the block-domain and global-domain sampled X-ray sets R = R l ∪R g , get the total sampled ray set R and the sampling pixel point set P corresponding to the ray set = P l ∪P g , the total number of sampling pixels is N=N l ∪N g .

5. The 3D reconstruction method and system based on multimodal 2D images according to claim 1, characterized in that: The step S3 comprises the following steps: Step S3.1: Construct a multi-scale feature aggregation unit. For the N sampling points obtained in step S2, input their spatial position coordinates. By mapping the multi-scale hash function to a high-dimensional space, feature encoding at multiple scales is obtained. C represents the feature dimension of the encoding; Step S3.2: Construct the probabilistic feature coupler M c , including feature coupler E, latent variable layer H, and probability generator D. Its training process is: the feature coupler E encodes the multi-scale feature F obtained in step S3.1 to obtain F E , F E =E(F), the latent variable layer H encodes the features obtained by the encoder E into latent variables that conform to the standard Gaussian distribution. The probability generator D reconstructs the input feature data from the latent variables and updates the parameters of the probability feature coupler through the reconstruction loss training to complete the probability feature coupler M c training; Step S3.3: Then construct the segmented adaptive heterogeneous attention module M Atten , fix the weight parameters of the characteristic coupler E, and set M c The characteristic of the output of coupler E is F E Spliced ​​to the segmented adaptive heterogeneous attention module M Atten The jump connection layer in the feature coupler M c The encoder E and the segment-adaptive heterogeneous attention module M Atten Together they form a piecewise adaptive neural radiation field model M based on probabilistic characteristic coupler. NeRF .

6. The method and system for 3D reconstruction based on multimodal 2D images according to claim 1, characterized in that: The step S4 comprises the following steps: Step S4.1: For the N samples sampled in step S2 p pixel position, the corresponding ray is o is the origin, d is the direction, and N samples are taken on each ray. s There are N spatial position points in total. p ×N s The spatial position sampling points are input into the model M constructed in step S3. NeRF , predict and output the radiation density ρ corresponding to each position; Step S4.2: According to the rendering formula The radiation intensity under the projection of the viewing angle predicted by the model is obtained, where I o represents the initial strength, ρ i =||P i+1 -P i || represents the distance between adjacent points; Step S4.3: Compare the predicted radiation intensity under the projection angle with the actual projection intensity I gt (r) Compare and calculate the mean square error loss in, Represents a batch of rays sampled during training, and uses the mean square error loss to iteratively optimize the weight parameters of the model using the gradient descent method.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the three-dimensional reconstruction method and system based on multimodal two-dimensional images as described in any one of claims 1 to 6 are implemented.

8. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the three-dimensional reconstruction method and system based on multimodal two-dimensional images as described in any one of claims 1 to 6 are implemented.

Images