3D Reconstruction Method, System, Electronic Device and Storage Medium for X-ray Images

Abstract

The present invention discloses an X-ray image three-dimensional reconstruction method, system, electronic device and storage medium. The method includes using two parallel encoder-decoder networks to perform three-dimensional reconstruction on the coronal plane and sagittal plane of two-dimensional X-ray images respectively. Among them, in the encoder network, a local feature enhancement module is used to extract two-dimensional feature maps of the coronal plane X-ray image and the sagittal plane X-ray image respectively; an attention interaction module is used to perform information exchange and fusion on the two-dimensional feature maps of the coronal plane X-ray image and the sagittal plane X-ray image; a dimension conversion unit is used to convert the two-dimensional feature map processed by the attention interaction module into a three-dimensional feature map; in the decoder network, a 3D transposed convolution module is used to obtain a three-dimensional image by gradually magnifying the three-dimensional feature map and performing three-dimensional convolution operations. The method of the present invention can complete the reconstruction of three-dimensional CT using only two X-ray pictures, reducing the radiation received by patients.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and particularly relates to a three-dimensional reconstruction method, system, electronic device and storage medium for X-ray images. Background Art

[0002] In clinical practice, techniques such as Computed Tomography (CT), Magnetic Resonance Imaging (MRI), and X-ray imaging are usually used to obtain medical images to further understand the conditions inside a patient's body. However, each of these three imaging techniques has its limitations. For example, CT images can provide detailed three-dimensional visual information of the internal structure of the human body, but patients need to bear a relatively high radiation dose, posing a safety hazard. In addition, the ionizing radiation used in the CT imaging process may have long-term health effects on the human body, especially for patients who need multiple examinations. MRI provides a non-invasive three-dimensional imaging method without ionizing radiation. However, due to its strong magnetic field characteristics, it cannot be applied to patients with metal implants. In addition, the MRI imaging process is time-consuming and the equipment is expensive, which limits its popular application in some medical institutions. At the same time, both CT and MRI imaging require patients to maintain a lying position, which results in an inconsistent load-bearing situation of human organs with the clinical diagnosis requirements and affects the accuracy of certain diagnoses.

[0003] In contrast, X-ray imaging is the most commonly used clinical modality due to its low cost, low radiation dose, and the ability to image standing patients. Clinicians usually obtain biplane X-ray images of patients through coronal and sagittal plane views to diagnose various internal diseases. However, the obvious drawback of X-ray imaging is that it can only obtain two-dimensional projections of three-dimensional objects, providing only limited two-dimensional information and unable to fully present three-dimensional images of the detailed internal structure of the human body. This drawback limits clinical evaluation and treatment options, and may lead to unstable and error-prone clinical decisions. For example, when faced with complex fractures or internal organ injuries, doctors may have difficulty accurately judging the degree and location of the injury based solely on two-dimensional X-ray images, thus affecting the treatment effect.

[0004] To solve this problem, many researchers have dedicated themselves to combining new technologies to achieve more effective three-dimensional X-ray reconstruction. Due to the excellent feature learning ability of neural networks, deep learning has achieved outstanding performance in computer vision, natural language processing, and their various downstream tasks. Researchers have introduced deep learning into the three-dimensional X-ray reconstruction task, transforming the traditional method that relies on manual and prior knowledge into a three-dimensional reconstruction method that relies on autonomous learning from a large amount of data. By training neural networks, deep learning can automatically extract complex features and patterns from a large number of X-ray images for efficient three-dimensional reconstruction. Compared with traditional methods, the method using deep learning has significantly improved the efficiency and quality of three-dimensional X-ray image reconstruction. For example, Convolutional Neural Networks (CNNs) can exhibit excellent performance when processing large-scale image data, and achieve high-precision three-dimensional reconstruction by extracting local and global features of images layer by layer.

[0005] Currently, the mainstream three-dimensional body reconstruction methods still face challenges such as blurred organs and unclear edges, mainly for three reasons. First, they only rely on convolutional layers for feature extraction during the encoder sampling stage; second, during the encoding stage, dual-view features are extracted independently; finally, the type of dimension conversion from two-dimensional to three-dimensional in the deep layer usually uses simple dimension expansion, resulting in insufficient depth information and unable to perform accurate three-dimensional feature reconstruction. Summary of the Invention

[0006] In view of the above problems, the present invention provides a three-dimensional X-ray image reconstruction method, system, electronic device, and storage medium, aiming to complete the reconstruction of three-dimensional CT using only two X-ray pictures, reducing the radiation received by patients.

[0007] According to the first aspect of the embodiments of the present disclosure, a three-dimensional X-ray image reconstruction method is provided. The method includes using two parallel encoder-decoder networks to perform three-dimensional reconstruction on the coronal plane and sagittal plane of two-dimensional X-ray images respectively. Among them, in the encoder network, a local feature enhancement module is used to extract two-dimensional feature maps of the coronal plane X-ray image and the sagittal plane X-ray image respectively; an attention interaction module is used to perform information exchange and fusion on the two-dimensional feature maps of the coronal plane X-ray image and the sagittal plane X-ray image; a dimension conversion unit is used to convert the two-dimensional feature maps processed by the attention interaction module into three-dimensional feature maps;

[0008] In the decoder network, a 3D transposed convolution module is used to obtain a three-dimensional image by gradually magnifying the three-dimensional feature maps and performing three-dimensional convolution operations.

[0009] In some embodiments, during the process of reconstructing the three-dimensional features, the decoder network also introduces a multi-level feature fusion technology to retain the features of the reconstructed three-dimensional image in terms of details and global structure by fusing three-dimensional feature information at different levels layer by layer.

[0010] In some embodiments, the local feature enhancement module includes a global feature extraction branch and a local feature extraction branch. The global feature extraction branch uses a standard attention mechanism to extract low-frequency global features; the local feature extraction branch is transformed through a fully connected layer to obtain , , , and local feature aggregation is performed on . After aggregation, is combined with to perform context-aware local enhancement, thereby obtaining high-frequency local features; the high-frequency local features and the low-frequency global features are linked along the channel dimension, and then the dimension is reduced through a fully connected layer to output the two-dimensional feature map.

[0011] In some embodiments, depth convolution is used to perform local feature aggregation on , and the weights of local feature aggregation are globally shared. After aggregation, is combined with to generate context-aware weights.

[0012] In some embodiments, after aggregation, is combined with to generate context-aware weights, which specifically includes: using two depth convolutions to respectively aggregate the local information of and ; calculating the Hadamard product of and , and obtaining context-aware weights between -1 and 1 by transforming the result of the Hadamard product.

[0013] In some embodiments, an attention interaction module is used to perform information exchange and fusion on the two-dimensional feature maps of the coronal plane X-ray image and the sagittal plane X-ray image, which specifically includes:[[]]

[0014] The two-dimensional feature maps of the two X-ray images are respectively passed through stacked convolutional layers and residual connections to obtain feature maps including key information;

[0015] Fuse the dual-view features along the epipolar line, connect the feature maps including key information in high dimensions, and learn the weights of each channel through a self-attention module and a channel compression module to obtain the fused features after dual-view interaction;

[0016] The two-dimensional feature maps of the two X-ray images are respectively added with the fused features after dual-view interaction, and the output feature maps are obtained after passing through the activation layer.

[0017] In some embodiments, the dimension conversion unit uses a Transformer-Encoder module and combines skip connections to achieve the conversion between two-dimensional features and three-dimensional features, specifically including:

[0018] Flatten the two-dimensional feature map into a one-dimensional vector, and then obtain an important feature map through a multi-head self-attention mechanism and a feed-forward layer;

[0019] Use a linear layer, a Dropout layer, and an activation layer to reshape the important feature map into a 3D feature map.

[0020] According to a second aspect of the embodiments of the present disclosure, there is provided an X-ray image three-dimensional reconstruction system. The system uses two parallel encoder-decoder networks to perform three-dimensional reconstruction on the coronal plane and the sagittal plane of a two-dimensional X-ray image respectively, including:

[0021] An encoder unit, configured to use a local feature enhancement module to extract two-dimensional feature maps of the coronal plane X-ray image and the sagittal plane X-ray image respectively; use an attention interaction module to perform information exchange and fusion on the two-dimensional feature maps of the coronal plane X-ray image and the sagittal plane X-ray image; use a dimension conversion unit to convert the two-dimensional feature map processed by the attention interaction module into a three-dimensional feature map;

[0022] A decoder unit, configured to use a 3D transposed convolution module to obtain a three-dimensional image by gradually magnifying the three-dimensional feature map and performing three-dimensional convolution operations.

[0023] According to a third aspect of the embodiments of the present disclosure, there is provided an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the steps of the above X-ray image three-dimensional reconstruction method are implemented.

[0024] According to a fourth aspect of the embodiments of the present disclosure, there is provided a non-transitory computer-readable storage medium, on which computer instructions are stored. When the instructions are executed by a processor, the steps of the above X-ray image three-dimensional reconstruction method are implemented.

[0025] A 3D reconstruction method, system, electronic device, and storage medium provided by an embodiment of the present disclosure. The method includes using two parallel encoder-decoder networks to perform 3D reconstruction on the coronal plane and sagittal plane of 2D X-ray images respectively. In the encoder network, a local feature enhancement module is used to extract 2D feature maps of the coronal plane X-ray image and the sagittal plane X-ray image respectively. This step strengthens the local features of the image through a series of convolution operations and non-linear activation functions, enabling the network to better capture the key detail information in the image. An attention interaction module is used to exchange and fuse the 2D feature maps of the coronal plane X-ray image and the sagittal plane X-ray image; this is to make full use of the complementary information of the dual-view images, and an attention interaction module is introduced between the dual-views. This module exchanges and fuses information between the feature maps of the two views through an attention mechanism, thereby improving the accuracy and richness of feature extraction. The attention interaction module can effectively capture the correlation between different views, enabling the model to comprehensively consider the information of multiple views during the reconstruction process and improving the accuracy of the reconstruction result. A dimension conversion unit is used to convert the 2D feature map processed by the attention interaction module into a 3D feature map; the design purpose of this module is to expand and convert the dimension while maintaining the high-dimensional feature information, ensuring that a high-quality 3D structure can be effectively reconstructed in the decoding stage. The dimension conversion unit adopts Transformer-Encoder operations and combines skip connections to maximize the retention of feature information and reduce feature loss during the dimension conversion process. In the decoder network, a 3D transposed convolution module is used to obtain a 3D image by gradually magnifying the 3D feature map and performing 3D convolution operations. This process can not only reconstruct a high-resolution 3D structure but also retain the detail information and texture features of the image. To further improve the reconstruction effect, a multi-level feature fusion technology is also introduced in the decoder. By fusing the feature information of different levels layer by layer, it is ensured that the reconstructed 3D image has high accuracy and consistency in terms of details and global structure.

[0026] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The accompanying drawings herein are incorporated into the specification and form a part of the specification, showing embodiments consistent with the present invention, and are used together with the specification to explain the principles of the present invention.

[0028] Figure 1 is a flowchart of the 3D reconstruction method for X-ray images in an embodiment of the present invention;

[0029] Figure 2 is a flowchart of the local feature enhancement module in an embodiment of the present invention;

[0030] Figure 3 It is a flowchart of the attention interaction module in an embodiment of the present invention;

[0031] Figure 4 It is a flowchart of the Transformer encoder module in an embodiment of the present invention;

[0032] Figure 5 It is a schematic diagram of the decoder skip connection in an embodiment of the present invention;

[0033] Figure 6 It is a flowchart of the 3D decoder in an embodiment of the present invention;

[0034] Figure 7 It is a schematic diagram of the structure of the X-ray image three-dimensional reconstruction system in an embodiment of the present invention;

[0035] Figure 8 It is a schematic diagram of an electronic device in an embodiment of the present invention. Specific implementation manners

[0036] The present invention will be further described in detail below with reference to the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the present invention, rather than limiting the present invention. Additionally, it should be noted that for the convenience of description, only the parts related to the present invention rather than all the structures are shown in the drawings.

[0037] Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the steps as sequential processes, many of the steps can be implemented in parallel, concurrently, or simultaneously. In addition, the order of the steps can be rearranged. The process can be terminated when its operations are completed, but it can also have additional steps not included in the drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0038] The embodiments of the present invention provide the following embodiments for an X-ray image three-dimensional reconstruction method, system, electronic device, and storage medium:

[0039] One of the embodiments is used to illustrate the X-ray image three-dimensional reconstruction method, as Figure 1 shown, the method includes using two parallel encoder-decoder networks to perform three-dimensional reconstruction on the coronal plane and sagittal plane of the two-dimensional X-ray image respectively, where,

[0040] In the encoder network, a local feature enhancement module is used to extract two-dimensional feature maps of coronal X-ray images and sagittal X-ray images respectively; an attention interaction module is used to perform information exchange and fusion on the two-dimensional feature maps of coronal X-ray images and sagittal X-ray images; a dimension conversion unit is used to convert the two-dimensional feature maps processed by the attention interaction module into three-dimensional feature maps;

[0041] In the decoder network, a 3D transposed convolution module is used to obtain a three-dimensional image by gradually magnifying the three-dimensional feature map and performing three-dimensional convolution operations.

[0042] In the specific implementation process, in order to reconstruct a three-dimensional CT image from two X-ray images through a neural network and improve the doctor's diagnosis effect. The embodiment consists of a neural network, where:

[0043] The local feature enhancement module includes a global feature extraction branch and a local feature extraction branch. The global feature extraction branch uses a standard attention mechanism to extract low-frequency global features; the local feature extraction branch is obtained through a fully connected neural network transformation , , , which is the same as the standard attention operation, where , , refers to three key components used in the attention mechanism, representing "Query", "Key", and "Value" respectively. Subsequently, local feature aggregation is performed on . After aggregation, combines with to perform context-aware local enhancement, thereby obtaining high-frequency local features; the high-frequency local features and the low-frequency global features are linked along the channel dimension, and then the dimension is reduced through a fully connected layer to output the two-dimensional feature map.

[0044] Using depth convolution to perform local feature aggregation on , and the weights of local feature aggregation are globally shared. After aggregation, combines with to generate context-aware weights.

[0045] After aggregation, combines with to generate context-aware weights, specifically including: using two depth convolutions to aggregate the local information of and respectively; calculating the Hadamard product of and , and obtaining context-aware weights between -1 and 1 by transforming the result of the Hadamard product.

[0046] In the specific implementation process, among common reconstruction methods, dense connections are usually adopted in the downsampling stage to extract feature maps. Although this method is effective, it has certain limitations in processing high-frequency local information. To overcome these problems, this embodiment proposes to use a local feature enhancement module to extract two-dimensional features, which can better retain the high-frequency local information in the feature map. As Figure 2 shown, the local feature enhancement module includes a global feature extraction branch and a local feature extraction branch. In the global branch, the present invention directly uses a standard attention mechanism to extract low-frequency global information, and the expression is as follows:

[0047] ,

[0048] The global branch aims to optimize the attention calculation and expand the receptive field globally. However, this method has certain deficiencies in capturing high-frequency local details. To overcome this limitation, the present invention introduces a local attention enhancement mechanism. The local feature extraction branch includes some standard attention operations. First, it is transformed through a fully connected layer to obtain , , which is the same as the standard attention mechanism, and the expression is as follows:

[0049] ,

[0050] where represents the fully connected layer, and is the input of the local attention module.

[0051] After performing the linear transformation, local feature aggregation is first performed on . Then, based on the processed and context-aware local enhancement is performed again. To achieve this, the present invention uses simple depthwise convolution (DWconv) to aggregate local information, and the weights of DWconv are globally shared:

[0052] ,

[0053] After aggregating the local information and shared weights of , context-aware weights are generated by combining and . The method of the present invention is more refined in processing local features. Specifically, in this paper, two DWconvs are first used to aggregate the local information of and respectively. Secondly, the calculation of and Hadamard product, and perform a series of transformations on the result to obtain context-aware weights between -1 and 1. Finally, the generated weights are used to enhance local features. Specifically, it is as follows:

[0054] ,

[0055] where represents the Swish activation function.

[0056] Finally, the present invention combines the high-frequency local features and the low-frequency global features along the channel dimension, and then reduces the dimension through a fully connected layer to output . The formula is as follows:

[0057] ,

[0058] This dual-branch structure design enables the network to capture both global and local feature information simultaneously, thereby better retaining image details during the 3D reconstruction process. The global branch expands the receptive field through the standard attention mechanism to capture low-frequency global information, while the local branch aggregates high-frequency local information through the local feature enhancement mechanism to retain the details of the feature map.

[0059] Use the attention interaction module to perform information exchange and fusion on the two-dimensional feature maps of the coronal X-ray image and the sagittal X-ray image, specifically including:

[0060] Pass the two-dimensional feature maps of the two X-ray images through stacked convolutional layers and residual connections respectively to obtain feature maps including key information;

[0061] Fuse the dual-view features along the epipolar line, connect the feature maps including key information in the high dimension, and learn the weights of each channel through the self-attention module and the channel compression module to obtain the fused features after dual-view interaction;

[0062] Add the fused features after dual-view interaction to the two-dimensional feature maps of the two X-ray images respectively, and obtain the output feature maps after passing through the activation layer.

[0063] In the specific implementation process, in the existing method, each view only extracts and transforms features from its own perspective, ignoring the correlation between the two views. This method limits the full utilization of information and may lead to poor 3D reconstruction results. To overcome this defect, this embodiment proposes a dual-view attention interaction module that exchanges dual-view features at each stage of downsampling, as Figure 3 shown. This module first fuses the dual-view features along the epipolar line and adds the fused features back to the original features for compensation, thereby achieving more comprehensive feature extraction and information fusion.

[0064] First, for the feature maps in the dual-view backbone network , they need to go through three stacked Basic2d layers and residual connections to extract their respective key information . To ensure the consistency of the feature map dimensions, the kernel size of the convolutional layer is set to 3×3, and the stride is 1. The expression is as follows:

[0065] ,

[0066] where represents the stacked convolutional layers represents the feature maps of each view after feature extraction and residual connection

[0067] Secondly, since the capture heights of the dual-view X-ray cameras are the same, the present invention fuses the dual-view features along the epipolar line, connects the feature maps in the height dimension, and uses the self-attention module and the channel compression module to learn the weights of each channel. The formula is as follows:

[0068] ,

[0069] where represents the self-attention mechanism represents the channel compression module represents the fused features after dual-view interaction

[0070] Finally, the original input feature map is respectively added with the interaction features , and after passing through an activation layer, the output feature map is obtained

[0071] ,

[0072] where represents the activation function

[0073] Through this design, the dual-view attention interaction module can effectively fuse the information of the dual views in the feature extraction stage, enabling the model to capture more details and context information. This not only improves the accuracy of feature extraction but also enhances the effect of 3D reconstruction

[0074] The dimension conversion unit uses the Transformer-Encoder module and combines skip connections to achieve the conversion between 2D features and 3D features, specifically including:

[0075] Flatten the 2D feature map into a 1D vector, and then obtain the important feature map through the multi-head self-attention mechanism and the feed-forward layer

[0076] Use a linear layer, a Dropout layer, and an activation layer to reshape the important feature map into a 3D feature map.

[0077] In the specific implementation process, to reduce the loss of three-dimensional spatial features, an additional Transformer Encoder module is introduced to bridge the gap between two-dimensional and three-dimensional features. The advantage of the Transformer module lies in its global computing ability. Compared with traditional neural networks, the Transformer module can preserve more valuable spatial information by expanding the receptive field of the image, collecting additional context data, and capturing richer X-ray image features. This module plays a key role in converting two-dimensional X-ray image features into three-dimensional CT space features. As Figure 4 shown, the two-dimensional feature map is flattened into a one-dimensional vector, and then the important feature map is obtained through the multi-head self-attention mechanism and the feed-forward layer, as follows:

[0078] ,

[0079] ,

[0080] where represents the flattened one-dimensional vector, represents the Transformer encoder module, represents the positional encoding of the sequence.

[0081] Secondly, use a linear layer, a Dropout layer, and an activation layer to reshape the feature into a 3D feature map. These steps reduce the loss in the process of mapping from 2D to 3D and improve the effectiveness of 3D CT reconstruction.

[0082] ,

[0083] where represents the Dropout layer, represents the linear layer, represents the ReLU activation function, represents the reshaped 3D feature map.

[0084] By introducing the Transformer Encoder module, the present invention achieves more accurate information transmission and preservation in the process of 2D to 3D feature conversion. The global computing ability of the Transformer module not only expands the receptive field of the image but also can capture more context data and detailed features, thereby reducing feature loss and improving the accuracy and effectiveness of three-dimensional reconstruction.

[0085] During the process of reconstructing three-dimensional features, the decoder network also introduces a multi-level feature fusion technique. By fusing three-dimensional feature information at different levels layer by layer, it is used to retain the features of the reconstructed three-dimensional image in terms of details and global structure.

[0086] As Figure 5 shown, in order to enable the decoder part to obtain more detailed information and at the same time avoid problems of information loss and information damage, skip connections in the decoder network help to improve the accuracy of image reconstruction. First, the number of channels on the encoder side is made equal to the number of channels on the decoder side through a 2D convolutional layer. Secondly, the 2D feature map is extended to a 3D feature map by replicating the 2D information along the third axis. Finally, the 3D features are encoded using a basic 3D convolutional layer.

[0087] As Figure 6 shown, in the decoder stage, the embodiment uses a 3D transposed convolution module to reconstruct three-dimensional features. The 3D transposed convolution module restores the high-dimensional features to a three-dimensional image by gradually magnifying the feature map and performing three-dimensional convolution operations.

[0088] Another embodiment is used to illustrate the three-dimensional reconstruction system for X-ray images. Refer to Figure 7 , the system 700 uses two parallel encoder-decoder networks to perform three-dimensional reconstruction on the coronal plane and sagittal plane of the two-dimensional X-ray image respectively, including:

[0089] An encoder unit 710, configured to use a local feature enhancement module to extract two-dimensional feature maps of the coronal plane X-ray image and the sagittal plane X-ray image respectively; use an attention interaction module to perform information exchange and fusion on the two-dimensional feature maps of the coronal plane X-ray image and the sagittal plane X-ray image; use a dimension conversion unit to convert the two-dimensional feature map processed by the attention interaction module into a three-dimensional feature map;

[0090] A decoder unit 720, configured to use a 3D transposed convolution module to obtain a three-dimensional image by gradually magnifying the three-dimensional feature map and performing three-dimensional convolution operations.

[0091] In addition to the above modules, the system 700 may also include other components. However, since these components are not related to the content of the embodiments of the present disclosure, their illustrations and descriptions are omitted here.

[0092] For the other specific working processes of the X-ray image three-dimensional reconstruction system 700, refer to the description of the above-mentioned embodiments of the X-ray image three-dimensional reconstruction method, and details are not repeated here.

[0093] Another embodiment is used to illustrate that the system of the present invention can also be implemented by means of Figure 8 the architecture of the computing device shown. Figure 8 shows the architecture of the computing device. As Figure 8As shown, there are computer system 810, system bus 830, one or more CPUs 840, input / output 820, memory 850, etc. Memory 850 can store various data or files used for computer processing and / or communication, as well as program instructions executed by the CPU, including the program instructions for implementing the X-ray image three-dimensional reconstruction method of the embodiments. Figure 8 The architecture shown is only exemplary. When implementing different devices, adjust one or more components therein according to actual needs. Figure 8 Memory 850, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the X-ray image three-dimensional reconstruction method in the embodiments of the present invention (for example, the encoder unit 710 and decoder unit 720 in the X-ray image three-dimensional reconstruction system 700). One or more CPUs 840 execute various functional applications and data processing of the system of the present invention by running the software programs, instructions, and modules stored in memory 850, that is, implement the above-mentioned X-ray image three-dimensional reconstruction method, which includes using two parallel codec networks to respectively perform three-dimensional reconstruction on the coronal plane and sagittal plane of the two-dimensional X-ray image. Among them,

[0094] In the encoder network, a local feature enhancement module is used to respectively extract the two-dimensional feature maps of the coronal plane X-ray image and the sagittal plane X-ray image; an attention interaction module is used to perform information exchange and fusion on the two-dimensional feature maps of the coronal plane X-ray image and the sagittal plane X-ray image; a dimension conversion unit is used to convert the two-dimensional feature map processed by the attention interaction module into a three-dimensional feature map;

[0095] In the decoder network, a 3D transposed convolution module is used to obtain a three-dimensional image by gradually magnifying the three-dimensional feature map and performing three-dimensional convolution operations.

[0096] Certainly, for the server provided by the embodiments of the present invention, its processor is not limited to performing the method operations as described above, and can also perform related operations in the X-ray image three-dimensional reconstruction method provided by any embodiment of the present invention.

[0097] Memory 850 may mainly include a program storage area and a data storage area. Among them, the program storage area can store an operating system and application programs required for at least one function; the data storage area can store data created according to the use of the terminal, etc. In addition, memory 850 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some instances, memory 850 may further include a memory remotely set relative to one or more CPUs 840, and these remote memories can be connected to the device through a network. Examples of the above network include but are not limited to the Internet, enterprise intranet, local area network, mobile communication network, and their combinations.

[0098] The input / output 820 can be used to receive input digital or character information and generate key signal inputs related to the user settings and function controls of the device. The input / output 820 may further include a display device such as a display screen.

[0099] An embodiment of the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the X-ray image three-dimensional reconstruction method described in the above embodiment. The computer-readable storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (non-exhaustive list) of the computer-readable storage medium include: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0100] The computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries the computer-readable program code. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0101] The program code contained on the storage medium can be transmitted by any appropriate medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination of the above.

[0102] In addition, for the other specific working processes of a non-transitory computer-readable storage medium, reference can be made to the description of the embodiment of the X-ray image three-dimensional reconstruction method above, and details are not repeated here.

[0103] Based on the technical solutions provided in the above embodiments, an X-ray image three-dimensional reconstruction method, system, electronic device, and storage medium, the method includes using two parallel encoder-decoder networks to perform three-dimensional reconstruction on the coronal plane and sagittal plane of two-dimensional X-ray images respectively. Among them, in the encoder network, a local feature enhancement module is used to extract two-dimensional feature maps of the coronal plane X-ray image and the sagittal plane X-ray image respectively. This step strengthens the local features of the image through a series of convolution operations and non-linear activation functions, enabling the network to better capture the key detail information in the image. An attention interaction module is used to perform information exchange and fusion on the two-dimensional feature maps of the coronal plane X-ray image and the sagittal plane X-ray image; this is to make full use of the complementary information of the dual-view images, and an attention interaction module is introduced between the dual-views. Through the attention mechanism, this module performs information exchange and fusion between the feature maps of the two views, thereby improving the accuracy and richness of feature extraction. The attention interaction module can effectively capture the correlation between different views, enabling the model to comprehensively consider the information of multiple views during the reconstruction process and improving the accuracy of the reconstruction result. A dimension conversion unit is used to convert the two-dimensional feature map processed by the attention interaction module into a three-dimensional feature map; the design purpose of this module is to expand and convert the dimension while maintaining high-dimensional feature information, ensuring that a high-quality three-dimensional structure can be effectively reconstructed in the decoding stage. The dimension conversion unit adopts the Transformer-Encoder operation and combines skip connections to maximize the retention of feature information and reduce feature loss during the dimension conversion process. In the decoder network, a 3D transposed convolution module is used to obtain a three-dimensional image by gradually magnifying the three-dimensional feature map and performing three-dimensional convolution operations. This process can not only reconstruct a high-resolution three-dimensional structure but also retain the detail information and texture features of the image. To further improve the reconstruction effect, a multi-level feature fusion technology is also introduced in the decoder. By layer-by-layer fusing of feature information at different levels, it is ensured that the reconstructed three-dimensional image has high accuracy and consistency in both details and global structure.

[0104] In this article, the term "including", "comprising", or any other variant thereof is intended to cover a non-exclusive inclusion, such that a step or method including a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such step or method.

[0105] The above content is a further detailed description of the present invention in combination with specific preferred implementation manners, and it cannot be determined that the specific implementation of the present invention is only limited to these descriptions. For those of ordinary skill in the technical field to which the present invention belongs, without departing from the concept of the present invention, several simple deductions or substitutions can still be made, and all should be regarded as belonging to the protection scope of the present invention.

Claims

1. A method for three-dimensional reconstruction of X-ray images, characterized in that: The method comprises using two parallel codec networks to perform three-dimensional reconstruction on the coronal plane and sagittal plane of a two-dimensional X-ray image, respectively, wherein: In the encoder network, a local feature enhancement module is used to extract two-dimensional feature maps of the coronal X-ray image and the sagittal X-ray image respectively; an attention interaction module is used to exchange and fuse information between the two-dimensional feature maps of the coronal X-ray image and the sagittal X-ray image; and a dimension conversion unit is used to convert the two-dimensional feature map processed by the attention interaction module into a three-dimensional feature map; In the decoder network, a 3D deconvolution module is used to gradually enlarge the 3D feature map and perform a 3D convolution operation to obtain a 3D image. Among them, the attention interaction module is used to exchange and fuse the two-dimensional feature maps of the coronal X-ray image and the sagittal X-ray image, specifically including: The two-dimensional feature maps of the two X-ray images are respectively subjected to stacked convolutional layers and residual connections to obtain feature maps containing key information; The dual-view features are fused along the epipolar line, the feature maps including key information are connected in high dimensions, and the weights of each channel are learned through the self-attention module and channel compression module to obtain the fused features after dual-view interaction; The two-dimensional feature maps of the two X-ray images are added with the fusion features after dual-view interaction, and the output feature map is obtained after the activation layer.

2. The X-ray image three-dimensional reconstruction method according to claim 1, characterized in that: In the process of reconstructing the three-dimensional image, the decoder network also introduces a multi-level feature fusion technology, which is used to retain the features of the reconstructed three-dimensional image in terms of details and global structure by fusing three-dimensional feature information of different levels layer by layer.

3. The X-ray image three-dimensional reconstruction method according to claim 1, characterized in that: The local feature enhancement module includes a global feature extraction branch and a local feature extraction branch. The global feature extraction branch uses a standard attention mechanism to extract low-frequency global features. The local feature extraction branch is transformed by a fully connected layer to obtain , , ,exist Local feature aggregation is performed on Combination , Performing context-aware local enhancement to obtain high-frequency local features; linking the high-frequency local features and the low-frequency global features along the channel dimension, and then reducing the dimension through a fully connected layer to output the two-dimensional feature map, , , Represents the components of a standard attention mechanism: query, key, and value.

4. The X-ray image three-dimensional reconstruction method according to claim 3, characterized in that: Using deep convolution Local feature aggregation is performed on the local feature aggregation, and the weight of local feature aggregation is shared globally. Combination , Generate context-aware weights.

5. The X-ray image three-dimensional reconstruction method according to claim 4, characterized in that: After aggregation Combination , Generate context-aware weights, specifically by using two deep convolutions to aggregate and Local information of and The Hadamard product of is taken and the context-aware weight between -1 and 1 is obtained by transforming the Hadamard product result.

6. The X-ray image three-dimensional reconstruction method according to claim 1, characterized in that: The dimension conversion unit uses the Transformer-Encoder module and combines the jump connection to realize the conversion of two-dimensional features and three-dimensional features, which specifically includes: Flatten the two-dimensional feature map into a one-dimensional vector, and then obtain the important feature map through the multi-head self-attention mechanism and feed-forward layer; Linear layers, Dropout layers, and activation layers are used to reshape the important feature maps into 3D feature maps.

7. A three-dimensional reconstruction system for X-ray images, characterized in that: The system utilizes two parallel codec networks to perform three-dimensional reconstruction of the coronal and sagittal planes of two-dimensional X-ray images, respectively, including: The encoder unit is used to extract two-dimensional feature maps of the coronal X-ray image and the sagittal X-ray image respectively by using the local feature enhancement module; exchange and fuse the two-dimensional feature maps of the coronal X-ray image and the sagittal X-ray image by using the attention interaction module; and convert the two-dimensional feature map processed by the attention interaction module into a three-dimensional feature map by using the dimension conversion unit; A decoder unit, configured to obtain a three-dimensional image by gradually enlarging the three-dimensional feature map and performing a three-dimensional convolution operation using a 3D deconvolution module; Among them, the attention interaction module is used to exchange and fuse the two-dimensional feature maps of the coronal X-ray image and the sagittal X-ray image, specifically including: The two-dimensional feature maps of the two X-ray images are respectively subjected to stacked convolutional layers and residual connections to obtain feature maps containing key information; The dual-view features are fused along the epipolar line, the feature maps including key information are connected in high dimensions, and the weights of each channel are learned through the self-attention module and channel compression module to obtain the fused features after dual-view interaction; The two-dimensional feature maps of the two X-ray images are added with the fusion features after dual-view interaction, and the output feature map is obtained after the activation layer.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the X-ray image three-dimensional reconstruction method as claimed in any one of claims 1 to 6 are implemented.

9. A non-transitory computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instructions are executed by the processor, the steps of the X-ray image three-dimensional reconstruction method as claimed in any one of claims 1 to 6 are implemented.

Images