X-ray image three-dimensional reconstruction method and system, electronic equipment and storage medium

Through parallel codec network and feature fusion technology, X-ray images are used for three-dimensional reconstruction, which solves the problem that X-ray imaging technology cannot obtain the three-dimensional structure, reduces radiation dose and improves image quality, and achieves more accurate diagnosis.

CN119027595AInactive Publication Date: 2024-11-26HARBIN INSTITUTE OF TECHNOLOGY (SHENZHEN) (INSTITUTE OF SCIENCE AND TECHNOLOGY INNOVATION HARBIN INSTITUTE OF TECHNOLOGY SHENZHEN)
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202411506157.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-28
Publication Date
2024-11-26
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Existing X-ray imaging technology can only obtain two-dimensional images and cannot fully present the detailed three-dimensional structure inside the human body, which limits clinical evaluation and treatment options, and patients need to withstand higher radiation doses, which poses safety risks.

Method used

Two X-ray images are used for three-dimensional reconstruction. Through a parallel codec network, local feature enhancement module, attention interaction module and dimension conversion unit are used to gradually extract and fuse feature information to realize the reconstruction of three-dimensional images.

Benefits of technology

It reduces the radiation dose received by the patient, improves the efficiency and quality of the three-dimensional reconstruction of X-ray images, and can more accurately reconstruct a high-resolution three-dimensional structure, retains the details and texture characteristics of the image, and improves the accuracy of diagnosis.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027595A_ABST
    Figure CN119027595A_ABST
Patent Text Reader

Abstract

The invention discloses an X-ray image three-dimensional reconstruction method and system, electronic equipment and a storage medium, and the method comprises the steps: carrying out the three-dimensional reconstruction of a coronal plane and a sagittal plane of a two-dimensional X-ray image through employing two parallel codec networks, utilizing a local feature enhancement module to respectively extract two-dimensional feature maps of the coronal plane X-ray image and the sagittal plane X-ray image; performing information exchange and fusion on the two-dimensional feature maps of the coronal plane X-ray image and the sagittal plane X-ray image by using an attention interaction module; converting the two-dimensional feature map processed by the attention interaction module into a three-dimensional feature map by using a dimension conversion unit; and in the decoder network, obtaining a three-dimensional image by gradually amplifying the three-dimensional feature map and performing a three-dimensional convolution operation by using a 3D deconvolution module. According to the method, three-dimensional CT reconstruction can be completed by using two X-ray pictures, and radiation accepted by a patient is reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and specifically relates to a method, system, electronic equipment and storage medium for three-dimensional reconstruction of X-ray images. Background Art

[0002] In clinical practice, medical images are usually obtained using technologies such as computed tomography (CT), magnetic resonance imaging (MRI), and X-ray imaging to further understand the patient's internal conditions. However, each of these three imaging technologies has its limitations. For example, CT images can provide detailed three-dimensional visual information of the internal structure of the human body, but patients need to bear a high radiation dose, which poses a safety hazard. In addition, the ionizing radiation used in the CT imaging process may cause long-term health effects on the human body, especially for patients who need multiple examinations. MRI provides a non-invasive three-dimensional imaging method without ionizing radiation, but due to its strong magnetic field characteristics, it cannot be applied to patients with metal implants. In addition, the MRI imaging process is time-consuming and the equipment is expensive, which limits its popularization and application in some medical institutions. At the same time, both CT and MRI imaging require patients to lie down, which results in the inconsistency between the weight-bearing conditions of human organs and clinical diagnostic needs, affecting the accuracy of certain diagnoses.

[0003] In contrast, X-ray imaging is the most commonly used clinical modality due to its low cost, low radiation dose, and the ability to image standing patients. Clinicians usually obtain dual-plane X-ray images of patients through coronal and sagittal views to diagnose various diseases in the body. However, the obvious disadvantage of X-ray imaging is that it can only obtain two-dimensional projections of three-dimensional objects, providing only limited two-dimensional information and unable to fully present three-dimensional images of detailed internal structures of the human body. This shortcoming limits clinical evaluation and treatment options, and may lead to unstable and error-prone clinical decisions. For example, when doctors are faced with complex fractures or internal organ injuries, it may be difficult to accurately judge the extent and location of the injury based on two-dimensional X-ray images alone, thus affecting the treatment effect.

[0004] To solve this problem, many researchers are committed to combining new technologies to achieve more effective X-ray 3D reconstruction. Due to the excellent feature learning ability of neural networks, deep learning has achieved outstanding performance in computer vision, natural language processing and its various downstream tasks. Researchers introduced deep learning into the task of X-ray 3D reconstruction, transforming the traditional method that relies on manual and prior knowledge into a 3D reconstruction method that relies on large amounts of data for autonomous learning. By training neural networks, deep learning can automatically extract complex features and patterns from a large number of X-ray images for efficient 3D reconstruction. Compared with traditional methods, the use of deep learning methods has significantly improved the efficiency and quality of 3D reconstruction of X-ray images. For example, convolutional neural networks (CNN) can show excellent performance when processing large-scale image data, and achieve high-precision 3D reconstruction by extracting local and global features of images layer by layer.

[0005] The current mainstream 3D reconstruction methods still face challenges such as blurred organs and unclear edges, mainly for three reasons. First, they rely only on convolutional layers for feature extraction in the encoder sampling stage; second, in the encoding stage, dual-view features are extracted independently; and finally, the deep 2D to 3D dimensional conversion type usually adopts a simple dimensional expansion, resulting in insufficient depth information and unable to accurately reconstruct 3D features. Summary of the invention

[0006] In view of the above problems, the present invention provides a method, system, electronic device and storage medium for three-dimensional reconstruction of X-ray images, aiming to complete the reconstruction of three-dimensional CT using two X-ray images and reduce the radiation received by the patient.

[0007] According to a first aspect of an embodiment of the present disclosure, a method for three-dimensional reconstruction of an X-ray image is provided, the method comprising using two parallel encoder-decoder networks to respectively perform three-dimensional reconstruction on the coronal plane and the sagittal plane of a two-dimensional X-ray image, wherein in the encoder network, a local feature enhancement module is used to respectively extract two-dimensional feature maps of the coronal plane X-ray image and the sagittal plane X-ray image; an attention interaction module is used to perform information exchange and fusion on the two-dimensional feature maps of the coronal plane X-ray image and the sagittal plane X-ray image; and a dimension conversion unit is used to convert the two-dimensional feature map processed by the attention interaction module into a three-dimensional feature map; In the decoder network, a 3D deconvolution module is used to obtain a 3D image by gradually enlarging the 3D feature map and performing a 3D convolution operation.

[0008] In some embodiments, during the process of reconstructing the three-dimensional features, the decoder network also introduces a multi-level feature fusion technology to preserve the details and global structure features of the reconstructed three-dimensional image by fusing three-dimensional feature information of different levels layer by layer.

[0009] In some embodiments, the local feature enhancement module includes a global feature extraction branch and a local feature extraction branch, wherein the global feature extraction branch uses a standard attention mechanism to extract low-frequency global features; the local feature extraction branch obtains , , ,exist Local feature aggregation is performed on Combination Context-aware local enhancement is performed to obtain high-frequency local features; the high-frequency local features and the low-frequency global features are linked along the channel dimension, and then the dimension is reduced through a fully connected layer to output the two-dimensional feature map.

[0010] In some embodiments, using depthwise convolution Local feature aggregation is performed on the local feature aggregation, and the weight of local feature aggregation is shared globally. Combination Generate context-aware weights.

[0011] In some embodiments, after polymerization Combination Generate context-aware weights, specifically by using two deep convolutions to aggregate and Local information of and The Hadamard product of is taken and the context-aware weight between -1 and 1 is obtained by transforming the Hadamard product result.

[0012] In some embodiments, the attention interaction module is used to exchange and fuse information between the two-dimensional feature maps of the coronal X-ray image and the sagittal X-ray image, specifically including: The two-dimensional feature maps of the two X-ray images are respectively subjected to stacked convolutional layers and residual connections to obtain feature maps containing key information; The dual-view features are fused along the epipolar line, the feature maps including key information are connected in high dimensions, and the weights of each channel are learned through the self-attention module and channel compression module to obtain the fused features after dual-view interaction; The two-dimensional feature maps of the two X-ray images are added with the fusion features after dual-view interaction, and the output feature map is obtained after the activation layer.

[0013] In some embodiments, the dimension conversion unit uses a Transformer-Encoder module and combines a jump connection to realize the conversion between two-dimensional features and three-dimensional features, specifically including: Flatten the two-dimensional feature map into a one-dimensional vector, and then obtain the important feature map through the multi-head self-attention mechanism and feed-forward layer; Linear layers, Dropout layers, and activation layers are used to reshape the important feature maps into 3D feature maps.

[0014] According to a second aspect of an embodiment of the present disclosure, a system for three-dimensional reconstruction of an X-ray image is provided. The system uses two parallel codec networks to perform three-dimensional reconstruction on the coronal plane and the sagittal plane of a two-dimensional X-ray image, respectively, and includes: The encoder unit is used to extract two-dimensional feature maps of the coronal X-ray image and the sagittal X-ray image respectively by using the local feature enhancement module; exchange and fuse the two-dimensional feature maps of the coronal X-ray image and the sagittal X-ray image by using the attention interaction module; and convert the two-dimensional feature map processed by the attention interaction module into a three-dimensional feature map by using the dimension conversion unit; The decoder unit is used to obtain a three-dimensional image by gradually enlarging the three-dimensional feature map and performing a three-dimensional convolution operation using a 3D deconvolution module.

[0015] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned X-ray image three-dimensional reconstruction method when executing the program.

[0016] According to a fourth aspect of an embodiment of the present disclosure, a non-temporary computer-readable storage medium is provided, on which computer instructions are stored, and when the instructions are executed by a processor, the steps of the above-mentioned X-ray image three-dimensional reconstruction method are implemented.

[0017] The disclosed embodiment provides a method, system, electronic device and storage medium for three-dimensional reconstruction of X-ray images. The method includes using two parallel encoder-decoder networks to perform three-dimensional reconstruction on the coronal plane and sagittal plane of a two-dimensional X-ray image, respectively. In the encoder network, a local feature enhancement module is used to extract two-dimensional feature maps of the coronal plane X-ray image and the sagittal plane X-ray image, respectively. This step strengthens the local features of the image through a series of convolution operations and nonlinear activation functions, so that the network can better capture the key detail information in the image. The attention interaction module is used to exchange and fuse information between the two-dimensional feature maps of the coronal plane X-ray image and the sagittal plane X-ray image; this is to fully utilize the complementary information of the dual-view images, and the attention interaction module is introduced between the dual views. The module exchanges and fuses information between the feature maps of the two views through the attention mechanism, thereby improving the accuracy and richness of feature extraction. The attention interaction module can effectively capture the correlation between different viewpoints, so that the model can comprehensively consider the information of multiple viewpoints during the reconstruction process, and improve the accuracy of the reconstruction results. The dimensional conversion unit is used to convert the two-dimensional feature map processed by the attention interaction module into a three-dimensional feature map; the design purpose of this module is to achieve dimensional expansion and conversion while maintaining high-dimensional feature information, so as to ensure that high-quality three-dimensional structures can be effectively reconstructed in the decoding stage. The dimension conversion unit adopts the Transformer-Encoder operation, combined with the jump connection, to maximize the retention of feature information and reduce feature loss during the dimension conversion process. In the decoder network, the 3D deconvolution module is used to gradually enlarge the three-dimensional feature map and perform a three-dimensional convolution operation to obtain a three-dimensional image. This process can not only reconstruct a high-resolution three-dimensional structure, but also retain the detail information and texture features of the image. In order to further improve the reconstruction effect, the decoder also introduces a multi-level feature fusion technology, which ensures that the reconstructed three-dimensional image has high accuracy and consistency in both details and global structure by fusing feature information of different levels layer by layer.

[0018] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present invention and, together with the description, serve to explain the principles of the present invention. Figure 1 is a flowchart of a method for three-dimensional reconstruction of X-ray images in an embodiment of the present invention; Figure 2 It is a flowchart of a local feature enhancement module in an embodiment of the present invention; Figure 3 is a flowchart of the attention interaction module in an embodiment of the present invention; Figure 4 is a flow chart of the Transformer encoder module in an embodiment of the present invention; Figure 5 is a schematic diagram of decoder jump connection in an embodiment of the present invention; Figure 6 is a flow chart of a 3D decoder according to an embodiment of the present invention; Figure 7 is a schematic diagram of the structure of a three-dimensional reconstruction system for X-ray images in an embodiment of the present invention; Figure 8 It is a schematic diagram of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0021] It should be mentioned before discussing the exemplary embodiments in more detail that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the steps as sequential processes, many of the steps therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of the steps can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0022] The present invention provides the following embodiments for a method, system, electronic device and storage medium for three-dimensional reconstruction of X-ray images: One of the embodiments is used to illustrate the three-dimensional reconstruction method of X-ray images. Figure 1 As shown, the method includes using two parallel encoder-decoder networks to perform three-dimensional reconstruction on the coronal plane and sagittal plane of the two-dimensional X-ray image, respectively, wherein, In the encoder network, a local feature enhancement module is used to extract two-dimensional feature maps of the coronal X-ray image and the sagittal X-ray image respectively; an attention interaction module is used to exchange and fuse information between the two-dimensional feature maps of the coronal X-ray image and the sagittal X-ray image; and a dimension conversion unit is used to convert the two-dimensional feature map processed by the attention interaction module into a three-dimensional feature map; In the decoder network, a 3D deconvolution module is used to obtain a 3D image by gradually enlarging the 3D feature map and performing a 3D convolution operation.

[0023] In the specific implementation process, in order to reconstruct a three-dimensional CT image from two X-ray images through a neural network to improve the doctor's diagnosis effect. The embodiment is composed of a neural network, in which: The local feature enhancement module includes a global feature extraction branch and a local feature extraction branch. The global feature extraction branch uses a standard attention mechanism to extract low-frequency global features; the local feature extraction branch obtains , , , which is the same as the standard attention operation, where , , Refers to the three key components used in the attention mechanism, which stand for "query", "key", and "value", followed by Local feature aggregation is performed on Combination Context-aware local enhancement is performed to obtain high-frequency local features; the high-frequency local features and the low-frequency global features are linked along the channel dimension, and then the dimension is reduced through a fully connected layer to output the two-dimensional feature map.

[0024] Using deep convolution Local feature aggregation is performed on the local feature aggregation, and the weight of local feature aggregation is shared globally. Combination Generate context-aware weights.

[0025] After aggregation Combination Generate context-aware weights, specifically by using two deep convolutions to aggregate and Local information of and The Hadamard product of is taken and the context-aware weight between -1 and 1 is obtained by transforming the Hadamard product result.

[0026] In the specific implementation process, in common reconstruction methods, dense connections are usually used to extract feature maps in the downsampling stage. Although this method is effective, it has certain limitations when processing high-frequency local information. In order to overcome these problems, this embodiment proposes to use a local feature enhancement module to extract two-dimensional features, which can better retain the high-frequency local information in the feature map. Figure 2 As shown, the local feature enhancement module includes a global feature extraction branch and a local feature extraction branch. In the global branch, the present invention directly uses the standard attention mechanism to extract low-frequency global information, and the expression is as follows: , The global branch aims to optimize the attention calculation and expand the receptive field globally. However, this method has certain shortcomings in capturing high-frequency local details. To overcome this limitation, the present invention introduces a local attention enhancement mechanism. The local feature extraction branch includes some standard attention operations. First, it is transformed by a fully connected layer to obtain , , Same as the standard attention mechanism, the expression is as follows: , in represents the fully connected layer, is the input of the local attention module.

[0027] After performing the linear transformation, first Then, based on the processed and Then perform context-aware local enhancement. To achieve this, the present invention uses a simple deep convolution (DWconv) to aggregate local information, while the weights of DWconv are globally shared: , polymerization After combining the local information and shared weights and To generate context-aware weights. The method of the present invention is more sophisticated in processing local features. Specifically, this paper first uses two DWconv to aggregate and Secondly, calculate and The Hadamard product of is taken and a series of transformations are performed on the result to obtain context-aware weights between -1 and 1. Finally, the generated weights are used to enhance the local features. The details are as follows: , in Represents the Swish activation function.

[0028] Finally, the present invention converts high-frequency local features and low-frequency global features and connected along the channel dimension, and then passed through a fully connected layer to reduce the dimension, thus outputting The formula is as follows: , This dual-branch structure design enables the network to capture global and local feature information at the same time, thereby better preserving image details during 3D reconstruction. The global branch expands the receptive field through the standard attention mechanism to capture low-frequency global information, while the local branch aggregates high-frequency local information through the local feature enhancement mechanism to preserve the details of the feature map.

[0029] The attention interaction module is used to exchange and fuse the two-dimensional feature maps of the coronal X-ray image and the sagittal X-ray image, including: The two-dimensional feature maps of the two X-ray images are respectively subjected to stacked convolutional layers and residual connections to obtain feature maps containing key information; The dual-view features are fused along the epipolar line, the feature maps including key information are connected in high dimensions, and the weights of each channel are learned through the self-attention module and channel compression module to obtain the fused features after dual-view interaction; The two-dimensional feature maps of the two X-ray images are added with the fusion features after dual-view interaction, and the output feature map is obtained after the activation layer.

[0030] In the specific implementation process, in the existing methods, each view only extracts and transforms features from its own perspective, ignoring the correlation between the two views. This method limits the full utilization of information and may lead to poor 3D reconstruction. To overcome this defect, this embodiment proposes a dual-view attention interaction module, which exchanges dual-view features at each stage of downsampling, such as Figure 3 This module first fuses the dual-view features along the epipolar lines and adds the fused features back to the original features for compensation, thus achieving more comprehensive feature extraction and information fusion.

[0031] First, for the feature map in the dual-view backbone network , need to go through three stacked Basic2d layers and residual connections to extract their respective key information, In order to ensure the consistency of feature map dimensions, the kernel size of the convolutional layer is set to 3×3 and the stride is 1, as shown below: , in represents the stacked convolutional layers, Represents the feature map of each view after feature extraction and residual connection.

[0032] Secondly, since the dual-view X-ray camera has the same capture height, the present invention fuses the dual-view features along the epipolar line, connects the feature maps in the height dimension, and uses the self-attention module and the channel compression module to learn the weight of each channel. The formula is as follows: , in represents the self-attention mechanism, represents the channel compression module, Represents the fusion features after dual-view interaction.

[0033] Finally, the original input feature map Add interaction features separately , after passing through an activation layer, the output feature map is obtained , , in express Activation function.

[0034] Through this design, the dual-view attention interaction module can effectively fuse the information of the two views in the feature extraction stage, allowing the model to capture more details and contextual information. This not only improves the accuracy of feature extraction, but also enhances the effect of 3D reconstruction.

[0035] The dimension conversion unit uses the Transformer-Encoder module and combines jump connections to achieve the conversion of two-dimensional features to three-dimensional features, including: Flatten the two-dimensional feature map into a one-dimensional vector, and then obtain the important feature map through the multi-head self-attention mechanism and feed-forward layer; Linear layers, Dropout layers, and activation layers are used to reshape the important feature maps into 3D feature maps.

[0036] In the specific implementation process, in order to reduce the loss of three-dimensional spatial features, an additional TransformerEncoder module is introduced to bridge the gap between two-dimensional and three-dimensional features. The advantage of the Transformer module lies in its global computing capability. Compared with traditional neural networks, the Transformer module can preserve more valuable spatial information by expanding the receptive field of the image, collecting additional contextual data, and capturing richer X-ray image features. This module plays a key role in converting two-dimensional X-ray image features into three-dimensional CT spatial features. Figure 4 As shown, the two-dimensional feature map is flattened into a one-dimensional vector, and then the important feature map is obtained through the multi-head self-attention mechanism and the feed-forward layer, as shown below: , , in represents the flattened one-dimensional vector, represents the Transformer encoder module, Represents the positional encoding of a sequence.

[0037] Secondly, use the linear layer, dropout layer and activation layer to transform the features Reshape into a 3D feature map. These steps reduce the loss in the mapping process from 2D to 3D and improve the effectiveness of 3D CT reconstruction. , in represents the Dropout layer, represents a linear layer, represents the ReLU activation function, Represents the reshaped 3D feature map.

[0038] By introducing the Transformer Encoder module, the present invention realizes more accurate information transmission and preservation in the process of 2D to 3D feature conversion. The global computing capability of the Transformer module not only expands the receptive field of the image, but also captures more contextual data and detailed features, thereby reducing feature loss and improving the accuracy and effectiveness of 3D reconstruction.

[0039] In the process of reconstructing three-dimensional features, the decoder network also introduces a multi-level feature fusion technology, which is used to retain the features of the reconstructed three-dimensional image in terms of details and global structure by fusing three-dimensional feature information of different levels layer by layer.

[0040] like Figure 5 As shown in the figure, in order to allow the decoder to obtain more detailed information while avoiding the problem of information loss and information loss, the decoder network uses jump connections to help improve the accuracy of image reconstruction. First, the number of channels on the encoder side is equal to the number of channels on the decoder side through the 2D convolution layer. Secondly, the 2D feature map is expanded to a 3D feature map by copying the 2D information along the third axis. Finally, the 3D features are encoded using the basic 3D convolution layer.

[0041] like Figure 6 As shown, in the decoder stage, the embodiment uses a 3D deconvolution module to reconstruct three-dimensional features. The 3D deconvolution module restores high-dimensional features to a three-dimensional image by gradually enlarging the feature map and performing a three-dimensional convolution operation.

[0042] Another embodiment is used to illustrate the X-ray image 3D reconstruction system, see Figure 7 The system 700 utilizes two parallel codec networks to perform three-dimensional reconstruction on the coronal plane and sagittal plane of a two-dimensional X-ray image, respectively, including: The encoder unit 710 is used to respectively extract the two-dimensional feature maps of the coronal plane X-ray image and the sagittal plane X-ray image using the local feature enhancement module; perform information exchange and fusion on the two-dimensional feature maps of the coronal plane X-ray image and the sagittal plane X-ray image using the attention interaction module; and convert the two-dimensional feature map processed by the attention interaction module into a three-dimensional feature map using the dimension conversion unit; The decoder unit 720 is used to obtain a three-dimensional image by using a 3D deconvolution module by gradually enlarging the three-dimensional feature map and performing a three-dimensional convolution operation.

[0043] In addition to the upper module, the system 700 may also include other components. However, since these components are irrelevant to the content of the embodiment of the present disclosure, their illustration and description are omitted here.

[0044] The other specific working processes of the X-ray image three-dimensional reconstruction system 700 refer to the description of the above-mentioned basic X-ray image three-dimensional reconstruction method embodiment and will not be repeated here.

[0045] Another embodiment is used to illustrate that the system of the present invention can also be used with the help of Figure 8 The architecture of the computing device shown is implemented. Figure 8 The architecture of the computing device is shown. Figure 8 As shown, a computer system 810, a system bus 830, one or more CPUs 840, an input / output 820, a memory 850, etc. The memory 850 can store various data or files used for computer processing and / or communication and program instructions including the embodiment X-ray image three-dimensional reconstruction method executed by the CPU. Figure 8 The architecture shown is only exemplary and can be adjusted according to actual needs when implementing different devices. Figure 8 One or more components in. The memory 850, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the X-ray image three-dimensional reconstruction method in the embodiment of the present invention (for example, the encoder unit 710 and the decoder unit 720 in the X-ray image three-dimensional reconstruction system 700). One or more CPUs 840 execute various functional applications and data processing of the system of the present invention by running the software programs, instructions and modules stored in the memory 850, that is, to implement the above-mentioned X-ray image three-dimensional reconstruction method, which includes using two parallel codec networks to perform three-dimensional reconstruction of the coronal plane and the sagittal plane of the two-dimensional X-ray image, respectively, wherein, In the encoder network, a local feature enhancement module is used to extract two-dimensional feature maps of the coronal X-ray image and the sagittal X-ray image respectively; an attention interaction module is used to exchange and fuse information between the two-dimensional feature maps of the coronal X-ray image and the sagittal X-ray image; and a dimension conversion unit is used to convert the two-dimensional feature map processed by the attention interaction module into a three-dimensional feature map; In the decoder network, a 3D deconvolution module is used to obtain a 3D image by gradually enlarging the 3D feature map and performing a 3D convolution operation.

[0046] Of course, the processor of the server provided in the embodiment of the present invention is not limited to executing the method operations described above, but can also execute related operations in the X-ray image three-dimensional reconstruction method provided in any embodiment of the present invention.

[0047] The memory 850 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 850 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 850 may further include a memory remotely arranged relative to one or more CPUs 840, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0048] The input / output 820 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the device. The input / output 820 may also include a display device such as a display screen.

[0049] The embodiment of the present invention also provides a non-temporary computer-readable storage medium, on which a computer program is stored. When the computer program is executed by the processor, the X-ray image three-dimensional reconstruction method described in the above embodiment is implemented. The computer-readable storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, device or device.

[0050] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0051] The program code contained on the storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination of the foregoing.

[0052] In addition, other specific working processes of a non-temporary computer-readable storage medium refer to the description of the above-mentioned X-ray image three-dimensional reconstruction method embodiment and will not be repeated here.

[0053] In summary, the technical solutions provided by the above embodiments provide a method, system, electronic device and storage medium for three-dimensional reconstruction of X-ray images. The method includes using two parallel encoder-decoder networks to perform three-dimensional reconstruction on the coronal plane and sagittal plane of a two-dimensional X-ray image, respectively. In the encoder network, a local feature enhancement module is used to extract two-dimensional feature maps of the coronal plane X-ray image and the sagittal plane X-ray image, respectively. This step strengthens the local features of the image through a series of convolution operations and nonlinear activation functions, so that the network can better capture the key detail information in the image. The attention interaction module is used to exchange and fuse information between the two-dimensional feature maps of the coronal plane X-ray image and the sagittal plane X-ray image; this is to make full use of the complementary information of the dual-view images, and the attention interaction module is introduced between the dual views. The module exchanges and fuses information between the feature maps of the two views through the attention mechanism, thereby improving the accuracy and richness of feature extraction. The attention interaction module can effectively capture the correlation between different viewpoints, so that the model can comprehensively consider the information of multiple viewpoints during the reconstruction process, and improve the accuracy of the reconstruction results. The dimensional conversion unit is used to convert the two-dimensional feature map processed by the attention interaction module into a three-dimensional feature map; the design purpose of this module is to achieve dimensional expansion and conversion while maintaining high-dimensional feature information, so as to ensure that high-quality three-dimensional structures can be effectively reconstructed in the decoding stage. The dimension conversion unit adopts the Transformer-Encoder operation, combined with the jump connection, to maximize the retention of feature information and reduce feature loss during the dimension conversion process. In the decoder network, the 3D deconvolution module is used to gradually enlarge the three-dimensional feature map and perform a three-dimensional convolution operation to obtain a three-dimensional image. This process can not only reconstruct a high-resolution three-dimensional structure, but also retain the detail information and texture features of the image. In order to further improve the reconstruction effect, the decoder also introduces a multi-level feature fusion technology, which ensures that the reconstructed three-dimensional image has high accuracy and consistency in both details and global structure by fusing feature information of different levels layer by layer.

[0054] In this document, the terms "comprises," "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, such that a step or method that includes a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such step or method.

[0055] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A method for three-dimensional reconstruction of X-ray images, characterized in that: The method comprises using two parallel codec networks to perform three-dimensional reconstruction on the coronal plane and sagittal plane of a two-dimensional X-ray image, respectively, wherein: In the encoder network, a local feature enhancement module is used to extract two-dimensional feature maps of the coronal X-ray image and the sagittal X-ray image respectively; an attention interaction module is used to exchange and fuse information between the two-dimensional feature maps of the coronal X-ray image and the sagittal X-ray image; and a dimension conversion unit is used to convert the two-dimensional feature map processed by the attention interaction module into a three-dimensional feature map; In the decoder network, a 3D deconvolution module is used to obtain a 3D image by gradually enlarging the 3D feature map and performing a 3D convolution operation.

2. The X-ray image three-dimensional reconstruction method according to claim 1, characterized in that: In the process of reconstructing the three-dimensional features, the decoder network also introduces a multi-level feature fusion technology, which is used to retain the features of the reconstructed three-dimensional image in terms of details and global structure by fusing three-dimensional feature information of different levels layer by layer.

3. The X-ray image three-dimensional reconstruction method according to claim 1, characterized in that: The local feature enhancement module includes a global feature extraction branch and a local feature extraction branch. The global feature extraction branch uses a standard attention mechanism to extract low-frequency global features. The local feature extraction branch is transformed by a fully connected layer to obtain , , ,exist Local feature aggregation is performed on Combination , Performing context-aware local enhancement to obtain high-frequency local features; linking the high-frequency local features and the low-frequency global features along the channel dimension, and then reducing the dimension through a fully connected layer to output the two-dimensional feature map, , , Represents the components of a standard attention mechanism: query, key, and value.

4. The X-ray image three-dimensional reconstruction method according to claim 3, characterized in that: Using deep convolution Local feature aggregation is performed on the local feature aggregation, and the weight of local feature aggregation is shared globally. Combination Generate context-aware weights.

5. The X-ray image three-dimensional reconstruction method according to claim 4, characterized in that: After aggregation Combination Generate context-aware weights, specifically by using two deep convolutions to aggregate and Local information of and The Hadamard product of is taken and the context-aware weight between -1 and 1 is obtained by transforming the Hadamard product result.

6. The X-ray image three-dimensional reconstruction method according to claim 1, characterized in that: The attention interaction module is used to exchange and fuse the two-dimensional feature maps of the coronal X-ray image and the sagittal X-ray image, including: The two-dimensional feature maps of the two X-ray images are respectively subjected to stacked convolutional layers and residual connections to obtain feature maps containing key information; The dual-view features are fused along the epipolar line, the feature maps including key information are connected in high dimensions, and the weights of each channel are learned through the self-attention module and channel compression module to obtain the fused features after dual-view interaction; The two-dimensional feature maps of the two X-ray images are added with the fusion features after dual-view interaction, and the output feature map is obtained after the activation layer.

7. The X-ray image three-dimensional reconstruction method according to claim 1, characterized in that: The dimension conversion unit uses the Transformer-Encoder module and combines the jump connection to realize the conversion of two-dimensional features and three-dimensional features, which specifically includes: Flatten the two-dimensional feature map into a one-dimensional vector, and then obtain the important feature map through the multi-head self-attention mechanism and feed-forward layer; Linear layers, Dropout layers, and activation layers are used to reshape the important feature maps into 3D feature maps.

8. A three-dimensional reconstruction system for X-ray images, characterized in that: The system utilizes two parallel codec networks to perform three-dimensional reconstruction of the coronal and sagittal planes of two-dimensional X-ray images, respectively, including: The encoder unit is used to extract two-dimensional feature maps of the coronal X-ray image and the sagittal X-ray image respectively by using the local feature enhancement module; to exchange and fuse the two-dimensional feature maps of the coronal X-ray image and the sagittal X-ray image by using the attention interaction module; and to convert the two-dimensional feature map processed by the attention interaction module into a three-dimensional feature map by using the dimension conversion unit; The decoder unit is used to obtain a three-dimensional image by gradually enlarging the three-dimensional feature map and performing a three-dimensional convolution operation using a 3D deconvolution module.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the X-ray image three-dimensional reconstruction method as described in any one of claims 1 to 7 are implemented.

10. A non-transitory computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instructions are executed by the processor, the steps of the three-dimensional reconstruction method of X-ray images as described in any one of claims 1 to 7 are implemented.

Citation Information

Patent Citations

  • Method and device and equipment for generating CT image and storage medium

    CN109745062A

  • Target tracking method and system based on convolution and attention combined feature extraction

    CN117710688A

  • Apparatus, method, and non-transitory computer-readable storage medium for improving image quality of a medical image volume

    US20230011759A1