Image three-dimensional reconstruction method, device, electronic device and storage medium

Abstract

This application is applicable to the field of image processing technology and provides a method, apparatus, electronic device, and storage medium for three-dimensional image reconstruction, including: acquiring a two-dimensional image; inputting the two-dimensional image into a trained feature extraction network for processing to obtain two-dimensional feature information corresponding to the two-dimensional image; inputting the two-dimensional feature information into a trained dimension transformation network for processing to obtain three-dimensional feature information; and inputting the three-dimensional feature information into a trained image generation network for processing to obtain a target three-dimensional reconstructed image; wherein the image generation network sequentially includes a three-dimensional point-by-point convolution layer and a three-dimensional deconvolution layer. Embodiments of this application can efficiently and accurately achieve three-dimensional image reconstruction.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular relates to a method, device, electronic device and storage medium for three-dimensional image reconstruction. Background Art

[0002] 3D image reconstruction involves inferring the 3D shape of an object from a single or multiple 2D images to reconstruct an image containing the object's 3D information. With the advancement of computer vision, 3D image reconstruction algorithms have been widely used in many fields, such as robotic navigation, 3D modeling, object recognition, scene recognition, and medical diagnosis. However, a method for 3D image reconstruction that guarantees both efficiency and accuracy is currently lacking. Summary of the Invention

[0003] In view of this, embodiments of the present application provide a method, apparatus, electronic device, and storage medium for three-dimensional image reconstruction to solve the problem of how to efficiently and accurately achieve three-dimensional image reconstruction in the prior art.

[0004] A first aspect of an embodiment of the present application provides a method for three-dimensional image reconstruction, comprising:

[0005] Acquire a two-dimensional image;

[0006] Inputting the two-dimensional image into a trained feature extraction network for processing to obtain two-dimensional feature information corresponding to the two-dimensional image;

[0007] Inputting the two-dimensional feature information into a trained dimension transformation network for processing to obtain three-dimensional feature information;

[0008] The three-dimensional feature information is input into a trained image generation network for processing to obtain a target three-dimensional reconstructed image; wherein the image generation network includes a three-dimensional point-by-point convolution layer and a three-dimensional deconvolution layer in sequence.

[0009] Optionally, the three-dimensional deconvolution layer of the image generation network includes a first preset number of three-dimensional convolution kernels of different scales.

[0010] Optionally, inputting the two-dimensional image into a trained feature extraction network for processing to obtain two-dimensional feature information corresponding to the two-dimensional image includes:

[0011] Inputting the two-dimensional image into a trained feature extraction network for processing to obtain two-dimensional feature information of at least two different scales; wherein the feature extraction network includes at least two network layers connected in sequence, and the two-dimensional feature information of different scales corresponds to feature information output by different network layers;

[0012] Correspondingly, the inputting the two-dimensional feature information into the trained dimension transformation network for processing to obtain three-dimensional feature information includes:

[0013] Each of the two-dimensional feature information is input into a trained dimension transformation network for processing to obtain each of the three-dimensional feature information corresponding to each of the two-dimensional feature information.

[0014] Optionally, the feature extraction network includes a first CBAM network layer, N ECA network layers, and a second CBAM network layer connected in sequence, and the ECA network layer includes a residual block;

[0015] The two-dimensional feature information of different scales includes the N two-dimensional feature information output by the N ECA network layers and the two-dimensional feature information output by the second CBAM network layer; wherein, the input of the first CBAM network layer is the two-dimensional image, the input of the first ECA network layer is the output of the first CBAM network layer, the input of the i-th ECA network layer is the output of the i-1-th ECA network layer, and the input of the second CBAM network layer is the output of the last ECA network layer; N is a positive integer greater than 1, and i is any positive integer greater than 1 and less than or equal to N.

[0016] Optionally, the dimensionality transformation network includes N+1 transformation sub-networks, the input of the j-th transformation sub-network is the output of the j-th ECA network layer, and the output of the j-th transformation sub-network is the three-dimensional feature information converted from the two-dimensional feature information output by the j-th ECA network layer; the input of the last transformation sub-network is the output of the second CBAM network layer, and the output of the last transformation sub-network is the three-dimensional feature information converted from the two-dimensional feature information output by the second CBAM network layer; wherein j is a positive integer less than or equal to N.

[0017] Optionally, the image generation network includes N+1 feature generation network layers and a target convolutional network layer; the input of the first feature generation network layer is the output of the first transformation subnetwork, and the input of the mth feature generation network layer includes the output of the mth transformation subnetwork and the output of the m-1th transformation subnetwork; the input of the target convolutional network layer is the output of the last transformation subnetwork, and the output of the target convolutional network layer is the target three-dimensional reconstructed image; wherein m is any positive integer greater than 1 and less than or equal to N+1.

[0018] Optionally, the three-dimensional image reconstruction method is applied to three-dimensional reconstruction of X-ray images, and before acquiring the two-dimensional image, further comprises:

[0019] Acquire a preset number of CT sample images from a preset CT image data set;

[0020] Obtaining a preset number of two-dimensional projection images according to the preset number of CT sample images and the DDR projection algorithm;

[0021] Generating a preset number of X-ray sample images according to the preset number of two-dimensional projection images and a preset style transfer algorithm;

[0022] The feature extraction network, the dimensionality transformation network and the image generation network are trained based on the preset number of X-ray sample images to obtain the trained feature extraction network, the trained dimensionality transformation network and the trained image generation network.

[0023] A second aspect of an embodiment of the present application provides a three-dimensional image reconstruction device, comprising:

[0024] an acquisition unit, configured to acquire a two-dimensional image;

[0025] a feature extraction unit, configured to input the two-dimensional image into a trained feature extraction network for processing, and obtain two-dimensional feature information corresponding to the two-dimensional image;

[0026] a dimensionality transformation unit, configured to input the two-dimensional feature information into a trained dimensionality transformation network for processing to obtain three-dimensional feature information;

[0027] A reconstruction unit is used to input the three-dimensional feature information into a trained image generation network for processing to obtain a target three-dimensional reconstructed image; wherein the image generation network includes a three-dimensional point-by-point convolution layer and a three-dimensional deconvolution layer in sequence.

[0028] A third aspect of an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the electronic device implements the steps of the three-dimensional image reconstruction method.

[0029] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the electronic device implements the steps of the image three-dimensional reconstruction method.

[0030] A fifth aspect of the embodiments of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the three-dimensional image reconstruction method described in any one of the first aspects above.

[0031] Compared with the prior art, the embodiments of the present application have the following advantages: in the embodiments of the present application, after acquiring a two-dimensional image, the two-dimensional image is input into a trained feature extraction network for processing to obtain the two-dimensional feature information corresponding to the two-dimensional image; the two-dimensional feature information is then input into a trained dimension transformation network for processing to obtain three-dimensional feature information; thereafter, the three-dimensional feature information is input into a trained image generation network for processing to obtain the target three-dimensional reconstructed image. Since the image generation network includes a three-dimensional deconvolution layer, accurate upsampling can be achieved through the three-dimensional deconvolution layer, compensating for the image resolution loss caused by the downsampling process of the convolution operation in the feature extraction network, thereby improving the image reconstruction effect; and since the image generation network also includes a three-dimensional point-by-point convolution layer before the three-dimensional deconvolution layer, by adding the three-dimensional point-by-point convolution layer, more details can be obtained in the generated image, and the number of redundant features input to the subsequent three-dimensional deconvolution layer can be effectively reduced, reducing the computational complexity of the subsequent three-dimensional deconvolution process, improving computational efficiency, and thus improving image reconstruction efficiency. Therefore, the image three-dimensional reconstruction method of the embodiments of the present application can improve image reconstruction efficiency while ensuring image reconstruction effect, and achieve image three-dimensional reconstruction efficiently and accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art.

[0033] Figure 1 This is a schematic diagram of an implementation flow of a method for three-dimensional image reconstruction provided by an embodiment of the present application;

[0034] Figure 2 This is a schematic diagram of the structure of a feature generation network layer provided in an embodiment of the present application;

[0035] Figure 3 This is a schematic diagram of the structure of a CBAM network layer provided in an embodiment of the present application;

[0036] Figure 4 This is a schematic diagram of the structure of an ECA network layer provided in an embodiment of the present application;

[0037] Figure 5 This is an example diagram of a three-dimensional image reconstruction network provided by an embodiment of the present application;

[0038] Figure 6 is a schematic diagram of a three-dimensional image reconstruction device provided in an embodiment of the present application;

[0039] Figure 7 Schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0041] In order to illustrate the technical solution described in this application, specific embodiments are provided below.

[0042] It will be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0043] It should also be understood that the terms used in this specification are for the purpose of describing specific embodiments only and are not intended to limit the present application. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0044] It should be further understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0045] As used in this specification and the appended claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0046] In addition, in the description of the present application, the terms "first", "second", "third", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0047] Currently, 3D image reconstruction is typically performed using convolutional neural networks (CNNs). However, because CNNs often cause resolution loss during network sampling and image reconstruction requires a large amount of computation, the accuracy and efficiency of current 3D image reconstruction remain low.

[0048] In order to solve the above-mentioned technical problems, the embodiments of the present application provide a method, device, electronic device and storage medium for three-dimensional reconstruction of an image. After acquiring a two-dimensional image, the two-dimensional image is input into a trained feature extraction network for processing to obtain two-dimensional feature information corresponding to the two-dimensional image; the two-dimensional feature information is then input into a trained dimensionality transformation network for processing to obtain three-dimensional feature information; thereafter, the three-dimensional feature information is input into an image generation network for sequentially performing three-dimensional point-by-point convolution processing and three-dimensional deconvolution processing to obtain a target three-dimensional reconstructed image.

[0049] Because the image generation network includes a three-dimensional deconvolution layer, accurate upsampling can be achieved through the three-dimensional deconvolution layer, compensating for the image resolution loss caused by the downsampling process of the convolution operation in the feature extraction network, and improving the image reconstruction effect; and because the image generation network also includes a three-dimensional point-by-point convolution layer before the three-dimensional deconvolution layer, by adding the three-dimensional point-by-point convolution layer, more details can be obtained in the generated image, and the number of redundant features input to the subsequent three-dimensional deconvolution layer can be effectively reduced, reducing the amount of calculation in the subsequent three-dimensional deconvolution process, improving the calculation efficiency, and thus improving the image reconstruction efficiency. Therefore, the image three-dimensional reconstruction method of the embodiment of the present application can improve the image reconstruction efficiency while ensuring the image reconstruction effect, and achieve image three-dimensional reconstruction efficiently and accurately.

[0050] For example, the three-dimensional image reconstruction method of the embodiment of the present application can be applied in medical scenarios.

[0051] In clinical practice, computed tomography (CT) technology can acquire three-dimensional volumetric information of patient tissue. However, in practice, this method requires projection from a large number of different angles to maintain acceptable resolution and mitigate the physics-based artifacts of the tomography scanner. Consequently, CT scanning technology suffers from high radiation exposure, long reconstruction times, and artifacts. In addition to CT scanning technology, the new Electro Optical System (EOS) offers a better option for full-body dual-plane X-ray scanning and three-dimensional reconstruction of the entire skeleton. However, the reconstruction process of EOS imaging is based on a statistical shape model (SSM), so the resulting model is not a complete reflection of the patient's tissue, but rather a semantically similar virtual model. Therefore, achieving three-dimensional reconstruction using CT or EOS technology still has certain drawbacks.

[0052] In order to solve this technical problem, an X-ray image of the patient can be taken, and based on the two-dimensional X-ray image, the image three-dimensional reconstruction method of the embodiment of the present application can be used to reconstruct a target three-dimensional reconstructed image. The target three-dimensional reconstructed image can be a CT sequence image containing spatial information of human tissue. Among them, the X-ray image refers to an image that can present different tissue parts of the human body by irradiating a part of the human body with X-rays (also called Roentgen rays) and utilizing the principle that different tissues of the human body absorb X-rays to different degrees; the X-ray image acquisition process is less irradiated to the patient than the CT scanning process, and the two-dimensional X-ray image actually contains hidden spatial information. Therefore, the image three-dimensional reconstruction method of the embodiment of the present application is used to perform three-dimensional reconstruction of the X-ray image, which can efficiently and accurately reconstruct the target three-dimensional reconstructed image containing three-dimensional spatial information of human tissue with little impact on the human body. This method can be applied to the three-dimensional reconstruction task of X-ray images of various human tissues such as the human spine, teeth, lungs, and chest, providing great convenience for doctors' work.

[0053] Example 1:

[0054] Figure 1 The figure shows a flow chart of a method for 3D reconstruction of an image provided by an embodiment of the present application. The method is applied to electronic devices, including but not limited to computers, tablet computers, image capture devices, etc. Figure 1 The three-dimensional reconstruction method of the image shown is described in detail as follows:

[0055] In S101 , a two-dimensional image is acquired.

[0056] The two-dimensional image in the embodiments of the present application can express information such as shadows, occlusions, and relative sizes through two-dimensional pixels, thereby reflecting the relative spatial position of the photographed object. In one embodiment, the two-dimensional image is obtained through projection radiography, and the pixels in the two-dimensional image implicitly contain information about transparent volumetric structures other than opaque surfaces. For example, the two-dimensional image can be an X-ray image.

[0057] In one embodiment, the electronic device of the present application is capable of communicating with an image capture device, and after the image capture device captures an object to generate a two-dimensional image, the electronic device receives the two-dimensional image transmitted by the image capture device. In another embodiment, the electronic device of the present application can be directly an image capture device, and after receiving a capture instruction, begins capturing an object in a designated capture area to generate the two-dimensional image. For example, the image capture device can be an X-ray machine.

[0058] In S102, the two-dimensional image is input into a trained feature extraction network for processing to obtain two-dimensional feature information corresponding to the two-dimensional image.

[0059] In an embodiment of the present application, the electronic device includes a trained image three-dimensional reconstruction network, which is a neural network. The trained image three-dimensional reconstruction network specifically includes three parts: a trained feature extraction network, a trained dimensionality transformation network, and a trained image generation network.

[0060] After acquiring a two-dimensional image, the image is input into the trained feature extraction network described above for feature extraction, thereby obtaining two-dimensional feature information corresponding to the image. Specifically, the feature extraction network is a deep convolutional network comprising convolutional layers, which extract feature information from the two-dimensional image through convolution operations. The feature extraction network typically includes convolution processing across multiple layers, and the feature extraction process includes a downsampling process, so that the ultimately extracted two-dimensional feature information includes deep feature information with a scale smaller than the input scale.

[0061] In S103, the two-dimensional feature information is input into a trained dimension transformation network for processing to obtain three-dimensional feature information.

[0062] In an embodiment of the present application, the dimension transformation network is specifically a neural network including a dimension conversion function and an activation function. For example, it is assumed that the two-dimensional feature information extracted by the feature extraction network includes two-dimensional feature information of size (C, H, W) (where C represents channel information, H represents image height information, and W represents image width information). By processing the two-dimensional feature information through the dimension conversion function, initial three-dimensional feature information of size (C, 1, H, W) can be obtained. The initial three-dimensional feature information (C, 1, H, W) can be further processed in sequence through a deconvolution layer with a kernel size of D×1×1, an activation function, and a normalization function (such as a batch normalization function) to obtain the three-dimensional feature information corresponding to the two-dimensional feature information. Wherein, D is the preset depth information. In some embodiments, the size of D can be equal to the current number of channels, or can be determined according to the current number of network layers. The aforementioned activation function can be a linear rectification function (ReLU).

[0063] In S104, the three-dimensional feature information is input into a trained image generation network for processing to obtain a target three-dimensional reconstructed image; wherein the image generation network includes a three-dimensional point-by-point convolution layer and a three-dimensional deconvolution layer in sequence.

[0064] In the embodiment of the present application, the image generation network is capable of reconstructing a target containing three-dimensional information of the image based on the three-dimensional feature information of the image. Figure 3 The image generation network specifically includes a 3D point-by-point convolutional layer and a 3D deconvolutional layer.

[0065] After obtaining the corresponding three-dimensional feature information based on the two-dimensional feature information of the two-dimensional image, the three-dimensional feature information is input into the trained image generation network. It first passes through the three-dimensional point-by-point convolution layer to extract single-point features, while obtaining more detailed feature information and reducing the convolution dimension. Afterwards, the feature information output by the three-dimensional point-by-point convolution layer is input into the three-dimensional deconvolution layer for processing. The three-dimensional deconvolution layer can achieve an upsampling effect, converting low-scale feature information into high-scale feature information for output. After obtaining the high-scale feature information, based on the high-scale feature information and the convolution operation, a target three-dimensional reconstructed image containing three-dimensional feature information is generated. In one embodiment, the target three-dimensional reconstructed image is a three-dimensional image. In another embodiment, the target three-dimensional reconstructed image is a sequence of multiple images containing three-dimensional volume information of the object. For example, if the two-dimensional image is an X-ray image, the target three-dimensional reconstructed image can be a CT sequence image containing human tissue volume information.

[0066] In an embodiment of the present application, after acquiring a two-dimensional image, the two-dimensional image is input into a trained feature extraction network for processing to obtain the two-dimensional feature information corresponding to the two-dimensional image; the two-dimensional feature information is then input into a trained dimension transformation network for processing to obtain three-dimensional feature information; thereafter, the three-dimensional feature information is input into a trained image generation network for processing to obtain a target three-dimensional reconstructed image. Since the image generation network includes a three-dimensional deconvolution layer, accurate upsampling can be achieved through the three-dimensional deconvolution layer, compensating for the image resolution loss caused by the downsampling process of the convolution operation in the feature extraction network, thereby improving the image reconstruction effect; and since the image generation network also includes a three-dimensional point-by-point convolution layer before the three-dimensional deconvolution layer, by adding the three-dimensional point-by-point convolution layer, more details can be obtained in the generated image, and at the same time, the number of redundant features input to the subsequent three-dimensional deconvolution layer can be effectively reduced, reducing the amount of calculation of the subsequent three-dimensional deconvolution process, improving the calculation efficiency, and thus improving the image reconstruction efficiency. Therefore, the image three-dimensional reconstruction method of the embodiment of the present application can improve the image reconstruction efficiency while ensuring the image reconstruction effect, and achieve image three-dimensional reconstruction efficiently and accurately.

[0067] Optionally, the feature extraction network includes a residual block and / or an attention module.

[0068] In one embodiment, the above-mentioned feature extraction network may include a residual block. Exemplarily, the network structure of the feature extraction network may be a ResNet structure (a residual network structure), such as a ResNet34 structure. At this time, the input of the feature extraction network may be a two-dimensional image of size 128×128. The first layer of the network is composed of a convolutional layer with a kernel size of 7×7 and a stride of 2. The second to fifth layers are composed of 4 residual blocks, and include two convolutional layers with a convolution kernel of 3×3. In this network structure, the number of channels of the convolutional layer in each residual block remains the same to ensure that the shortcut path and the residual path can maintain the same size during the element-by-element addition operation. Through the setting of the residual block, the intermediate feature information can be directly connected to the last residual block through a long connection, so that the subsequent network layer can learn the previous intermediate feature information, reduce the information loss and loss problem of the feature information when passing through each convolution layer, improve the accuracy of feature extraction, and solve the gradient disappearance problem during the training process.

[0069] In another embodiment, the feature extraction network further includes an attention module. After a two-dimensional image is input into the feature extraction network, the attention module first processes the image to obtain attention feature information for the two-dimensional image. This attention feature information is then used to perform convolution processing on the image to obtain the corresponding two-dimensional feature information. This attention module enhances the extraction of effective features from the two-dimensional image, reduces the extraction of redundant, useless information, improves image reconstruction efficiency, and reduces redundant features in the final reconstructed image.

[0070] Optionally, the three-dimensional deconvolution layer of the image generation network includes a first preset number of three-dimensional convolution kernels of different scales.

[0071] In the embodiment of the present application, the image generation network specifically includes at least one feature generation network layer, which includes a three-dimensional point-by-point convolution layer and a three-dimensional deconvolution layer in sequence, and the three-dimensional deconvolution layer includes a first preset number of three-dimensional convolution kernels of different scales. The first preset number is greater than 1 and can be 2, 3, 4, etc., which is set according to actual needs. For example, the feature generation network layer is as follows: Figure 2 As shown in the figure. The three-dimensional feature information generated by the dimensionality transformation network is input into a 3D point-by-point convolution layer of size 1×1×1, an activation layer ReLu, and a normalization layer Batch Normalization. It is then input into a 3D deconvolution layer. This 3D deconvolution layer contains two branches: the upper branch contains a deconvolution with a convolution kernel of 5×5×5, and the lower branch contains a deconvolution with a convolution kernel of 3×3×3. These two deconvolution layers are also connected to the activation layer ReLu and the normalization layer Batch Normalization.

[0072] In the embodiment of the present application, since the three-dimensional deconvolution layer contains different branch structures, each branch contains convolution kernels of different scales, thereby being able to generate information of different scales during deconvolution, generate richer and more accurate detail feature information, and improve the accuracy of three-dimensional image reconstruction.

[0073] Optionally, inputting the two-dimensional image into a trained feature extraction network for processing to obtain two-dimensional feature information corresponding to the two-dimensional image includes:

[0074] Inputting the two-dimensional image into a trained feature extraction network for processing to obtain two-dimensional feature information of at least two different scales; wherein the feature extraction network includes at least two network layers connected in sequence, and the two-dimensional feature information of different scales corresponds to feature information output by different network layers;

[0075] Correspondingly, the inputting the two-dimensional feature information into the trained dimension transformation network for processing to obtain three-dimensional feature information includes:

[0076] Each of the two-dimensional feature information is input into a trained dimension transformation network for processing to obtain each of the three-dimensional feature information corresponding to each of the two-dimensional feature information.

[0077] In the embodiment of the present application, the feature extraction network specifically includes at least two network layers, and each network layer has a convolution operation. After the two-dimensional image is input into the trained feature extraction network, it undergoes convolution operations and downsampling operations of different network layers in sequence to obtain two-dimensional feature information of different scales from large to small, that is, the two-dimensional feature information of different scales is the feature information output by different network layers. In the network layer of the feature extraction network, the two-dimensional feature information input by the shallow network has a larger scale and a stronger ability to represent semantic information, but lacks spatial geometric detail information; the two-dimensional feature information output by the deep network has a smaller scale and a stronger ability to represent geometric detail information, but a weaker semantic representation ability.

[0078] After obtaining the two-dimensional feature information of different scales, each two-dimensional feature information is input into the trained dimension transformation network for dimension conversion processing to obtain each three-dimensional feature information corresponding to each two-dimensional feature information.

[0079] In the embodiment of the present application, since it is possible to obtain two-dimensional feature information of different scales by acquiring feature information output by different network layers of the feature extraction network, and generate three-dimensional feature information corresponding to these two-dimensional feature information through the dimensionality transformation network, these three-dimensional feature information can be input into the image generation network for processing, and the three-dimensional features of the image at different scales can be fused, so that the semantic information and deep geometric detail information of the image can be accurately reconstructed, thereby making the final reconstructed image clearer and more accurate.

[0080] Optionally, the feature extraction network includes a first CBAM network layer, N ECA network layers, and a second CBAM network layer connected in sequence, and the ECA network layer includes a residual block;

[0081] The two-dimensional feature information of different scales includes the N two-dimensional feature information output by the N ECA network layers and the two-dimensional feature information output by the second CBAM network layer; wherein, the input of the first CBAM network layer is the two-dimensional image, the input of the first ECA network layer is the output of the first CBAM network layer, the input of the i-th ECA network layer is the output of the i-1-th ECA network layer, and the input of the second CBAM network layer is the output of the last ECA network layer; N is a positive integer greater than 1, and i is any positive integer greater than 1 and less than or equal to N.

[0082] The feature extraction network in the embodiment of the present application specifically includes a convolutional block attention module (CBAM) network layer and an efficient channel attention module (Efficient channel Attention module) network layer. Both attention modules are lightweight attention modules, which enable efficient and accurate feature extraction of two-dimensional images without increasing the complexity of the network.

[0083] For example, the structure of the CBAM network layer in the embodiment of the present application is as follows: Figure 3 As shown, it includes a channel attention module and a spatial attention module, so that the feature information of the two-dimensional image can be extracted more accurately and effectively based on the channel attention and spatial attention.

[0084] For example, the structure of the ECA network layer in the embodiment of the present application is as follows: Figure 4 As shown in the figure, there are two residual blocks at the beginning and end, and two convolution layers with kernels of 1×1×C in the middle (where C is the number of channels). This ECA is a local cross-channel interaction module that does not reduce the feature dimension. It obtains local cross-channel interaction information by combining each channel with its K adjacent channels. Where K is a positive integer, which can be specifically obtained by a preset function on the channel. Adaptive determination.

[0085] The feature extraction network in the embodiment of the present application is specifically improved on the basis of the ResNet34 network structure. The first and last layers of the feature extraction network are both CBAM network layers (to distinguish them, they are respectively referred to as the first CBAM network layer and the second CBAM network layer). It can improve the feature adaptive extraction capability while ensuring that the overall network structure is not affected and the scale of the input and output features is consistent with the scale of the original ResNet34 network. The intermediate network layer can include a second preset number of ECA network layers, which can efficiently obtain local cross-channel interaction information without reducing the feature dimension and network complexity.

[0086] The feature extraction network of the embodiment of the present application specifically includes a first CBAM network layer, N ECA network layers, and a second CBAM network layer in sequence, where N is a positive integer greater than 1, such as 2, 3, 4, etc.

[0087] In the feature extraction network, the two-dimensional image input to the feature extraction network is used as the input of the first CBAM network layer, and the output of the first CBAM network layer is used as the initial feature information X0. Since the first CBAM network layer is the first network layer, the feature information output by the first CBAM network layer is relatively shallow and insufficient to effectively represent the features of the two-dimensional image. At this time, it is not used as two-dimensional feature information, but is passed to the next network layer for processing, that is, it is input into the first ECA network layer (for example Figure 5 The ECA network layer 1) shown is processed.

[0088] For each of the N ECA network layers, the input to the first ECA network layer is the output of the first CBAM network layer preceding it. Furthermore, for the ECA network layer following the first ECA network layer, i.e., the i-th ECA network layer, its input is the output of the i-1th ECA network layer. For each of the N ECA network layers, one branch of its output is input to the next network layer (i.e., the next ECA network layer or the second CBAM network layer), and the other branch of its output is a two-dimensional feature information. In other words, each of the N ECA network layers outputs N pieces of two-dimensional feature information.

[0089] For the last network layer of the feature extraction network, that is, the second CBAM network layer, its input is the feature information output by the last ECA network layer among the N ECA network layers, and its output is the last two-dimensional feature information with the smallest scale.

[0090] The two-dimensional feature information of different scales finally output by the feature extraction network includes the two-dimensional feature information output by each ECA network layer and the two-dimensional feature information output by the second CBAM network, totaling N+1 two-dimensional feature information.

[0091] In an embodiment of the present application, the feature extraction network is composed of a CBAM network layer and an ECA network layer in an orderly manner. Through the processing of the CBAM network layer and the ECA network layer, it is possible to accurately and effectively realize the feature extraction of two-dimensional images based on the attention mechanism while lightweighting the network and reducing the amount of calculation, thereby improving the efficiency and accuracy of three-dimensional reconstruction of the image.

[0092] Optionally, the dimensionality transformation network includes N+1 transformation sub-networks, the input of the j-th transformation sub-network is the output of the j-th ECA network layer, and the output of the j-th transformation sub-network is the three-dimensional feature information converted from the two-dimensional feature information output by the j-th ECA network layer; the input of the last transformation sub-network is the output of the second CBAM network layer, and the output of the last transformation sub-network is the three-dimensional feature information converted from the two-dimensional feature information output by the second CBAM network layer; wherein j is a positive integer less than or equal to N.

[0093] In this embodiment of the present application, the dimension transformation network includes N+1 transformation sub-networks corresponding to the N+1 two-dimensional feature information output by the feature extraction network. Each transformation sub-network performs dimension transformation processing on a corresponding two-dimensional feature information. For example, each transformation sub-network includes the dimension transformation function described in step S103 above, a deconvolution layer with a kernel size of D×1×1, an activation function, and a normalization function.

[0094] Specifically, for any one of the first N transformation sub-networks, i.e., the j-th transformation sub-network, its input is the output of the j-th ECA network layer, and its output is the three-dimensional feature information obtained by dimensionally transforming the two-dimensional feature information output by its corresponding j-th ECA network layer. For the last transformation sub-network, i.e., the N+1-th transformation sub-network, its input is the output of the second CBAM network layer, and its output is the three-dimensional feature information obtained by dimensionally transforming the two-dimensional feature information output by its corresponding second CBAM network layer.

[0095] In an embodiment of the present application, each two-dimensional feature information is processed separately by each transformation sub-network in the dimensional transformation network, so that each three-dimensional feature information corresponding to each two-dimensional feature information can be accurately generated, thereby improving the accuracy of three-dimensional reconstruction of the image.

[0096] Optionally, the image generation network includes N+1 feature generation network layers and a target convolutional network layer; the input of the first feature generation network layer is the output of the last transformation subnetwork, and the input of the mth feature generation network layer includes the output of the mth to last transformation subnetwork and the output of the m-1th transformation subnetwork; the input of the target convolutional network layer is the output of the last transformation subnetwork, and the output of the target convolutional network layer is the target three-dimensional reconstructed image; wherein m is any positive integer greater than 1 and less than or equal to N+1.

[0097] In the embodiments of the present application, the image generation network specifically includes a feature generation network layer for upsampling and fusing the 3D feature information of the image, and a target convolutional network layer for ultimately reconstructing the target 3D feature information to obtain the target 3D reconstructed image. The number of feature generation network layers is the same as the number of 3D feature information, which is N+1.

[0098] Specifically, due to the downsampling that occurs during the convolution of the feature extraction network, the scales of the N+1 two-dimensional feature information output by the feature extraction network decrease from to . Correspondingly, the N+1 three-dimensional feature information Y1 to Y2 are output by the transformation sub-network 1 to the transformation sub-network N+1. N+1 In the N+1 feature generation network layers, the input of the first feature generation network layer is directly the smallest scale three-dimensional feature information Y N+1 , which corresponds to the output of the last transformation subnetwork (i.e., transformation subnetwork N+1). For each feature generation network layer after the first feature generation network layer, i.e., the mth feature generation network layer, its input includes two branches, one input is the output of the mth transformation subnetwork from the end (i.e., the N+1-mth transformation subnetwork from the positive end), and the other input is the output of the m-1th feature generation network layer. Through this network structure, each time the three-dimensional feature information of the previous scale is upsampled and the three-dimensional feature information of the current scale is input into the feature generation network layer for three-dimensional point-by-point convolution and three-dimensional deconvolution processing, so as to realize the fusion of the three-dimensional feature information of the two scales and finally obtain the target three-dimensional feature information. The target three-dimensional feature information is the fusion of the three-dimensional feature information of each different scale. The target three-dimensional feature information can fuse the shallow semantic information and deep geometric detail information of the image.

[0099] After generating the target three-dimensional feature information, the target three-dimensional feature information is input into the target convolutional network layer for processing, and a target three-dimensional reconstructed image with the same scale as the original two-dimensional image, clear details and clear boundaries can be obtained.

[0100] In an embodiment of the present application, since it is possible to achieve upsampling and fusion processing of three-dimensional feature information of different scales based on a feature generation network including a three-dimensional point-by-point convolution layer and a three-dimensional deconvolution layer, it is possible to accurately generate target three-dimensional feature information, so that a clear target three-dimensional reconstructed image can be accurately obtained based on the target three-dimensional feature information.

[0101] Optionally, the three-dimensional image reconstruction method is applied to three-dimensional reconstruction of X-ray images, and before acquiring the two-dimensional image, further comprises:

[0102] Acquire a preset number of CT sample images from a preset CT image data set;

[0103] Obtaining a preset number of two-dimensional projection images according to the preset number of CT sample images and the DDR projection algorithm;

[0104] Generating a preset number of X-ray sample images according to the preset number of two-dimensional projection images and a preset style transfer algorithm;

[0105] The feature extraction network, the dimensionality transformation network and the image generation network are trained based on the preset number of X-ray sample images to obtain the trained feature extraction network, the trained dimensionality transformation network and the trained image generation network.

[0106] In the embodiment of the present application, the neural network composed of a feature extraction network, a dimensionality transformation network and an image generation network can be referred to as an image three-dimensional reconstruction network. When the image three-dimensional reconstruction method of the embodiment of the present application is specifically applied to the three-dimensional reconstruction of X-ray images, that is, when the two-dimensional image of the embodiment of the present application is an X-ray image, then correspondingly, the image three-dimensional reconstruction network needs to be trained based on X-ray sample images. However, there is currently basically no public X-ray image dataset with a large amount of data, and it is relatively laborious and costly to manually construct an X-ray image dataset. However, there is a CT image dataset with sufficient data in the existing public datasets. Therefore, in the embodiment of the present application, based on the CT image dataset and the style transfer algorithm, X-ray sample images can be constructed, and then based on the constructed X-ray sample images, the image three-dimensional reconstruction network can be accurately trained.

[0107] First, a preset number of CT sample images can be obtained from a preset CT image dataset. The preset number can be determined according to actual training needs, for example, 1,000 images.

[0108] After a preset number of CT sample images are acquired, each CT sample image is processed using a digitally reconstructed radiograph (DRR) projection algorithm, and ultimately a preset number of two-dimensional projection images can be generated.

[0109] After generating the two-dimensional projection images, a preset style transfer algorithm (such as a style transfer neural network obtained through pre-training) can be used based on an actual original X-ray image to perform style transfer processing on each two-dimensional projection image separately, so that each two-dimensional projection image can imitate the characteristics of the original X-ray image and generate a preset number of X-ray sample images.

[0110] After generating a preset number of X-ray sample images, these preset number of X-ray sample images can be input into the image three-dimensional reconstruction network, and the feature extraction network, dimensionality transformation network and image generation network therein can be trained end-to-end to finally obtain a trained image three-dimensional reconstruction network, that is, a trained feature extraction network, a trained dimensionality transformation network and a trained image generation network.

[0111] In the embodiment of the present application, since a preset number of X-ray sample images can be generated conveniently and effectively based on the CT image data set, the DRR projection algorithm and the style transfer algorithm, the feature extraction network, the dimensionality transformation network and the image generation network can be accurately trained based on these X-ray sample images, and finally the trained network is accurately generated, so that the three-dimensional reconstruction of the image can be efficiently and accurately achieved based on these networks in the future.

[0112] For example, assuming that N=3, the image 3D reconstruction network of the embodiment of the present application can be as follows: Figure 5 As shown, the details are as follows:

[0113] The feature extraction network consists of a first CBAM network layer, three ECA network layers, and a second CBAM network layer, connected in sequence. The first CBAM network layer processes the two-dimensional image, generating initial feature information X0. This initial feature information X0 serves as the input to the first ECA network layer, namely, ECA network layer 1. The two-dimensional feature information X1 generated by ECA network layer 1 is divided into two branches. One branch is input to the corresponding transformation subnetwork 1 in the dimensional transformation network for dimensional transformation, and the other branch is input to the next ECA network layer, namely, ECA network layer 2, for further processing. Similarly, after ECA network layer 2 processes the two-dimensional feature information input by ECA network layer 1, the obtained two-dimensional feature information X2 is divided into two branches, one branch is input into the corresponding transformation sub-network 2 in the dimensional transformation network for dimensional transformation processing, and the other branch is input into the next ECA network layer, that is, ECA network layer 3 for processing; after ECA network layer 3 processes the two-dimensional feature information input by ECA network layer 2, the obtained two-dimensional feature information X3 is divided into two branches, one branch is input into the corresponding transformation sub-network 3 in the dimensional transformation network for dimensional transformation processing, and the other branch is input into the next network layer, that is, the second CBAM network layer for processing; the second CBAM network layer processes the two-dimensional feature information input by ECA network layer 3 to obtain two-dimensional feature information X4, and directly inputs it into the transformation sub-network 4 in the dimensional transformation network for processing.

[0114] In the four two-dimensional feature information X1 to X4 output by the feature extraction network layer, the scales change from large to small; correspondingly, the scales of the four three-dimensional feature information Y1 to Y4 output by the dimension transformation network also change from large to small.

[0115] After the three-dimensional feature information Y1-Y4 is input into the image generation network, the smallest-scale three-dimensional feature information Y4 output by the last transformation subnetwork, transformation subnetwork 4, is first input into the first feature generation network layer, namely, feature generation network layer 1, for three-dimensional point-by-point convolution and deconvolution processing, obtaining scale-upgraded three-dimensional feature information Y4'. This Y4' is then input into feature generation network layer 2 along with the three-dimensional feature information Y3 for processing, obtaining scale-upgraded three-dimensional feature information Y3'. This Y3' is then input into feature generation network layer 3 along with the three-dimensional feature information Y2 for processing, obtaining scale-upgraded three-dimensional feature information Y2'. This Y2' is then input into feature generation network layer 4 along with the three-dimensional feature information Y1 for processing, resulting in target three-dimensional feature information whose scale matches the scale of the two-dimensional feature information X0 output by the first CBAM network layer (i.e., the length and width are consistent). This target three-dimensional feature information is then input into the target convolutional network layer for processing, obtaining the final target three-dimensional reconstructed image.

[0116] Through the image 3D reconstruction network, based on lightweight attention modules, residual blocks and multi-scale feature fusion, the accuracy of the reconstructed image is guaranteed while greatly reducing the required computing resources. At the same time, based on the 3D point-by-point convolution layer and 3D deconvolution layer of the feature generation network, the pixel loss in the reconstruction process can be accurately and efficiently improved, thereby realizing efficient and accurate 3D reconstruction of the image and obtaining clear and accurate target 3D reconstructed images.

[0117] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0118] Example 2:

[0119] Figure 3 A schematic diagram of the structure of a three-dimensional image reconstruction device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown:

[0120] The image 3D reconstruction device includes: an acquisition unit 61, a feature extraction unit 62, a dimension transformation unit 63, and a reconstruction unit 64.

[0121] The acquisition unit 61 is configured to acquire a two-dimensional image.

[0122] The feature extraction unit 62 is used to input the two-dimensional image into a trained feature extraction network for processing to obtain two-dimensional feature information corresponding to the two-dimensional image; the feature extraction network includes a convolutional layer.

[0123] The dimension transformation unit 63 is used to input the two-dimensional feature information into the trained dimension transformation network for processing to obtain three-dimensional feature information.

[0124] The reconstruction unit 64 is used to input the three-dimensional feature information into a trained image generation network for processing to obtain a target three-dimensional reconstructed image; wherein the image generation network includes a three-dimensional point-by-point convolution layer and a three-dimensional deconvolution layer in sequence.

[0125] Optionally, the three-dimensional deconvolution layer of the image generation network includes a first preset number of three-dimensional convolution kernels of different scales.

[0126] Optionally, the feature extraction unit 62 is specifically configured to input the two-dimensional image into a trained feature extraction network for processing to obtain two-dimensional feature information of at least two different scales; wherein the feature extraction network includes at least two network layers connected in sequence, and the two-dimensional feature information of different scales corresponds to feature information output by different network layers;

[0127] Correspondingly, the dimension transformation unit 63 is specifically configured to input each of the two-dimensional feature information into a trained dimension transformation network for processing, so as to obtain each of the three-dimensional feature information corresponding to each of the two-dimensional feature information.

[0128] Optionally, the feature extraction network includes a first CBAM network layer, N ECA network layers, and a second CBAM network layer connected in sequence, and the ECA network layer includes a residual block;

[0129] The two-dimensional feature information of different scales includes the N two-dimensional feature information output by the N ECA network layers and the two-dimensional feature information output by the second CBAM network layer; wherein, the input of the first CBAM network layer is the two-dimensional image, the input of the first ECA network layer is the output of the first CBAM network layer, the input of the i-th ECA network layer is the output of the i-1-th ECA network layer, and the input of the second CBAM network layer is the output of the last ECA network layer; N is a positive integer greater than 1, and i is any positive integer greater than 1 and less than or equal to N.

[0130] Optionally, the dimensionality transformation network includes N+1 transformation sub-networks, the input of the j-th transformation sub-network is the output of the j-th ECA network layer, and the output of the j-th transformation sub-network is the three-dimensional feature information converted from the two-dimensional feature information output by the j-th ECA network layer; the input of the last transformation sub-network is the output of the second CBAM network layer, and the output of the last transformation sub-network is the three-dimensional feature information converted from the two-dimensional feature information output by the second CBAM network layer; wherein j is a positive integer less than or equal to N.

[0131] Optionally, the image generation network includes N+1 feature generation network layers and a target convolutional network layer; the input of the first feature generation network layer is the output of the first transformation subnetwork, and the input of the mth feature generation network layer includes the output of the mth transformation subnetwork and the output of the m-1th transformation subnetwork; the input of the target convolutional network layer is the output of the last transformation subnetwork, and the output of the target convolutional network layer is the target three-dimensional reconstructed image; wherein m is any positive integer greater than 1 and less than or equal to N+1.

[0132] Optionally, the three-dimensional image reconstruction method is applied to three-dimensional reconstruction of X-ray images, and the three-dimensional image reconstruction device further includes:

[0133] A training unit is used to obtain a preset number of CT sample images from a preset CT image data set; obtain a preset number of two-dimensional projection images based on the preset number of CT sample images and a DDR projection algorithm; generate a preset number of X-ray sample images based on the preset number of two-dimensional projection images and a preset style migration algorithm; and train a feature extraction network, a dimensionality transformation network, and an image generation network based on the preset number of X-ray sample images to obtain the trained feature extraction network, the trained dimensionality transformation network, and the trained image generation network.

[0134] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of this application. Their specific functions and technical effects can be found in the method embodiment section and will not be repeated here.

[0135] Example 3:

[0136] Figure 7 Schematic diagram of an electronic device provided by an embodiment of the present application. Figure 7 As shown, the electronic device 7 of this embodiment includes: a processor 70, a memory 71, and a computer program 72 stored in the memory 71 and executable on the processor 70, such as a 3D image reconstruction program. When the processor 70 executes the computer program 72, the steps in each of the above-mentioned 3D image reconstruction method embodiments are implemented, such as Figure 1 Alternatively, when the processor 70 executes the computer program 72, the functions of the modules / units in the above-mentioned device embodiments are realized, for example, Figure 6 The functions of the acquisition unit 61 to the reconstruction unit 64 are shown.

[0137] Exemplarily, the computer program 72 may be divided into one or more modules / units, which are stored in the memory 71 and executed by the processor 70 to implement the present application. The one or more modules / units may be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program 72 in the electronic device 7.

[0138] The electronic device 7 can be a computing device such as a desktop computer, a notebook, a PDA, an X-ray machine, etc. The electronic device can include, but is not limited to, a processor 70 and a memory 71. It can be understood by those skilled in the art that Figure 7It is only an example of the electronic device 7 and does not constitute a limitation of the electronic device 7. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

[0139] The processor 70 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0140] The memory 71 may be an internal storage unit of the electronic device 7, such as a hard disk or memory of the electronic device 7. The memory 71 may also be an external storage device of the electronic device 7, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 7. Furthermore, the memory 71 may include both an internal storage unit of the electronic device 7 and an external storage device. The memory 71 is used to store the computer program and other programs and data required by the electronic device. The memory 71 may also be used to temporarily store data that has been output or is about to be output.

[0141] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0142] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0143] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0144] In the embodiments provided in the present application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely schematic. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0145] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0146] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0147] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, it can implement the steps of the above-mentioned various method embodiments. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0148] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A method for three-dimensional image reconstruction, characterized in that: include: Acquire a two-dimensional image; Inputting the two-dimensional image into a trained feature extraction network for processing to obtain two-dimensional feature information corresponding to the two-dimensional image; Inputting the two-dimensional feature information into a trained dimension transformation network for processing to obtain three-dimensional feature information; Inputting the 3D feature information into a trained image generation network for processing to obtain a target 3D reconstructed image; wherein the image generation network sequentially includes a 3D point-by-point convolution layer and a 3D deconvolution layer, wherein the 3D point-by-point convolution layer is used to extract features of a single point, thereby obtaining more detailed feature information while reducing the convolution dimension; Inputting the two-dimensional image into a trained feature extraction network for processing to obtain two-dimensional feature information corresponding to the two-dimensional image includes: Inputting the two-dimensional image into a trained feature extraction network for processing to obtain two-dimensional feature information of at least two different scales; wherein the feature extraction network includes at least two network layers connected in sequence, and the two-dimensional feature information of different scales corresponds to feature information output by different network layers; Correspondingly, the inputting the two-dimensional feature information into the trained dimension transformation network for processing to obtain three-dimensional feature information includes: Inputting each of the two-dimensional feature information into a trained dimension transformation network for processing to obtain each of the three-dimensional feature information corresponding to each of the two-dimensional feature information; The feature extraction network includes a first CBAM network layer, N ECA network layers, and a second CBAM network layer connected in sequence, and the ECA network layer includes a residual block; The two-dimensional feature information of different scales includes the N two-dimensional feature information output by the N ECA network layers and the two-dimensional feature information output by the second CBAM network layer; wherein, the input of the first CBAM network layer is the two-dimensional image, the input of the first ECA network layer is the output of the first CBAM network layer, the input of the i-th ECA network layer is the output of the i-1-th ECA network layer, and the input of the second CBAM network layer is the output of the last ECA network layer; N is a positive integer greater than 1, and i is any positive integer greater than 1 and less than or equal to N; The dimensionality transformation network includes N+1 transformation sub-networks, the input of the j-th transformation sub-network is the output of the j-th ECA network layer, and the output of the j-th transformation sub-network is the three-dimensional feature information converted from the two-dimensional feature information output by the j-th ECA network layer; the input of the last transformation sub-network is the output of the second CBAM network layer, and the output of the last transformation sub-network is the three-dimensional feature information converted from the two-dimensional feature information output by the second CBAM network layer; wherein j is a positive integer less than or equal to N.

2. The method for three-dimensional image reconstruction according to claim 1, wherein: The three-dimensional deconvolution layer of the image generation network includes a first preset number of three-dimensional convolution kernels of different scales.

3. The method for three-dimensional image reconstruction according to claim 1, wherein: The image generation network includes N+1 feature generation network layers and a target convolutional network layer; the input of the first feature generation network layer is the output of the last transformation subnetwork, and the input of the mth feature generation network layer includes the output of the mth to last transformation subnetwork and the output of the m-1th transformation subnetwork; the input of the target convolutional network layer is the output of the last transformation subnetwork, and the output of the target convolutional network layer is the target three-dimensional reconstructed image; wherein m is any positive integer greater than 1 and less than or equal to N+1.

4. The method for three-dimensional image reconstruction according to any one of claims 1 to 3, wherein: The three-dimensional image reconstruction method is applied to the three-dimensional reconstruction of X-ray images, and before acquiring the two-dimensional image, further comprises: Acquire a preset number of CT sample images from a preset CT image data set; Obtaining a preset number of two-dimensional projection images according to the preset number of CT sample images and the DDR projection algorithm; Generating a preset number of X-ray sample images according to the preset number of two-dimensional projection images and a preset style transfer algorithm; The feature extraction network, the dimensionality transformation network and the image generation network are trained based on the preset number of X-ray sample images to obtain the trained feature extraction network, the trained dimensionality transformation network and the trained image generation network.

5. A three-dimensional image reconstruction device, characterized in that: include: an acquisition unit, configured to acquire a two-dimensional image; a feature extraction unit, configured to input the two-dimensional image into a trained feature extraction network for processing, and obtain two-dimensional feature information corresponding to the two-dimensional image; a dimensionality transformation unit, configured to input the two-dimensional feature information into a trained dimensionality transformation network for processing to obtain three-dimensional feature information; a reconstruction unit, configured to input the 3D feature information into a trained image generation network for processing to obtain a target 3D reconstructed image; wherein the image generation network sequentially includes a 3D point-by-point convolution layer and a 3D deconvolution layer, wherein the 3D point-by-point convolution layer is configured to extract features of a single point, thereby obtaining more detailed feature information while reducing the convolution dimension; The feature extraction unit is specifically configured to input the two-dimensional image into a trained feature extraction network for processing to obtain two-dimensional feature information of at least two different scales; wherein the feature extraction network includes at least two network layers connected in sequence, and the two-dimensional feature information of different scales corresponds to feature information output by different network layers; Correspondingly, the dimension transformation unit is specifically used to input each of the two-dimensional feature information into the trained dimension transformation network for processing, so as to obtain each of the three-dimensional feature information corresponding to each of the two-dimensional feature information; The feature extraction network includes a first CBAM network layer, N ECA network layers, and a second CBAM network layer connected in sequence, and the ECA network layer includes a residual block; the two-dimensional feature information of different scales includes N two-dimensional feature information output by the N ECA network layers and two-dimensional feature information output by the second CBAM network layer; wherein, the input of the first CBAM network layer is the two-dimensional image, the input of the first ECA network layer is the output of the first CBAM network layer, the input of the i-th ECA network layer is the output of the i-1-th ECA network layer, and the input of the second CBAM network layer is the output of the last ECA network layer. The output of the CA network layer; N is a positive integer greater than 1, and i is any positive integer greater than 1 and less than or equal to N; the dimensionality transformation network includes N+1 transformation sub-networks, the input of the j-th transformation sub-network is the output of the j-th ECA network layer, and the output of the j-th transformation sub-network is the three-dimensional feature information converted from the two-dimensional feature information output by the j-th ECA network layer; the input of the last transformation sub-network is the output of the second CBAM network layer, and the output of the last transformation sub-network is the three-dimensional feature information converted from the two-dimensional feature information output by the second CBAM network layer; wherein, j is a positive integer less than or equal to N.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the electronic device implements the steps of the method according to any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the electronic device implements the steps of the method according to any one of claims 1 to 4.

Images