Image reconstruction method and apparatus, and electronic device and storage medium

By performing feature encoding and decoding of CT images, image reconstruction is carried out using limited two-dimensional image information, which solves the problem of large calculation and long time in traditional CT image reconstruction, and efficient and accurate three-dimensional image reconstruction is achieved, reducing radiation Hazards to the human body.

WO2024229631A1PCT designated stage expired Publication Date: 2024-11-14SHENZHEN INST OF ADVANCED TECH
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2023/092567
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-06
Publication Date
2024-11-14

AI Technical Summary

Technical Problem

Traditional CT image reconstruction methods require hundreds or thousands of X-ray projection images, resulting in large amounts of calculations, long time, and increased radiation health damage to the human body.

Method used

By obtaining the set number of two-dimensional images of the target, performing feature encoding processing, obtaining the target query vector, decoding to obtain grayscale information, using this information to reconstruct the image, reducing the calculation amount, reducing memory consumption, and achieving efficient reconstruction.

Benefits of technology

It improves the accuracy of image reconstruction, shortens the reconstruction time, meets real-time requirements, and reduces the harm of radiation to the human body.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2023092567_14112024_PF_FP_ABST
    Figure CN2023092567_14112024_PF_FP_ABST
Patent Text Reader

Abstract

The present application relates to the technical field of medical treatments. Provided are an image reconstruction method and apparatus, and an electronic device and a storage medium. The method comprises: acquiring a set number of two-dimensional images of a target; performing feature coding processing on the two-dimensional images to obtain target features; acquiring a plurality of target query vectors used for locating feature points in the target features, and decoding the target features according to the target query vectors, so as to obtain grayscale information, wherein the grayscale information comprises grayscale values corresponding to locations in a reconstructed three-dimensional image; and performing image reconstruction according to the grayscale information, so as to obtain a three-dimensional image of the target. The present application solves the problems in the prior art of a large computation amount in image reconstruction and the long duration thereof.
Need to check novelty before this filing date? Find Prior Art

Description

Image reconstruction method, device, electronic device and storage medium Technical Field

[0001] The present application relates to the field of medical technology, and more specifically, to an image reconstruction method, device, and electronic equipment. Background Art

[0002] With the continuous development of computer technology, image reconstruction techniques are also constantly improving. In medical imaging, CT (Computed Tomography) image reconstruction is a key application. CT is a medical imaging technique that uses X-rays to scan the interior of the human body to obtain information about the body's structure. Due to their high penetrating power, X-rays can penetrate the body, forming X-ray projection images. CT reconstruction technology converts these X-ray projection images into three-dimensional images of the human body's structure.

[0003] Traditional CT image reconstruction methods require a complete rotational scan of the target to obtain hundreds or thousands of X-ray projection images. Image reconstruction is then performed based on these hundreds or thousands of X-ray projection images, resulting in a high-quality three-dimensional image. However, this method of image reconstruction suffers from the large number of X-ray projection images and the computational complexity of image reconstruction, leading to long reconstruction times and high resource usage. Furthermore, the acquisition of X-ray projection images exposes the human body to radiation. The more X-ray projection images required, the greater the radiation exposure, which can cause certain health risks.

[0004] Summary of the Invention

[0005] This application provides an image reconstruction method, device, electronic device, and storage medium, which can solve the problems of large image reconstruction calculation amount and long time in related technologies. The technical solution is as follows:

[0006] According to one aspect of the present application, a set number of two-dimensional images of a target are obtained; feature encoding processing is performed on each of the two-dimensional images to obtain target features; multiple target query vectors for locating feature points in the target features are obtained, and the target features are decoded according to each of the target query vectors to obtain grayscale information; wherein the grayscale information includes grayscale values ​​corresponding to each position in the reconstructed three-dimensional image; image reconstruction is performed based on the grayscale information to obtain a three-dimensional image of the target.

[0007] According to one aspect of the present application, an image acquisition module is used to acquire a set number of two-dimensional images of a target; an image processing module is used to perform feature encoding processing on each of the two-dimensional images to obtain target features; a feature decoding module is used to acquire multiple target query vectors for locating feature points in the target features, and decode the target features according to each of the target query vectors to obtain grayscale information; wherein the grayscale information includes grayscale values ​​corresponding to each position in the reconstructed three-dimensional image; and a three-dimensional reconstruction module is used to perform image reconstruction according to the grayscale information to obtain a three-dimensional image of the target.

[0008] According to one aspect of the present application, an electronic device includes at least one processor and at least one memory, wherein the memory stores program instructions or codes; the program instructions or codes are loaded and executed by the processor, so that the electronic device implements the image reconstruction method as described above.

[0009] According to one aspect of the present application, a storage medium stores program instructions or codes thereon, and the program instructions or codes are loaded and executed by a processor to implement the image reconstruction method as described above.

[0010] According to one aspect of the present application, an application product includes program instructions or codes, the program instructions or codes are stored in a storage medium, and a processor of an electronic device reads the program instructions or codes from the storage medium, loads and executes the program instructions or codes, so that the electronic device implements the image reconstruction method as described above.

[0011] The beneficial effects of the technical solution provided by this application are:

[0012] In the above technical solution, on the one hand, by performing feature encoding processing on each two-dimensional image to obtain target features, the information contained in the two-dimensional image can be fully explored, making the reconstructed three-dimensional image more realistic; on the other hand, the grayscale information obtained by decoding the target features according to each target query vector is reconstructed to obtain a three-dimensional image of the target, which can reduce the amount of calculation and memory consumption, thereby speeding up image reconstruction, achieving efficient image reconstruction, and meeting real-time requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments of the present application.

[0014] FIG1 is a schematic diagram of an implementation environment according to the present application;

[0015] FIG2 is a flow chart showing an image reconstruction method according to an exemplary embodiment;

[0016] FIG3 is a flow chart of step 330 in the embodiment corresponding to FIG2 in one embodiment;

[0017] FIG4 is a schematic structural diagram of an image reconstruction model in an embodiment corresponding to FIG2 ;

[0018] FIG5 is a schematic diagram of a specific implementation of an image reconstruction method in an application scenario;

[0019] FIG5 a is a three-dimensional image obtained by verifying a small-scale dataset of 50 patients in the application scenario of FIG5 ;

[0020] FIG5 b is a three-dimensional image obtained by verification of 1018 patients in the application scenario of FIG5 ;

[0021] Figure 5c shows a 3D image obtained by training the image reconstruction model on the LIDC-IDRI dataset and verifying it on the LNDb dataset;

[0022] FIG6 is a structural block diagram of an image reconstruction apparatus according to an exemplary embodiment;

[0023] FIG7 is a hardware structure diagram of a server according to an exemplary embodiment;

[0024] Fig. 8 is a structural block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0025] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and are not to be construed as limiting the present application.

[0026] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an", "said" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and / or" used herein includes all or any units and all combinations of one or more associated listed items.

[0027] As mentioned above, traditional CT image reconstruction is performed based on hundreds or thousands of X-ray projection images, which is not only time-consuming but also expends a large amount of radiation.

[0028] Currently, image reconstruction models can be trained through machine learning methods, and then used to reconstruct three-dimensional images from X-ray projection images. For example, the X2CT-GAN model can reconstruct CT images from two X-ray projection images through deep learning, combining a two-dimensional U-shaped convolutional network with a three-dimensional U-shaped convolutional network to generate a complete CT image end-to-end.

[0029] Although the above method reduces the requirement on the number of X-ray projection images, the computational cost of image reconstruction using this method is high, and the authenticity of the reconstructed CT image is insufficient.

[0030] As can be seen from the above, the related technologies still have the defects of large image reconstruction calculation amount and long time.

[0031] To this end, the image reconstruction method provided in the present application can effectively improve the accuracy of image reconstruction. Accordingly, the image reconstruction method is applicable to an image reconstruction device, and the image reconstruction device can be deployed in an electronic device configured with a von Neumann architecture, for example, the electronic device can be a desktop computer, a laptop computer, a server, etc.

[0032] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0033] Figure 1 is a schematic diagram of an implementation environment involved in an image processing method. It should be noted that this implementation environment is only an example adapted to the present invention and should not be considered as providing any limitation on the scope of application of the present invention.

[0034] The implementation environment includes a collection end 110 and a service end 130 .

[0035] Specifically, the acquisition end 110 can also be considered as an image acquisition device, including but not limited to electronic devices with shooting functions such as cameras, cameras, and camcorders, and can also be a device with the function of acquiring X-ray projection images. For example, the acquisition end 110 is a computer tomography camera.

[0036] Server 130 can be an electronic device such as a desktop computer, laptop computer, or server. It can also be a computer cluster consisting of multiple servers, or even a cloud computing center consisting of multiple servers. Server 130 is used to provide background services, such as, but not limited to, image reconstruction services.

[0037] A network communication connection is pre-established between the server 130 and the acquisition terminal 110 via a wired or wireless method, and data transmission between the server 130 and the acquisition terminal 110 is achieved via the network communication connection. The transmitted data includes but is not limited to: a two-dimensional image of the target, etc.

[0038] In one application scenario, through the interaction between the acquisition terminal 110 and the server 130 , the acquisition terminal 110 captures and acquires a two-dimensional image of the target, and uploads the two-dimensional image to the server 130 to request the server 130 to provide image reconstruction service.

[0039] After receiving the 2D image of the target uploaded by the acquisition terminal 110, the server 130 invokes the image reconstruction service. It performs feature encoding processing based on multiple 2D images of the target to obtain target features. It then obtains target query vectors, decodes the target features using each query vector, and obtains grayscale information. Finally, it reconstructs the image based on the grayscale information to obtain a 3D image of the target. This solves the problem of high computational complexity and long processing time in related technologies for image reconstruction.

[0040] Please refer to FIG. 2 . An embodiment of the present application provides an image reconstruction method. The method is applicable to an electronic device, which may be the server 130 in the implementation environment shown in FIG. 1 .

[0041] In the following method embodiments, for ease of description, the execution subject of each step of the method is taken as an electronic device as an example for illustration, but this does not constitute a specific limitation.

[0042] As shown in FIG2 , the method may include the following steps:

[0043] Step 310: Acquire a set number of two-dimensional images of the target.

[0044] The target refers to an object or a human body part that needs to be three-dimensionally reconstructed, for example, the target is the human chest cavity.

[0045] The two-dimensional image of the target is obtained by scanning and capturing the target through an image acquisition device, wherein the image acquisition device can be a device capable of capturing X-ray projection images, such as a computer tomography machine.

[0046] In one possible implementation, a computer tomography (CT) camera is used to scan the patient's chest to obtain a plurality of X-ray projection images, wherein the patient's chest is the target and the X-ray projection images are two-dimensional images of the target.

[0047] Regarding the acquisition of two-dimensional images, the images can be derived from two-dimensional images scanned and captured in real time by an image acquisition device, or they can be images pre-stored in the electronic device and scanned and captured by the image acquisition device during a historical time period. Therefore, after the image acquisition device scans and captures the image, the electronic device can process the image in real time, or it can pre-store the image for later processing, for example, processing the two-dimensional image when the electronic device's CPU is low, or processing the two-dimensional image according to instructions from a staff member. Therefore, the image reconstruction in this embodiment can be based on two-dimensional images captured in real time or captured during a historical time period, without specific limitations here.

[0048] It's understandable that the more 2D images referenced during image reconstruction, the richer the information captured from each 2D image, and the better the quality of the reconstructed 3D image. However, the more 2D images, the more computational effort required for image reconstruction, and the longer the reconstruction time required. Therefore, to balance reconstruction time and quality, a set number of 2D images of the target is acquired.

[0049] The set number can be set according to the processing capability of the electronic device. For example, if the processing capability of the electronic device is strong, the set number can be set relatively large. If the processing capability of the electronic device is weak, the set number can be set relatively small. In one possible implementation, at least two two-dimensional images of the target are acquired.

[0050] Further explaining, to extract more information from a limited number of 2D images and reconstruct a higher-quality 3D image, in one possible implementation, each 2D image includes two orthogonal 2D images. It will be appreciated that the information contained in these two orthogonal 2D images complements each other, providing more accurate and comprehensive information about the target's surface and internal structure. Consequently, 3D reconstruction based on these two orthogonal 2D images not only reduces the computational effort and shortens the reconstruction time, but also improves reconstruction efficiency while ensuring uncompromised reconstruction quality.

[0051] Step 330: Perform feature encoding processing on each two-dimensional image to obtain target features.

[0052] Target features are used to indicate information about the target's 2D image. Different 2D images contain different information, and image reconstruction is based on this information. Therefore, before 3D reconstruction, image denoising, feature extraction, position encoding, and feature splicing can be performed on each 2D image to obtain target features, which can then be used for image reconstruction.

[0053] Based on this, in a possible implementation, as shown in FIG3 , step 330 may further include the following steps:

[0054] Step 331 : performing feature fusion on the image features of each two-dimensional image to obtain a first image feature that introduces position information.

[0055] First of all, it should be noted that different two-dimensional images can extract different image features. In one possible implementation, step 331 includes: extracting features from each two-dimensional image separately to obtain image features of each two-dimensional image; splicing the image features of each two-dimensional image and performing position encoding during the splicing process to obtain a first image feature that introduces position information.

[0056] Among them, in one possible implementation, each two-dimensional image can be input into a residual network feature extractor (ResNet-50) for extraction to obtain image features of each two-dimensional image. On the one hand, the image features extracted by ResNet-50 have better expression capabilities. On the other hand, since ResNet-50 has jump connections in the depth direction, feature information can be quickly transmitted, so the calculation speed of ResNe-t50 is relatively fast. Of course, CNN feature extractors, GAN feature extractors, etc. can also be used to extract features from each two-dimensional image, which is not specifically limited here.

[0057] Regarding position coding, it is used to encode the position information of different positions in the image feature to obtain a first image feature that incorporates the position information. The first image feature that incorporates the position information is then used to reconstruct the image. This allows for a better understanding of the relationship between different positions in the first image feature, thereby more accurately reconstructing the image during decoding. In one possible implementation, trigonometric position coding is performed on the image features. Trigonometric position coding can better preserve position information and is more robust.

[0058] It is further noted that different image features can be position-encoded separately to better preserve the image features of each image. Alternatively, different image features can be concatenated and the resulting first image features position-encoded. This concatenated encoding can reduce computational complexity. It should be noted that the choice of position-encoding the image features of each two-dimensional image or the first image features concatenated from each image feature depends on the specific application scenario and requirements.

[0059] In step 333, a multi-head attention mechanism is used to encode the first image feature that introduces the position information to obtain the target feature.

[0060] First, it should be noted that the multi-head attention mechanism can aggregate information between and within two-dimensional images, thereby fully utilizing limited information for image reconstruction. In one possible implementation, a six-layer multi-head attention mechanism can be used to encode the first image feature that introduces position information.

[0061] In one possible implementation, the self-attention mechanism is used to perform intra-image self-attention encoding on the first image feature that introduces position information to obtain the first image feature after self-attention encoding; and the cross-attention mechanism is used to perform inter-image cross-attention encoding on the first image feature after self-attention encoding to obtain the target feature.

[0062] It should be noted that using self-attention and cross-attention to process the first image features can better capture the relationship within the image and the relationship between images, thereby obtaining more information for image reconstruction.

[0063] Step 350 : Acquire multiple target query vectors for locating feature points in the target feature, and decode the target feature according to each target query vector to obtain grayscale information.

[0064] Grayscale information includes the grayscale values ​​corresponding to each location in the reconstructed 3D image. The target query vector is used to locate feature points within the target feature. As can be seen, the target feature consists of several feature points, each with a corresponding grayscale value. Grayscale information is crucial in 2D images, reflecting tissue density and thickness.

[0065] In one possible implementation, the position information in the target feature is processed according to a voxel space search strategy to obtain multiple target query vectors.

[0066] Regarding the voxel space search strategy, since the target feature includes position information, each feature point of the target feature can be mapped into a three-dimensional voxel space based on the position information, and the feature vector corresponding to each feature point is assigned to each voxel corresponding to the three-dimensional voxel space. According to the voxel space search strategy, different voxels are traversed to find feature points, and finally a query vector is generated based on the feature points found.

[0067] It is further explained that if only the voxel space search strategy is used to traverse the three-dimensional voxel space, then the process of obtaining the query vector requires a lot of calculations, which affects the image reconstruction efficiency. Based on this, the principle of sparse sampling can be used to accelerate the voxel space search strategy. In one possible implementation method, sparse sampling is performed on the position information in the target feature to obtain multiple target position information; calculations are performed based on each target position information and the feature points corresponding to each target position information to obtain each target query vector.

[0068] Therefore, obtaining the target query vector through sparse sampling can reduce the time and space complexity of the voxel space search strategy, improve efficiency, and thus speed up image reconstruction.

[0069] Then, after obtaining the target query vector, the target feature can be decoded through the target query vector, and the grayscale information can be output according to the position information of the indicated feature point. For example, for the feature point (x, y, z) indicated by the target query vector, the grayscale value within the 4*4*4 rectangular block is output, where the grayscale value within the 4*4*4 rectangular block corresponding to (x, y, z) is the grayscale information.

[0070] Step 370: Reconstruct the image based on the grayscale information to obtain a three-dimensional image of the target.

[0071] As mentioned above, the grayscale information includes grayscale values ​​corresponding to each position in the reconstructed three-dimensional image. Based on this, image reconstruction can be performed according to the grayscale values ​​corresponding to each position, thereby obtaining a three-dimensional image of the target.

[0072] It should be noted that an image reconstruction model generated by training a neural network model can be used to perform image reconstruction based on each two-dimensional image to obtain a three-dimensional image of the target. Specifically, in one possible implementation, the image reconstruction model includes a feature extraction layer, a Transformer encoding layer, a query vector layer, and a Transformer decoding layer.

[0073] As shown in Figure 4, Figure 4 shows a structural diagram of an image reconstruction model. The feature extraction layer can be used to extract features from a two-dimensional image and to splice the extracted image features to obtain a first image feature; the position encoding layer can be used to position encode the first image feature to obtain a first image feature that introduces position information; the Transformer encoding layer can be used to encode the first image feature that introduces position information using a multi-head attention mechanism to obtain a target feature; the query vector layer can be used to process the position information of the target feature according to a voxel space search strategy to obtain a target query vector; the Transformer decoding layer can be used to decode the target feature according to the target query vector to obtain grayscale information, thereby reconstructing the image based on the grayscale information and outputting a three-dimensional image of the target. Based on this, the Transformer encoder and Transformer decoder can aggregate information between and within images, thereby making full use of limited information for image reconstruction.

[0074] In combination with the above embodiments, on the one hand, by performing feature encoding processing on each two-dimensional image to obtain target features, the information contained in the two-dimensional image can be fully explored, making the reconstructed three-dimensional image more realistic; on the other hand, the grayscale information obtained by decoding the target features according to each target query vector is reconstructed to obtain a three-dimensional image of the target, which can reduce the amount of calculation and memory consumption, thereby speeding up image reconstruction, achieving efficient image reconstruction, and meeting real-time requirements.

[0075] FIG5 is a schematic diagram of a specific implementation of an image reconstruction method in an application scenario, which can be applied to image reconstruction of medical images.

[0076] Taking an X-ray projection image as an example, specifically, as shown in FIG5 , the image reconstruction method may include the following steps:

[0077] Through step 801, the two X-ray projection images are respectively output to the residual network feature extractor to obtain corresponding image features.

[0078] Through step 803, the two image features are spliced ​​and trigonometric function position encoding is performed to obtain a first image feature that introduces position information.

[0079] Through step 805, the first image feature that introduces the position information is input into the Transformer encoding layer, and self-attention encoding and cross-attention encoding are performed to obtain the target feature.

[0080] Through step 807, according to the voxel space search strategy, the target query vector is obtained from the position information of the target feature.

[0081] Through step 809, the target feature is input into the Transformer decoding layer, and the target query vector is input into the Transformer decoding layer for query to obtain grayscale information.

[0082] Through step 811, image reconstruction is performed according to the grayscale information to output a three-dimensional image of the target.

[0083] The present invention was validated on two datasets: a small-scale dataset of 50 patients provided by a hospital, and a large-scale publicly available dataset, LIDC-IDRI. Furthermore, the generalization capability of the algorithm was also validated on the LNDb dataset. The validation results are shown in Figures 5a and 5b.

[0084] FIG5a is a three-dimensional image obtained by verification of a small-scale data set of 50 patients. The left side of FIG5a is a three-dimensional image obtained by using a traditional image reconstruction method to reconstruct hundreds or thousands of X-ray projection images, and the right side of FIG5a is a three-dimensional image obtained by using the present method to reconstruct two X-ray projection images.

[0085] Figure 5b is a three-dimensional image obtained by verification on 1018 patients, wherein the left side of Figure 5b is a three-dimensional image obtained by using a traditional image reconstruction method, which is reconstructed based on hundreds or thousands of X-ray projection images, and the right side of Figure 5b is a three-dimensional image obtained by using this method, which is reconstructed based on two X-ray projection images.

[0086] FIG5c is a three-dimensional image obtained by training the image reconstruction model on the LIDC-IDRI dataset and verifying it on the LNDb dataset. The left side of FIG5c is a three-dimensional image obtained by using the traditional image reconstruction method to reconstruct hundreds or thousands of X-ray projection images, and the right side of FIG5c is a three-dimensional image obtained by using the present method to reconstruct two X-ray projection images.

[0087] It can be verified through observation that although this method only reconstructs images based on two X-ray projection images, the reconstructed three-dimensional image still has good image quality and structural accuracy.

[0088] In this application scenario, the image reconstruction obtained three-dimensional images with high quality and fast speed (a three-dimensional image can be reconstructed in only 40 milliseconds), proving that this method performs well in terms of image quality, structural accuracy and generalization ability. It can be applied to scenarios with requirements for real-time and image quality. For example, it provides an efficient, accurate and practical solution for real-time guided radiotherapy (IGRT).

[0089] The following are embodiments of the apparatus of the present application, which can be used to perform the image reconstruction method involved in the present application. For details not disclosed in the apparatus embodiments of the present application, please refer to the method embodiments of the image reconstruction method involved in the present application.

[0090] Please refer to FIG. 6 . An embodiment of the present application provides an image reconstruction device 900 , including but not limited to: an image acquisition module 910 , an image processing module 930 , a feature decoding module 950 , and a 3D reconstruction module 970 .

[0091] The image acquisition module 910 is used to acquire a set number of two-dimensional images of the target.

[0092] The image processing module 930 is used to perform feature coding processing on each two-dimensional image to obtain target features.

[0093] The feature decoding module 950 is used to obtain multiple target query vectors for locating feature points in the target feature, decode the target feature according to each target query vector, and obtain grayscale information; wherein the grayscale information includes the grayscale value corresponding to each position in the reconstructed three-dimensional image.

[0094] The three-dimensional reconstruction module 970 is used to perform image reconstruction based on the grayscale information to obtain a three-dimensional image of the target.

[0095] It should be noted that, when the image reconstruction device provided in the above embodiment performs image reconstruction, the division of the above functional modules is only used as an example to illustrate. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the image reconstruction device will be divided into different functional modules to complete all or part of the functions described above.

[0096] In addition, the image reconstruction device and the image reconstruction method provided in the above embodiments belong to the same concept, and the specific manner in which each module performs operations has been described in detail in the method embodiments and will not be repeated here.

[0097] Fig. 7 shows a schematic diagram of the structure of a server according to an exemplary embodiment. The server is applicable to the server 130 in the implementation environment shown in Fig. 1 .

[0098] It should be noted that the server is only an example adapted for the present application and cannot be considered to provide any limitation on the scope of use of the present application. The server cannot be interpreted as needing to rely on or necessarily having one or more components in the exemplary server 2000 shown in Figure 7.

[0099] The hardware structure of the server 2000 may vary greatly due to different configurations or performances. As shown in FIG. 7 , the server 2000 includes a power supply 210 , an interface 230 , at least one memory 250 , and at least one central processing unit (CPU) 270 .

[0100] Specifically, the power supply 210 is used to provide operating voltage for each hardware device on the server 2000 .

[0101] The interface 230 includes at least one wired or wireless network interface for interacting with external devices, for example, the interaction between the acquisition terminal 110 and the service terminal 130 in the implementation environment shown in FIG1 .

[0102] Of course, in other examples adapted by this application, the interface 230 may further include at least one serial-to-parallel conversion interface 233, at least one input-output interface 235, and at least one USB interface 237, as shown in FIG7 , which is not specifically limited here.

[0103] The memory 250 serves as a carrier for resource storage and can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon include an operating system 251, application 253 and data 255, etc. The storage method can be temporary storage or permanent storage.

[0104] Among them, the operating system 251 is used to manage and control the various hardware devices and application programs 253 on the server 200 to enable the central processing unit 270 to calculate and process the massive data 255 in the memory 250. It can be Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0105] The application 253 is a program instruction or code that performs at least one specific task based on the operating system 251. It may include at least one module (not shown in FIG. 7 ), each of which may include program instructions or code for the server 2000. For example, the image reconstruction device may be considered as the application 253 deployed on the server 2000.

[0106] The data 255 may be a photo or picture stored in a disk, or a two-dimensional image stored in the memory 250 .

[0107] The central processing unit 270 may include one or more processors and is configured to communicate with the memory 250 via at least one communication bus to read program instructions or codes stored in the memory 250, thereby performing operations and processing on the massive amount of data 255 in the memory 250. For example, the image reconstruction method is completed by the central processing unit 270 reading a series of program instructions or codes stored in the memory 250.

[0108] In addition, the present application can also be implemented through hardware circuits or hardware circuits combined with software. Therefore, the implementation of the present application is not limited to any specific hardware circuits, software, or a combination of the two.

[0109] Please refer to FIG8 . An embodiment of the present application provides an electronic device 4000 . The electronic device 400 may include a desktop computer, a laptop computer, a server, etc.

[0110] In FIG. 8 , the electronic device 4000 includes at least one processor 4001 and at least one memory 4003 .

[0111] Data exchange between the processor 4001 and the memory 4003 can be achieved through at least one communication bus 4002. The communication bus 4002 may include a path for transmitting data between the processor 4001 and the memory 4003. The communication bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus 4002 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in FIG8 , but this does not mean that there is only one bus or one type of bus.

[0112] Optionally, the electronic device 4000 may further include a transceiver 4004, which may be used for data exchange between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the number of transceivers 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present application.

[0113] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0114] The memory 4003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disc storage (including compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program instructions or codes in the form of instructions or data structures and can be accessed by the electronic device 4000, but is not limited to these.

[0115] Program instructions or codes are stored in the memory 4003 , and the processor 4001 can read the program instructions or codes stored in the memory 4003 through the communication bus 4002 .

[0116] When the program instructions or codes are executed by the processor 4001 , the image reconstruction methods in the above-mentioned embodiments are implemented.

[0117] In addition, an embodiment of the present application provides a storage medium on which program instructions or codes are stored. The program instructions or codes are loaded and executed by a processor to implement the image reconstruction method as described above.

[0118] In an embodiment of the present application, an application product is provided, which includes program instructions or codes. The program instructions or codes are stored in a storage medium. The processor of the electronic device reads the program instructions or codes from the storage medium, loads and executes the program instructions or codes, so that the electronic device implements the image reconstruction method described above.

[0119] Compared with related technologies, the target features are obtained by performing feature encoding processing on each two-dimensional image, which can fully explore the information contained in the two-dimensional image and make the reconstructed three-dimensional image more realistic; by using the self-attention mechanism and the cross-attention mechanism for encoding, the information between and within images can be aggregated, so as to make full use of the limited information for image reconstruction; the target query vector is obtained by accelerating the voxel space search strategy through sparse sampling, which can reduce the time and space complexity of the voxel space search strategy and improve efficiency, thereby accelerating image reconstruction; the grayscale information obtained by decoding the target features through each target query vector is reconstructed to obtain a three-dimensional image of the target, which can reduce the amount of calculation and memory consumption, thereby accelerating image reconstruction, achieving efficient image reconstruction, and meeting real-time requirements.

[0120] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0121] The above description is only part of the implementation methods of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. An image reconstruction method, characterized in that: include: acquiring a set number of two-dimensional images of the target; Performing feature coding processing on each of the two-dimensional images to obtain target features; Acquire multiple target query vectors for locating feature points in the target feature, and decode the target feature according to each of the target query vectors to obtain grayscale information; wherein the grayscale information includes grayscale values ​​corresponding to each position in the reconstructed three-dimensional image; Image reconstruction is performed according to the grayscale information to obtain a three-dimensional image of the target.

2. The method according to claim 1, characterized in that The performing feature encoding processing on each of the two-dimensional images to obtain target features includes: Performing feature fusion on the image features of each of the two-dimensional images to obtain a first image feature that introduces position information; The first image feature that introduces the position information is encoded using a multi-head attention mechanism to obtain the target feature.

3. The method according to claim 2, characterized in that The encoding process of the first image feature that introduces the position information by using the multi-head attention mechanism to obtain the target feature includes: Using a self-attention mechanism, the first image feature that introduces the position information is self-attention encoded within the image to obtain a first image feature after self-attention encoding; The cross-attention mechanism is used to perform cross-attention encoding between images on the first image feature after the self-attention encoding to obtain the target feature.

4. The method according to claim 2, characterized in that The step of fusing the image features of each of the two-dimensional images to obtain a first image feature that introduces position information includes: Performing feature extraction on each of the two-dimensional images to obtain image features of each of the two-dimensional images; The image features of each of the two-dimensional images are spliced ​​and position-encoded during the splicing process to obtain the first image feature that introduces position information.

5. The method according to claim 2, characterized in that The obtaining of a plurality of target query vectors for locating feature points in the target feature comprises: The position information in the target feature is processed according to a voxel space search strategy to obtain a plurality of target query vectors.

6. The method according to claim 5, characterized in that The method further comprises: processing the position information in the first image feature according to the voxel space search strategy to obtain a plurality of target query vectors; Sparse sampling is performed on the position information in the target feature to obtain a plurality of target position information, wherein the target position information corresponds to a feature point in the target feature; Calculation is performed based on each of the target position information and feature points corresponding to each of the target position information to obtain each of the target query vectors.

7. The method according to any one of claims 1 to 6, characterized in that: Each of the two-dimensional images includes two orthogonal two-dimensional images.

8. The method according to any one of claims 1 to 6, characterized in that The method further comprises: Reconstructing the image based on each of the two-dimensional images using an image reconstruction model generated by training the neural network model to obtain a three-dimensional image of the target; The image reconstruction model includes a feature extraction layer, a Transformer encoding layer, a query vector layer and a Transformer decoding layer.

9. An image reconstruction device, characterized in that: include: An image acquisition module, used to acquire a set number of two-dimensional images of the target; An image processing module, used for performing feature coding processing on each of the two-dimensional images to obtain target features; A feature decoding module, used to obtain a plurality of target query vectors for locating feature points in the target feature, and decode the target feature according to each of the target query vectors to obtain grayscale information; wherein the grayscale information includes a grayscale value corresponding to each position in the reconstructed three-dimensional image; The three-dimensional reconstruction module is used to perform image reconstruction according to the grayscale information to obtain a three-dimensional image of the target.

10. An electronic device, characterized in that: include: at least one processor and at least one memory, wherein: The memory stores program instructions or codes; The program instructions or codes are loaded and executed by the processor, so that the electronic device implements the image reconstruction method according to any one of claims 1 to 8.

Citation Information

Patent Citations

  • Global matching optimized ultrasound image three-dimension reconstruction method

    CN104306021A

  • Method and system used from human face two-dimensional images to human face three-dimensional reconstruction

    CN107730519A

  • Structured light based three-dimensional image reconstruction method and device and storage medium

    CN108122254A

  • Method and device and equipment for generating CT image and storage medium

    CN109745062A

  • Image matching method based on Transform encoder

    CN115937540A