An image display method, apparatus, storage medium, and electronic device

Abstract

The application discloses an image display method, device, storage medium and electronic equipment, wherein the method comprises the following steps: acquiring a real-time tomographic image of a target object; acquiring a predicted depth image corresponding to the real-time tomographic image based on a preset corresponding relationship between the tomographic image and the depth image; acquiring a current depth image corresponding to a real-time endoscopic image collected by an endoscopic device; acquiring a real-time fusion image of the real-time tomographic image and the real-time endoscopic image based on the predicted depth image and the current depth image, and displaying the real-time fusion image. The application can quickly predict the real-time tomographic image by using the preset corresponding relationship between the tomographic image and the depth image, thereby quickly obtaining the corresponding predicted depth image. Subsequently, the real-time fusion matching can be performed based on the predicted depth image and the current depth image, and the fusion and display of the two-dimensional image (endoscopic image) and the three-dimensional position information (CT image) can be realized in real time and accurately.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of medical image processing, in particular to an image display method, device, storage medium and electronic equipment. BACKGROUND

[0002] Surgery refers to the treatment of a patient's body by a doctor using medical instruments. It is an operation performed on a part of the human body with instruments such as knives, scissors, and needles to maintain the health of the patient. The purpose is to cure or diagnose diseases, such as removing diseased tissue, repairing damage, transplanting organs, and improving the function and appearance of the body. Traditional surgery is limited by factors such as operation method and operation field of view, and cannot ensure success. Therefore, image technology is needed to guide the surgery.

[0003] At present, the existing image-guided surgery navigation device and medical imaging equipment are independent devices in terms of physical location and working time, so their coordinate systems are independent. Moreover, the working time of the two is also quite different, so the state of the human body when medical imaging is performed and the state of the human body when minimally invasive surgery is performed may change due to the long time difference. Moreover, the existing image-guided surgery device cannot transmit the medical imaging coordinate position information of the human body to the surgery navigation device after the patient completes the medical imaging scan. The patient must complete the re-unification calibration of the patient's medical imaging coordinate system and the surgery navigation device coordinate system through an independent position sensor such as an optical position positioning device before the surgery, so as to accurately transmit the medical imaging position information of the patient to the surgery navigation device. This will prolong the preparation time of the surgery when the patient uses the navigation device due to the need to complete the optical position calibration step in advance, and will increase the possibility of optical calibration failure. At the same time, the number of position coordinate information conversions will also increase, which will reduce the accuracy of the final surgery navigation position.

[0004] Therefore, there is an urgent need for an image display method to solve the problem that the existing technology cannot perform real-time and accurate image fusion and display, and thus cannot accurately guide the surgery. SUMMARY

[0005] Therefore, the present application provides an image display method, device, storage medium and electronic equipment, which mainly aims to solve the problem that the current image fusion and display cannot be performed in real time and accurately, and thus cannot accurately guide the surgery.

[0006] To solve the above problems, the present application provides an image display method, characterized in that it comprises:

[0007] obtaining a real-time tomographic image of a target object;

[0008] Based on the preset correspondence between the tomographic image and the depth image, a predicted depth image corresponding to the real-time tomographic image is obtained;

[0009] A current depth image corresponding to a real-time endoscopic image collected by an endoscopic device is obtained.

[0010] Based on the predicted depth image and the current depth image, a real-time fusion image of the real-time tomographic image and the real-time endoscopic image is obtained and displayed.

[0011] Optionally, before obtaining the real-time tomographic image of the target object, the following steps are included:

[0012] Sample three-dimensional images of a plurality of sample objects are obtained.

[0013] Based on the sample three-dimensional images and virtual camera technology, global depth maps of the sample objects are obtained.

[0014] Based on the sample three-dimensional images and the global depth maps, a correspondence between the tomographic image and the depth image is established.

[0015] Optionally, establishing the correspondence between the tomographic image and the depth image based on the sample three-dimensional images and the global depth maps includes:

[0016] Based on the sample three-dimensional images and the global depth maps, a plurality of tomographic images and a plurality of depth images corresponding to the positions of the plurality of tomographic images are obtained.

[0017] The plurality of tomographic images are taken as a training set, the plurality of depth images are taken as a label set, a relationship model is trained, and the trained relationship model is taken as the correspondence between the tomographic image and the depth image.

[0018] Optionally, obtaining the global depth maps of the sample objects based on the sample three-dimensional images and virtual camera technology includes:

[0019] Based on the sample three-dimensional images, centerlines of the sample objects are obtained.

[0020] The virtual camera is placed at the centerline position and random positions outside the centerline for roaming, and virtual endoscopic images of the sample objects are obtained.

[0021] The virtual endoscopic images are subjected to depth estimation, and the global depth maps are obtained.

[0022] Optionally, obtaining the real-time fusion image of the real-time tomographic image and the real-time endoscopic image based on the predicted depth image and the current depth image includes:

[0023] perform global pixel-level registration based on the predicted depth image and the current depth image to obtain deformation field information;

[0024] perform fusion of the real-time tomographic image and the real-time endoscopic image based on the deformation field information to obtain a real-time fused image.

[0025] Optionally, the image display method further comprises:

[0026] reconstruct a target three-dimensional image of the target object based on the real-time tomographic image;

[0027] obtain position information of the real-time tomographic image on the target three-dimensional image based on the real-time tomographic image and the target three-dimensional image;

[0028] based on the position information, mark and display a position of the real-time fused image on the target three-dimensional image.

[0029] Optionally, the image display method further comprises:

[0030] obtain real-time pose information of the endoscope or the surgical instrument based on the real-time tomographic image;

[0031] display the real-time pose information on the target three-dimensional image.

[0032] To solve the above problems, the present application provides an image display device, comprising:

[0033] a first obtaining module configured to obtain a real-time tomographic image of a target object;

[0034] a prediction module configured to obtain a predicted depth image corresponding to the real-time tomographic image based on a preset corresponding relationship between tomographic images and depth images;

[0035] a second obtaining module configured to obtain a current depth image corresponding to a real-time endoscopic image collected by an endoscopic device;

[0036] a display module configured to obtain a real-time fused image of the real-time tomographic image and the real-time endoscopic image based on the predicted depth image and the current depth image, and perform display.

[0037] To solve the above problems, the present application provides a storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the image display method described in any of the above.

[0038] To solve the above problems, the application provides an electronic device, at least comprising a memory and a processor, the memory stores a computer program, and the processor implements the steps of the image display method of any one of the above when executing the computer program on the memory.

[0039] The image display method, device, storage medium and electronic device in the application can quickly predict the real-time tomographic image by using the preset corresponding relationship between the tomographic image and the depth image, thereby quickly obtaining the corresponding predicted depth image, and then can perform real-time fusion and matching based on the predicted depth image and the current depth image, thereby realizing real-time, accurate and intuitive fusion and display of the two-dimensional image (endoscope image) and the three-dimensional position information (CT image). Therefore, the doctor can be assisted to accurately plan the surgical path, thereby laying a foundation for realizing accurate guided surgery.

[0040] The above description is only a summary of the technical scheme of the application, in order to more clearly understand the technical means of the application, the application can be implemented according to the content of the specification, and in order to make the above and other purposes, characteristics and advantages of the application more obvious and easy to understand, the following specific embodiments of the application are described. BRIEF DESCRIPTION OF DRAWINGS

[0041] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The accompanying drawings are included to provide a description of the preferred embodiments and are not meant to limit the present application. Furthermore, the same reference numerals are used throughout the several drawings to refer to same or like parts. In the drawings:

[0042] Figure 1 A flow chart of an image display method according to an embodiment of the application;

[0043] Figure 2 A principle block diagram of an image display method according to an embodiment of the application;

[0044] Figure 3 A structure block diagram of an image display device according to another embodiment of the application;

[0045] Figure 4 A structure block diagram of an electronic device according to another embodiment of the application. DETAILED DESCRIPTION

[0046] The various aspects and features of the application are described herein with reference to the accompanying drawings.

[0047] It should be understood that various modifications can be made to the embodiments described herein. Therefore, the above description should not be considered to be limiting, but merely as an example of the embodiments. Other modifications within the scope and spirit of the application will occur to those skilled in the art.

[0048] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the application and, together with the general description of the application given above, and the detailed description of the embodiments given below, serve to explain the principles of the present application.

[0049] These and other characteristics of the present application will become apparent from the following description of the preferred forms of the application given, by way of non-limiting example, with reference to the accompanying drawings.

[0050] It should also be understood that, although the present application has been described above with reference to particular means, materials and embodiments, the present application is by no means limited to the particulars described and as such extends to all alternative constructions falling within the scope of the application.

[0051] The above and other aspects, features and advantages of the present application will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, when considered in conjunction with the following detailed description.

[0052] Specific embodiments of the present application are described hereinafter, by way of non-limiting example; however, it should be understood that the described embodiments are merely examples of the present application, which can be implemented in numerous ways. Functionally similar or equivalent components have not been described in detail, in order to avoid obscuring the present application unnecessarily or superfluously. Therefore, specific structural and functional details disclosed herein are not intended to limit the present application, but merely as a basis for the patent claims and a representative basis for teaching one of ordinary skill in the art to variously employ the present application in virtually any appropriate detail.

[0053] The specification can use phrases like "in one embodiment", "in another embodiment", "in yet another embodiment", or "in other embodiments", which can refer to one or more of the same or different embodiments under the present application.

[0054] Embodiments of the present application provide an image display method, as shown in the accompanying drawings, comprising the following steps: Figure 1 The method comprises the following steps:

[0055] In step S101, a real-time tomographic image of a target object is acquired.

[0056] In this step, the target object refers to an organ object or a part object, etc., which can be an organ, a part, a tissue, etc., in which a region to be operated is located, etc., such as the right coronary artery, the anterior descending branch, the circumflex branch, etc., in the heart coronary artery. The human organs and parts can be divided according to actual needs, thereby obtaining a plurality of organ objects or part objects.

[0057] In step S102, a predicted depth image corresponding to the real-time tomographic image is acquired based on a preset correspondence between the tomographic image and the depth image.

[0058] The preset correspondence between the tomographic image and the depth image in this step can be a depth map prediction model, that is, a preset model is used to predict the real-time tomographic image to obtain a predicted depth map.

[0059] In step S103, a current depth image corresponding to the real-time endoscopic image collected by the endoscopic device is obtained.

[0060] In the specific implementation process of this step, the endoscopic device has a camera, which can be a normal monocular camera endoscope. Therefore, the camera can be used to take real-time videos during the operation to obtain video images. Then, a preset image depth method is used to process the video images to obtain a current depth image based on an RGB image depth map. The specific preset image depth method can be an image SFM depth method. In the specific implementation process of this step, a lens with RGB-D can also be used to directly obtain the depth image of the endoscopic device.

[0061] In step S103, based on the predicted depth image and the current depth image, a real-time fusion image of the real-time tomographic image and the real-time endoscopic image is obtained and displayed.

[0062] After the image fusion is completed in this step, the doctor can intuitively determine the lesion position and plan the surgical path based on the fusion image.

[0063] Taking the surgical guidance scene of the cooperation of the CT device and the endoscopic device as an example, in the specific implementation process of this step, the current depth map and the predicted depth image can be matched and processed in a pixel-level global matching manner to determine the position information of the current depth map in the global depth map. Then, the position information of the surgical instrument is determined through real-time CT scanning during the operation, so that the doctor can track the surgical incision position. The doctor can more accurately and intuitively plan and judge the surgical path based on the real-time provided position information of the surgical instrument and the fusion image of the endoscopic device and the CT image.

[0064] The image display method in this embodiment can quickly predict the real-time tomographic image by using the preset correspondence between the tomographic image and the depth image, thereby quickly obtaining the corresponding predicted depth image. Subsequently, real-time fusion matching can be performed based on the predicted depth image and the current depth image, and the fusion and display of the two-dimensional image (endoscopic image) and the three-dimensional position information (CT image) can be realized in real time, accurately and intuitively. Therefore, the doctor can be assisted to accurately plan the surgical path. Real-time CT scanning can also be used for tracking the position of the surgical instrument to confirm and plan the deviation of the surgical path.

[0065] Another embodiment of the present application provides an image display method, comprising:

[0066] Step S201, obtaining sample three-dimensional images of a plurality of sample objects

[0067] In the implementation process, the sample CT tomographic images of the sample personnel can be obtained in advance, i.e., sample CT images. Then, each sample CT image is segmented to obtain a sub-sample CT image corresponding to each part object. Specifically, different organs and tissues are classified, and a specific organ or tissue is segmented from the image sequence. In the implementation process, the following CT image segmentation methods can be used: region growing segmentation algorithm, threshold segmentation algorithm, deep learning-based segmentation algorithm, etc. After obtaining a plurality of sub-sample CT images, three-dimensional reconstruction is performed on each sub-sample CT image to obtain a sample three-dimensional image. The three-dimensional reconstruction can be realized by using an open source toolkit and commercial software of each company.

[0068] Step S202, obtaining global depth maps of each sample object based on each sample three-dimensional image and virtual camera technology

[0069] In the implementation process, after obtaining the sample three-dimensional images, the depth maps can be collected based on the sample three-dimensional images / three-dimensional models obtained by three-dimensional reconstruction. In the depth map collection, the following methods can be used: obtaining the center lines of each sample object based on each sample three-dimensional image; placing a virtual camera at the center line position and a random position outside the center line to roam, obtaining a virtual endoscope image of each sample object; performing depth estimation on the virtual endoscope image to obtain each global depth map.

[0070] Specifically, the virtual camera can be used to roam in the three-dimensional reconstruction structure to obtain the internal scene image of the tissue or organ and the corresponding depth of field information, i.e., to obtain the labeled depth image (global depth map). Further, a CT image and RGB-D depth image dataset can be constructed, which lays a foundation for subsequent training of the correspondence relationship (depth map model) between the tomographic image and the depth image based on the dataset. In this step, the virtual camera roaming technology can be used to roam along the center line inside the organ in some regions of interest or key zones of the lesion. In order to ensure the richness of the dataset, the virtual camera can be placed at the center line position and the random position in combination in this step. Thus, the virtual image / labeled depth image of the organ can be obtained in multiple directions, multiple degrees of freedom, and multiple perspectives.

[0071] Step S204, establishing a correspondence relationship between the tomographic image and the depth image based on each sample three-dimensional image and each global depth map

[0072] In the specific implementation process of this step, the plurality of tomographic images and the plurality of depth images corresponding to the positions of the plurality of tomographic images can be obtained based on the plurality of sample three-dimensional images and the plurality of global depth images; then the plurality of tomographic images are taken as a training set, the plurality of depth images are taken as a label set, the relationship model is trained, and the trained relationship model is taken as the corresponding relationship between the tomographic image and the depth image. That is, based on the tomographic image and the depth image corresponding to the tomographic image, the initial encoder-decoder (encoding-decoding network) model is trained to obtain the target encoder-decoder (encoding-decoding network) model, so as to obtain the relationship model, and the model can also be called the target depth image prediction model.

[0073] In the specific implementation process of this step, the network structure of the model adopts an encoder-decoder network (encoding-decoding network) with a skip connection. The encoder-decoder network (encoding-decoding network) with a skip connection is used to fuse the features between multiple layers, and a typical example is a U-Net network. The depth error loss and the structural similarity loss. Let y' represent the model estimate value, and y represent the true depth value. The adopted Loss function is shown in the following formula: L1 is the root mean square error loss function of the depth image, L2 is the structural similarity loss error. L is the total loss function.

[0074]

[0075]

[0076] L = L2 + λL1

[0077] Step S205, obtaining a real-time tomographic image of a target object;

[0078] Step S206, based on the preset corresponding relationship between the tomographic image and the depth image, obtaining a predicted depth image corresponding to the real-time tomographic image;

[0079] In the specific implementation process of this step, the real-time tomographic image can be first subjected to three-dimensional reconstruction processing to obtain the target three-dimensional image. Three-dimensional reconstruction can be realized by using an open source tool kit and commercial software of various companies. Then, the depth image of the target object is predicted by using the corresponding relationship (i.e., the target depth image prediction model), and the predicted depth image of the target object is obtained.

[0080] In the process of obtaining the depth image, the depth of the image in the virtual style can be estimated by using a deep neural network to regress the pixels, so as to obtain the predicted depth image of the target object.

[0081] In step S207, a global pixel-level registration is performed based on the predicted depth image and the current depth image to obtain deformation field information; and based on the deformation field information, the real-time tomographic image and the real-time endoscopic image are fused to obtain a real-time fused image, which is displayed.

[0082] In this step, during surgery, the real-time endoscopic image and the real-time CT image are fused together by the fusion technology based on the real-time image feedback of the endoscopic device, and the fused image (or a single image, such as the endoscopic image or the CT image) is displayed on the display device, so that the surgeon can easily find the corresponding surgical point and the corresponding surgical path from the intraoperative fused image; during surgery, the spatial position information of the surgical instrument can be obtained by real-time CT scanning, so as to accurately judge the positional error between the surgical instrument and the lesion, and then guide the surgery.

[0083] In the embodiment, based on the real-time tomographic image and the target three-dimensional image, the position information of the real-time tomographic image on the target three-dimensional image is obtained; and based on the position information, the position of the real-time fused image on the target three-dimensional image is marked and displayed.

[0084] Specifically, the display method in the embodiment can also obtain the real-time pose information of the endoscope or the surgical instrument based on the real-time tomographic image; and display the real-time pose information on the target three-dimensional image.

[0085] After obtaining the fused image of the current depth image and the predicted depth image, the embodiment can further determine the current position information of the surgical instrument based on the real-time CT scan image, or obtain the pose information of the endoscope or the surgical instrument based on the positioning unit or other positioning module of the endoscope or the surgical instrument, and then determine the target surgical path based on the current position information and the target position of the lesion in the fused image, so as to determine the offset of the planned path. Specifically, the target surgical path and the offset of the planned path can be prompted. For example, the deviation of the position and the planned path can be prompted according to a predetermined prompt mode to guide the surgery. For example, the deviation of the planned path can be displayed by using the display device, or the deviation of the planned path can be prompted by voice by using the voice broadcast mode, etc.

[0086] The process of the image display method in the embodiment can be combined with the process of the image fusion method. Figure 2As shown in the figure. The embodiment acquires a video image by using a common camera of an endoscope device and a CT image for real-time fusion. The embodiment makes up for the problem of inaccurate guiding position caused by guiding the operation in real time only by using the common camera image. Moreover, the method in the embodiment can also display the lesion position by using the rich content information in the CT image. The application can overcome deformation and improve the guiding accuracy of the operation, and can improve the real-time and accuracy of the operation.

[0087] Another embodiment of the present application provides an image display device, which comprises Figure 3 As shown in the figure, the image display device 1 in the embodiment comprises:

[0088] A first acquisition module 11 is configured to acquire a real-time tomographic image of a target object.

[0089] A prediction module 12 is configured to acquire a predicted depth image corresponding to the real-time tomographic image based on a preset corresponding relationship between a tomographic image and a depth image.

[0090] A second acquisition module 13 is configured to acquire a current depth image corresponding to a real-time endoscopic image collected by an endoscope device.

[0091] A display module 14 is configured to acquire a real-time fusion image of the real-time tomographic image and the real-time endoscopic image based on the predicted depth image and the current depth image, and display the real-time fusion image.

[0092] In the specific implementation process of the embodiment, the image display device further comprises an establishing module, and the establishing module comprises: a first acquisition subunit, configured to acquire sample three-dimensional images of a plurality of sample objects before acquiring a real-time tomographic image of a target object; a second acquisition subunit, configured to acquire global depth images of the sample objects based on the sample three-dimensional images and virtual camera technology; and an establishing subunit, configured to establish a corresponding relationship between a tomographic image and a depth image based on the sample three-dimensional images and the global depth images.

[0093] In the specific implementation process of the embodiment, the establishing subunit is specifically configured to: acquire a plurality of tomographic images and a plurality of depth images corresponding to positions of the plurality of tomographic images based on the sample three-dimensional images and the global depth images; use the plurality of tomographic images as a training set and the plurality of depth images as a label set to train a relationship model, and use the trained relationship model as the corresponding relationship between the tomographic image and the depth image.

[0094] In the implementation process of the embodiment, the second acquisition subunit is specifically configured to: acquire a center line of each sample object based on each sample three-dimensional image; place a virtual camera at a center line position and a random position outside the center line to perform roaming, and obtain a virtual endoscope image of each sample object; and perform depth estimation on the virtual endoscope image to obtain each global depth map.

[0095] In the implementation process of the embodiment, the pre-display module includes a registration subunit and a fusion subunit; the registration subunit is configured to perform global pixel-level registration based on the predicted depth image and the current depth image to acquire deformation field information; and the fusion subunit is configured to perform fusion of the real-time tomographic image and the real-time endoscopic image based on the deformation field information to obtain a real-time fusion image.

[0096] In the implementation process of the embodiment, the image display device further includes a position information acquisition module configured to: obtain a target three-dimensional image of a target object based on real-time tomographic image reconstruction; acquire position information of the real-time tomographic image on the target three-dimensional image based on the real-time tomographic image and the target three-dimensional image; and display the position of the real-time fusion image on the target three-dimensional image based on the position information.

[0097] In the implementation process of the embodiment, the image display device further includes a position information acquisition module configured to: obtain real-time position information of the endoscope or surgical instrument based on the real-time tomographic image; and display the real-time position information on the target three-dimensional image.

[0098] In the implementation process of the embodiment, the corresponding relationship between the preset tomographic image and the depth image is used to quickly predict the real-time tomographic image, so that the corresponding predicted depth image is quickly obtained. Subsequently, real-time fusion matching can be performed based on the predicted depth image and the current depth image, and the fusion and display of the two-dimensional image (endoscopic image) and the three-dimensional position information (CT image) can be realized in real time, accurately and intuitively. Therefore, the doctor can be assisted in accurately planning the surgical path, and a foundation for realizing accurate guided surgery is laid.

[0099] Another embodiment of the present application provides a system for guiding surgery, comprising:

[0100] A CT scanning system is configured to acquire preoperative and intraoperative CT images. The data of preoperative scanning can be reconstructed, stored and displayed in three dimensions by a CT imaging machine and image processing software. The CT scanning system includes a main control console, an imaging machine and a graphic workstation. The system can segment and reconstruct organs or tissues from a CT image sequence to obtain a three-dimensional RGB image of the CT image and center line information of the scanning position.

[0101] a positioning module, which is capable of acquiring pose information of surgical instruments and endoscopic equipment during intraoperative CT scanning and transmitting the information to a data processing module.

[0102] an endoscopic equipment, which is used to acquire a video image in a surgical human body in real time based on a camera, and then send the video image to the data processing module, so as to process the video image by using the data processing model to obtain a depth map of an RGB image, that is, a current depth map. The depth map / current depth map of the endoscopic equipment can also be directly obtained by using a lens with RGB-D.

[0103] a data processing module, which is used to perform depth map prediction on an intraoperative CT scan image based on a depth map prediction model to obtain a predicted depth map (global depth map), and is also used to process a video image sent by the endoscopic equipment based on an image SFM depth algorithm to obtain a current depth map, or directly receive the current depth map sent by the endoscopic equipment, and is also used to perform pixel-level global matching on the current depth map and the predicted depth map to obtain deformation field information, and perform fusion of the real-time tomographic image and the real-time endoscopic image based on the deformation field information to obtain a real-time fusion image.

[0104] a display module, which is used to display the intraoperative CT image and the image obtained by fusion and matching of the current depth map and the predicted depth map, and is also used to display three-dimensional information of surgical instruments and a human body, and the like.

[0105] The embodiment can compensate for the problem of inaccurate guiding position caused by real-time guiding of surgery only by using a common camera image, and the method in the embodiment can also display a lesion position by using rich content information in a CT image. The application can overcome deformation and improve guiding precision of surgery, and can improve real-time and accuracy of surgery.

[0106] Another embodiment of the application provides a storage medium, which stores a computer program, and the computer program is executed by a processor to implement the following method steps:

[0107] Step 1: acquiring a real-time tomographic image of a target object;

[0108] Step 2: acquiring a predicted depth image corresponding to the real-time tomographic image based on a preset corresponding relationship between a tomographic image and a depth image;

[0109] Step 3: acquiring a current depth image corresponding to a real-time endoscopic image collected by endoscopic equipment;

[0110] Step four, based on the predicted depth image and the current depth image, obtaining a real-time fusion image of the real-time tomographic image and the real-time endoscopic image, and displaying.

[0111] The specific implementation process of the above method steps can be referred to the embodiments of any of the above image display modes, which will not be repeated here.

[0112] The storage medium in the present application can quickly predict the real-time tomographic image by using the preset corresponding relationship between the tomographic image and the depth image, thereby quickly obtaining the corresponding predicted depth image. Subsequently, real-time fusion matching can be performed based on the predicted depth image and the current depth image, and the fusion and display of the two-dimensional image (endoscopic image) and the three-dimensional position information (CT image) can be realized in real time, accurately and intuitively. Therefore, the doctor can be assisted to accurately plan the surgical path, which lays a foundation for realizing the accurate guided surgery.

[0113] Another embodiment of the present application provides an electronic device, as shown in the figure, at least comprising a memory 1 and a processor 2, the memory 1 stores a computer program, and the processor 2 realizes the following method steps when executing the computer program on the memory 1. Figure 4 The specific implementation process of the above method steps can be referred to the embodiments of any of the above image display modes, which will not be repeated here.

[0114] Step one, obtaining a real-time tomographic image of a target object;

[0115] Step two, based on the preset corresponding relationship between the tomographic image and the depth image, obtaining a predicted depth image corresponding to the real-time tomographic image;

[0116] Step three, obtaining a current depth image corresponding to a real-time endoscopic image collected by an endoscopic device;

[0117] Step four, based on the predicted depth image and the current depth image, obtaining a real-time fusion image of the real-time tomographic image and the real-time endoscopic image, and displaying.

[0118] The specific implementation process of the above method steps can be referred to the embodiments of any of the above image display modes, which will not be repeated here.

[0119] The electronic device in the present application can quickly predict the real-time tomographic image by using the preset corresponding relationship between the tomographic image and the depth image, thereby quickly obtaining the corresponding predicted depth image. Subsequently, real-time fusion matching can be performed based on the predicted depth image and the current depth image, and the fusion and display of the two-dimensional image (endoscopic image) and the three-dimensional position information (CT image) can be realized in real time, accurately and intuitively. Therefore, the doctor can be assisted to accurately plan the surgical path, which lays a foundation for realizing the accurate guided surgery.

[0120] The above examples are only exemplary embodiments of the present application, and are not intended to limit the present application, and the protection scope of the present application is defined by the claims. Those skilled in the art can make various modifications or equivalent replacements to the present application within the spirit and protection scope of the present application, and such modifications or equivalent replacements are also considered to fall within the protection scope of the present application.

Claims

1. An image display method, characterized in that, include: Acquire real-time tomographic images of the target object; Based on a preset correspondence between tomographic images and depth images, a predicted depth image corresponding to the real-time tomographic image is obtained; wherein, the correspondence is obtained by training multiple tomographic images and multiple depth images corresponding to the positions of the multiple tomographic images. Acquire the current depth image corresponding to the real-time endoscopic image captured by the endoscopic device; Based on the predicted depth image and the current depth image, a real-time fused image of the real-time tomographic image and the real-time endoscopic image is obtained and displayed.

2. The image display method according to claim 1, characterized in that, Before acquiring real-time tomographic images of the target object, the following steps are included: Obtain three-dimensional images of several sample objects; Based on the 3D images of each sample and virtual camera technology, obtain the global depth map of each sample object; Based on the three-dimensional images of each sample and the global depth map of each sample, a correspondence between the tomographic image and the depth image is established.

3. The image display method according to claim 2, characterized in that, Based on the three-dimensional images of each sample and the global depth maps, the correspondence between the tomographic images and the depth images is established, including: Based on the three-dimensional images of each sample and the global depth map, multiple tomographic images and multiple depth images corresponding to the positions of the multiple tomographic images are obtained. The multiple tomographic images are used as a training set, and the multiple depth images are used as a label set to train the relation model. The trained relation model serves as the correspondence between the tomographic images and the depth images.

4. The image display method according to claim 2, characterized in that, Based on the 3D images of each sample and virtual camera technology, a global depth map of each sample object is obtained, including: Based on the three-dimensional images of each sample, obtain the centerline of each sample object; The virtual camera is placed at the center line position and at random positions outside the center line to roam and obtain virtual endoscopic images of each of the sample objects. Depth estimation is performed on the virtual endoscope image to obtain each of the global depth maps.

5. The image display method according to claim 1, characterized in that, Based on the predicted depth image and the current depth image, a real-time fused image of the real-time tomographic image and the real-time endoscopic image is obtained, including: Global pixel-level registration is performed based on the predicted depth image and the current depth image to obtain deformation field information; Based on the deformation field information, the real-time tomographic image and the real-time endoscopic image are fused to obtain a real-time fused image.

6. The image display method according to claim 1, characterized in that, The method also includes: A target 3D image of the target object is obtained by real-time tomographic image reconstruction. Based on the real-time tomographic image and the target 3D image, obtain the position information of the real-time tomographic image on the target 3D image; Based on the location information, the position of the real-time fused image on the target 3D image is marked and displayed.

7. The image display method according to claim 6, characterized in that, The method also includes: Based on the real-time tomographic images, the real-time pose information of the endoscopic device or surgical instrument is obtained; The real-time pose information is displayed on the target 3D image.

8. An image display device, comprising: The first acquisition module is used to acquire real-time tomographic images of the target object; The prediction module is used to obtain a predicted depth image corresponding to the real-time tomographic image based on a preset correspondence between tomographic images and depth images; wherein the correspondence is obtained by training multiple tomographic images and multiple depth images corresponding to the positions of the multiple tomographic images. The second acquisition module is used to acquire the current depth image corresponding to the real-time endoscopic image collected by the endoscopic device; The display module is used to acquire and display a real-time fused image of the real-time tomographic image and the real-time endoscopic image based on the predicted depth image and the current depth image.

9. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the image display method steps described in any one of claims 1-7.

10. An electronic device, characterized in that, It includes at least a memory and a processor, wherein the memory stores a computer program, and the processor, when executing the computer program in the memory, implements the steps of the image display method described in any one of claims 1-7.

Images