System and method for processing 3D image, and method for rendering 3D image

Abstract

A method and system for processing a three-dimensional image are provided. The method comprises: pre-processing at least one original image set, each of the original image sets comprising a plurality of two-dimensional source images, each of the two-dimensional source images representing a cross-sectional view of a three-dimensional object at a different position along an axis in a three-dimensional space; and constructing a three-dimensional image representing the three-dimensional object by gathering the 3D grid points extracted from the at least one original image set in preprocessing, wherein the three-dimensional image can be read by a first image viewer, and the first image viewer is used for rendering the three-dimensional image and facilitating operation of the three-dimensional image.

Description

Technical Field

[0001] The present disclosure relates to a system and method for processing 3D images, and in particular, but not exclusively, to a system and method for constructing a 3D image or model using a high-performance image processing pipeline. Background Art

[0002] Augmented Reality (AR) or Mixed Reality (MR) technology can be used in healthcare and medical applications, including providing supplemental information to surgeons performing surgery on patients. In some examples, a three-dimensional (3D) model or image of a target object can be displayed to a surgeon or assistant in an operating room, or provided to a clinician as a medical diagnostic analysis tool.

[0003] However, the inclusion of these additional images is a complex process that may require many hours of processing by medically trained personnel to prepare the images to obtain data useful to the surgeon. In order to ensure that the created 3D model is consistent with the actual object / target being operated on, additional information extracted from medical diagnostic scans obtained using various medical imaging systems may need to be accurately processed and assembled. This further increases the time and resources required to prepare the necessary models before AR or MR can be used during medical treatment. Summary of the invention

[0004] According to a first aspect of the present disclosure, a three-dimensional image processing method is provided. The method comprises: preprocessing at least one original image set, each of the original image sets comprising a plurality of two-dimensional source images, wherein each of the two-dimensional source images represents a cross-sectional view of a three-dimensional object at different positions along an axis in a three-dimensional space; and constructing a three-dimensional image representing the three-dimensional object by aggregating 3D grid points extracted from the at least one original image set in the preprocessing; wherein the three-dimensional image can be read by a first image viewer, which is used to render the three-dimensional image and facilitate the operation of the three-dimensional image.

[0005] According to a first aspect, the plurality of two-dimensional source images comprises medical diagnostic scan images.

[0006] According to the first aspect, each of the at least one original image set includes a plurality of medical diagnostic scan images obtained by a selected one of computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET) and X-ray imaging.

[0007] According to the first aspect, the medical diagnostic scan image is readable by a second image viewer.

[0008] According to a first aspect, the step of constructing the three-dimensional image comprises the step of embedding the plurality of two-dimensional source images into the three-dimensional image at corresponding positions along the axis.

[0009] According to a first aspect, the first image viewer is for presenting a cross-sectional view of the three-dimensional object embedded with a two-dimensional view of the cross section reproduced based on the medical diagnostic scan image captured at the corresponding position.

[0010] According to the first aspect, the two-dimensional view of the cross section is substantially equal to a corresponding medical diagnostic scan image read by the second image viewer.

[0011] According to the first aspect, the method further comprises the step of mapping the plurality of two-dimensional source images and the positions of the corresponding cross sections in the three-dimensional object along the axis.

[0012] According to the first aspect, the method further comprises the step of identifying and segmenting a plurality of components of different attributes or properties in the three-dimensional object.

[0013] According to a first aspect, the plurality of components comprises bone, soft tissue, fluid or an implant.

[0014] According to a first aspect, the three-dimensional object comprises at least a portion of an organ, one or more organs, or a combination thereof of a living organism.

[0015] According to a first aspect, identifying a plurality of components includes facilitating manual concealment of one or more of the plurality of components.

[0016] According to a first aspect, said pre-processing said at least one original image set comprises: before the step of identifying and segmenting said plurality of components, a step of performing image optimization processing to optimize said at least one original image set.

[0017] According to a first aspect, the image optimization process comprises an image fusion process for fusing the plurality of two-dimensional source images obtained in different original image sets.

[0018] According to a first aspect, the image optimization process includes an interpolation process for generating one or more augmented images representing cross-sectional views of the three-dimensional object acquired at a location between two adjacent two-dimensional source images in the corresponding original image set captured along an axis.

[0019] According to the first aspect, the method further comprises: identifying and segmenting different sets of original images obtained by different medical diagnostic scanning methods.

[0020] According to a second aspect of the present disclosure, a method for presenting a three-dimensional image constructed by the method described in the first aspect is provided, characterized in that it includes: presenting the three-dimensional image on a display module, wherein the display module is provided with a user interface for facilitating the operation of the three-dimensional image.

[0021] According to a second aspect, the manipulation of the three-dimensional image comprises at least one of scaling, rotating, translating the three-dimensional image and / or moving along the axis a cross-sectional plane for segmenting the three-dimensional image.

[0022] According to a second aspect, presenting the three-dimensional image comprises superimposing the three-dimensional image on the three-dimensional object when viewed by a user of the display module.

[0023] According to the second aspect, the step of presenting a three-dimensional image further comprises: a step of identifying one or more anchor points marked on the three-dimensional object.

[0024] According to a second aspect, the method further comprises the step of simultaneously displaying one or more two-dimensional source images selected from the at least one original image set and the three-dimensional image.

[0025] According to a second aspect, the display module comprises a head mounted display module.

[0026] According to a third aspect of the present disclosure, an image processing pipeline is provided. The pipeline comprises: a 2D image preprocessing module, which is used to preprocess at least one original image set, each of which includes a plurality of two-dimensional source images, wherein each of the two-dimensional source images represents a cross-sectional view of a three-dimensional object at different positions along an axis in a three-dimensional space; and a 3D model building engine, which is used to construct a three-dimensional image representing the three-dimensional object by performing processing on 3D grid points extracted from the at least one original image set in the preprocessing; wherein the three-dimensional image can be read by a first image viewer, which is used to render the three-dimensional image and facilitate the operation of the three-dimensional image.

[0027] According to a third aspect, the plurality of two-dimensional source images comprises medical diagnostic scan images.

[0028] According to a third aspect, each of the at least one original image set comprises a plurality of medical diagnostic scan images obtained by a selected one of computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET) and X-ray imaging.

[0029] According to a third aspect, the medical diagnostic scan image is readable by a second image viewer.

[0030] According to a third aspect, the 3D model building engine is configured to embed the plurality of two-dimensional source images into the three-dimensional image at corresponding positions along the axis.

[0031] According to a third aspect, the first image viewer is for presenting a cross-sectional view of the three-dimensional object embedded with a two-dimensional view of the cross section reproduced based on the medical diagnostic scan image captured at the corresponding position.

[0032] According to a third aspect, the two-dimensional view of the cross section is substantially equal to a corresponding medical diagnostic scan image read by the second image viewer.

[0033] According to a third aspect, the image processing pipeline further comprises: an image mapping module for mapping the plurality of two-dimensional source images and the positions of the corresponding sections in the three-dimensional object along the axis.

[0034] According to the third aspect, the pre-processing module is further used for the steps of identifying and segmenting multiple components of different attributes or characteristics in the three-dimensional object.

[0035] According to a third aspect, the plurality of components comprises bone, soft tissue, fluid or implant.

[0036] According to a third aspect, the three-dimensional object comprises at least a portion of an organ, one or more organs, or a combination thereof of a living organism.

[0037] According to a third aspect, the pre-processing module is configured to receive manual input to facilitate concealing one or more of the plurality of components.

[0038] According to the third aspect, the pre-processing module is further configured to perform an image optimization process to optimize the at least one original image set before the steps of identifying and segmenting the plurality of components.

[0039] According to a third aspect, the pre-processing module is further configured to perform an image fusion process to fuse the multiple two-dimensional source images obtained from different original image sets.

[0040] According to the third aspect, the pre-processing module is further used to perform an interpolation process to generate one or more supplemented images, which represent a cross-sectional view of the three-dimensional object obtained at a position between two adjacent two-dimensional source images in the corresponding original image set captured along the axis.

[0041] According to a third aspect, the pre-processing module is further used to identify and segment different sets of original images obtained by different medical diagnostic scanning methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Hereinafter, embodiments of the present disclosure will be described by way of examples with reference to the accompanying drawings.

[0043] Figure 1 Schematic diagram of a computer server according to an embodiment of the present disclosure, where the computer server is used to implement an image processing pipeline.

[0044] Figure 2 The block diagram shows an image processing pipeline for processing an image according to an embodiment of the present disclosure.

[0045] Figure 3 To show Figure 2 Flowchart of the processing steps in the image processing pipeline.

[0046] Figures 4A to 4D Shown by Figure 3 An image processing pipeline performs example operations in a selected number of processing steps.

[0047] Figure 5 Shown in Figure 2 Multiple medical diagnostic scan images of a cross section of a part of a human body are shown before and after denoising and smoothing processing performed by a 2D image preprocessing module in an image processing pipeline.

[0048] Figure 6 A plurality of medical diagnostic scan images are shown, which show that Figure 2 The 2D image preprocessing module in the image processing pipeline generates a cross-section of a part of a human body before and after interpolation slices.

[0049] Figure 7 shows a medical diagnostic scan image showing Figure 2 The 2D image pre-processing module in the image processing pipeline performs a scaling / resizing process on a cross-section of a part of a human body before and after.

[0050] Fig. 8A and 8B Shown in Figure 2 Another set of multiple medical diagnostic scan images showing a cross-section of a part of a human body before and after the denoising and smoothing processing performed by the 2D image pre-processing module in the image processing pipeline.

[0051] Figures 9A to 9C Shown in Figure 2 The image processing pipeline is composed of Fig. 9B Medical diagnostic scan images of a cross section of a part of a human body are shown before and after the covert processing performed by the covert 2D image preprocessing module.

[0052] Figures 10A to 10B shows a medical diagnostic scan image showing Figure 2A cross section of a human body part before and after the 2D image preprocessing module in the image processing pipeline performs bone edge enhancement processing.

[0053] Fig.11 A 3D image of a 3D model of a bone structure constructed according to an embodiment of the present disclosure is shown, the 3D model including an original model, an enhanced intermediate model, and a final model with a movie effect.

[0054] Fig.12 Shown by Figure 2 A medical diagnostic scan image showing a cross-section of a part of a human body before and after CT / MRI image mapping processing performed by a 2D image preprocessing module in an image processing pipeline.

[0055] Fig.13 FIG. 5 is a diagram showing a first-person view of a user wearing a head-mounted display according to an embodiment of the present disclosure.

[0056] Fig.14 FIG. 1 is a diagram showing a first-person view of a user wearing a head mounted display according to an embodiment of the present disclosure during a surgical procedure.

[0057] Fig.15 FIG. 1 is a diagram showing a first-person view of a user wearing a head mounted display according to an embodiment of the present disclosure during a surgical procedure. DETAILED DESCRIPTION

[0058] refer to Figure 1 , shows an embodiment of a computer system, which can be implemented as a system for processing 3D images according to an aspect of the present disclosure. The system embodiment for processing 3D images (or 3D models or 3D objects) provides an image processing pipeline including the following. The pipeline includes: a 2D image preprocessing module, which is used to preprocess at least one original image set, each original image includes multiple two-dimensional source images, wherein each two-dimensional source image represents a cross-sectional view of a three-dimensional object at different positions along an axis in a three-dimensional space; and a 3D model construction engine, which is used to construct a three-dimensional image representing a three-dimensional object by aggregating 3D grid points extracted from at least one original image set in preprocessing; wherein the three-dimensional image can be read by a first image viewer, which is used to render the three-dimensional image and facilitate the operation of the three-dimensional image.

[0059] In this exemplary embodiment, the system for processing 3D images may include an interface and a processor, which is implemented by a computer with an appropriate user interface. The computer may be implemented by any computing architecture, including: a portable computer, a tablet computer, a stand-alone personal computer (PC), a smart device, an Internet of Things (IoT) device, an edge computing device, a client / server architecture, a "dumb" terminal / host architecture, a cloud computing-based architecture, or any other appropriate architecture. The computing device may be appropriately programmed to implement the present disclosure.

[0060] The system can be used to process medical images to generate 3D images (for the purposes of this document, the term 3D image includes 3D models or 3D objects, such as 3D graphical objects), which can be provided to users, such as medical experts or practitioners, for them to view or otherwise manipulate. The system is used to process one or more medical imaging sources, such as medical imaging sources captured by existing medical imaging tools (ultrasound, X-ray, MRI, CT scan, etc.), and process these images in an automatic or semi-automatic manner to produce 3D models that can be presented, viewed, examined, or manipulated by users. These systems may be advantageous because users (e.g., medical practitioners) may be able to use these systems to create unique 3D images or models with less effort, resources, or time, and also be able to use the 3D images or models to manipulate or examine before or during surgery, and thus improve the quality of the medical care provided.

[0061] like Figure 1As shown, a schematic diagram of a computer system or computer server 100 is shown. The computer system or computer server 100 is used to implement an example embodiment of a system for processing images. In this embodiment, the system includes a server 100. The server 100 includes appropriate components required to receive, store and execute appropriate computer instructions. These components may include: a processing unit 102, which includes a central processing unit (CPU), a math co-processing unit (Math Co-processing Unit, Math Processor), a graphics processing unit (Graphics Processing Unit, GPU) or a tensor processing unit (Tensor Processing Unit, TPU) for tensor or multi-dimensional array calculation or operation; a read-only memory (Read-Only Memory, ROM) 104; a random access memory (Random-Access Memory, RAM) 106; an input / output (Input / Output, I / O) device (e.g., a disk drive 108); an input device 110 (e.g., an Ethernet port, a USB port, etc.); a display 112, for example, a liquid crystal display, a light-emitting display or any other suitable display; and a communication link 114. The server 100 may include instructions that may be stored in ROM 104, RAM 106, or disk drive 108 and executed by the processing unit 102. A plurality of communication links 114 may be provided that may be variously connected to one or more computing devices, such as servers, personal computers, terminals, wireless or handheld computing devices, IoT devices, smart devices, and edge computing devices. At least one of the plurality of communication links may be connected to an external computing network via a telephone line or other type of communication link.

[0062] The server 100 may include storage devices such as a disk drive 108, which may include a solid-state drive, a hard disk drive, an optical drive, a tape drive, or a remote or cloud-based storage device. The server 100 may use a single disk drive or multiple disk drives, or a remote storage service 120. The server 100 may also have a suitable operating system 116 resident on the disk drive or in ROM of the server 100.

[0063] The computer or computing device may also provide the necessary computing power to operate or interface with a machine learning network such as a neural network to provide various functions and outputs. The neural network may be implemented locally, or may be accessed or partially accessed through a server or cloud-based service. The machine learning network may also be untrained, partially trained, or fully trained, and / or may also be retrained, modified, or updated over time.

[0064] refer to Figure 2 , shows an embodiment of a system 200 for processing images, such as an image processing pipeline. In this embodiment, the server 100 is a part of the system 200, and the system 200 is used to receive RAW images 202, such as multiple 2D CT or MRI scans or target objects or parts of targets, process these input images 202, such as by denoising and filtering these images 202, and finally generate a 3D model 204 combined with the 2D image 202 (or processed image 202A) of the target object (such as an organ of a living organism).

[0065] Preferably, the system 200 can automatically optimize the input image 202, which can be viewed by using a second image viewer (e.g., a DICOM viewer designed to open 2D medical diagnostic scan images of different formats), and construct a 3D image or model 204, which can be viewed and manipulated by a surgeon or medical staff using a first image viewer (e.g., a viewer application installed in a display device that can be used in an operating room). This enables the possibility of observing the original 2D medical diagnostic scan image as if the 2D scan image or the original image 202 was viewed by using the second image viewer or the DICOM viewer.

[0066] For example, after obtaining these 2D scans from an MRI scanning system, a set of magnetic resonance imaging (MRI) scans showing cross-sectional views of a patient's liver at different positions along a predetermined axis can be provided as input RAW images to the image processing pipeline. The image processing pipeline can identify a plurality of 3D grid points 212 representing feature points of a 3D model of the liver by extracting points from the RAW images; by aggregating all the 3D grid points 212, the image processing pipeline can construct a 3D model of the liver for further observation by a surgeon (for example) before or during a surgical operation on the patient.

[0067] Alternatively, other types of medical diagnostic scan images obtained by computed tomography (CT), positron emission tomography (PET), or X-ray imaging may be used as input, depending on the different applications or characteristics of the target organ or object. For example, soft tissues may be better captured using MRI, while tissues with higher density (e.g., bone) may be captured using CT. These medical diagnostic scan images may be viewed or opened by a 2D image viewer, such as a DICOM viewer designed to open these medical diagnostic scan images.

[0068] Without wishing to be bound by theory, a medical diagnostic scan image can be obtained by acquiring a plurality of cross-sectional images as slices at predetermined intervals along a predetermined axis, and these images are acquired from the body of a living organism (e.g., a human being). Therefore, by connecting or superimposing the plurality of cross-sectional images, a three-dimensional image of the body part can be generated. It should be understood that the quality of the generated 3D image can be enhanced by increasing the resolution of the source 2D image and the number of slices in the set medical diagnostic scan image.

[0069] Preferably, the generated 3D model 204 or image can be read by a dedicated image viewer, which is designed to open, view and operate the 3D image. In an exemplary embodiment, a display module, such as a head mounted display (HMD) module, can be worn by a surgeon when performing a surgical operation, and the HMD can be equipped with such an image viewer (first image viewer) for viewing and operating, such as zooming, rotating, translating the three-dimensional image and / or moving the cutting plane to cut the three-dimensional image along the axis of the three-dimensional model 204. It should be understood by those skilled in the art that the 3D image can also be displayed or rendered on other display modules or devices.

[0070] Preferably, the 2D view of the cross section is substantially the same as the corresponding medical diagnostic scan image read by the second image viewer. When viewing the 3D image using a dedicated image viewer, for example, when a surgeon uses an HMD to view a cross-sectional view of a 3D model of an object, a 2D view of the original 2D medical diagnostic scan image can be simultaneously observed, and the corresponding 2D medical diagnostic scan image can be presented to the surgeon and / or other observers without the need to use a separate DICOM viewer to read the 2DRAW image.

[0071] Continue to refer Figure 3 and Figures 4A to 4D , the processing method 300 involved in the image processing pipeline includes: a plurality of steps of processing the 2D RAW images 202 before and after generating the 3D model 204 or image of the object of interest. Preferably, the 2D image pre-processing module 206 may use one or more pre-processing to optimize the plurality of images 202 received by the pipeline in order to improve the accuracy of the 3D mesh points generated for 3D model construction.

[0072] For example, the processing method 300 may start by providing a plurality of medical diagnostic scan images at step 302, which may be referred to as a DICOM system, which includes one or more sets of CT / MRI / PET / X-ray scans, which may be viewed and observed by a medical practitioner using a DICOM viewer (a second image viewer). The medical diagnostic scan images 202 may be obtained by capturing a new scan using an appropriate medical imaging device, or retrieved from a database. The process may be manual or semi-automatic, and an operator may be required to specify which set of images should be provided for further processing. DICOM is a standard format for reading and processing different image modalities in the same software platform, and the DICOM system may provide a robust infrastructure for a DICOM Modality Worklist that may be effectively expanded as demand grows and provide a highly compatible solution to accommodate different hospitals.

[0073] Once the RAW image 202 is provided, the 2D image pre-processing module 206 may perform an image optimization process to optimize the raw image 202 , which is useful for improving subsequent steps of identifying and segmenting multiple components in the 2D image 202 .

[0074] Preferably, in step 304, the pre-processing module 206 may perform an optional image fusion process to fuse multiple (when available) two-dimensional source images obtained from different raw image sets. For example, two different sets of RAW images 202 obtained by CT and MRI may be provided to the system 200 for 3D image construction.

[0075] For example, patients can have medical scans across different modalities, at different times, in different postures, and with different scanning protocols by different technicians and organizational standards. As a result, even within the same clinical case, images can vary widely. However, clinicians may need to review different imaging scans for accurate diagnosis and case planning. Preferably, accurate fusion of different images will greatly facilitate understanding of pathology.

[0076] According to the inventors' concept, recovering high-quality anatomical information may be relatively simple, especially from high-resolution raw data, but what is challenging is processing the highly varying noise from different types of raw images and subsequently reconstructing a high-quality 3D anatomical model.

[0077] Preferably, the image preprocessing module 206 can also perform AI-assisted image fusion processing and subsequent 3D reconstruction processing for enhancing DICOM raw data. The image fusion process may include the following steps, including the following preferred steps: resampling pixel spacing (step 304-1), super-resolution processing on DICOM raw data (step 304-2), smoothing and denoising (step 304-3), enhancing edge (e.g., between bone and soft tissue) contrast (step 304-4), AI-based interpolation resampling slice thickness (step 304-5), and finally exporting enhanced DICOM data (step 304-6).

[0078] Preferably, AI-based medical image denoising can adopt different strategies for CT and MR images. Bone and soft tissue contrast is enhanced by AI-based edge enhancement technology. Figure 5 , the noise in the RAW image on the left has been removed, so the features presented in the denoised image are clearer. Fig. 8A and 8B Another set of original RAW images and pre-processed images obtained after denoising and smoothing are shown in Figure 2, respectively. The features of the scan become more visible to the naked eye and therefore easier to recognize by the image processor. This is because the edges of the features are clearer (or have higher contrast), so the 3D grid points extracted based on these edges can be more accurate.

[0079] In addition, the pre-processing module 206 also performs an interpolation process to generate one or more supplemented images representing a cross-sectional view of the three-dimensional object acquired at a location between two adjacent two-dimensional source images in the corresponding set of original images captured along the axis.

[0080] refer to Figure 6 , AI-based interpolation technology can help increase the number of original data slices, for example, for creating an interpolated slice 606 representing a cross-sectional image at a location between two adjacent scans 602 and 604, i.e., slice 001 and slice 002 at a distance of 3 mm along the axis. This process generates higher quality model data by adding another layer of 3D grid points at locations between the original slices initially spaced 6 mm apart, approximately adding richer point cloud data to provide higher precision data for the subsequent process of 3D surface reconstruction.

[0081] Alternatively or additionally, the pre-processing module 206 may also resample or rescale the input RAW image 202 so that the 3D model building engine 208 can process all RAW images with the same size and resolution. Figure 7, the GAN-based image super-resolution technology can be used to enlarge the original RAW image with a resolution of 512×512 to an output image with a resolution of 1024×1024 for further processing. In addition, the pre-processing module can resample the DICOM raw data into a standard pixel spacing system (for example, converting the pixel spacing 0.617578125mm×0.617578125mm to 0.5mm×0.5mm). Advantageously, the unified data standard will facilitate CT and MRI registration based on image sensitive areas, thereby achieving higher accuracy in the constructed 3D model 204.

[0082] After the image fusion process, in step 306, the pipeline may perform a concealment process to exclude any useless intensities from the raw data. Figures 9A to 9C , a mask 900 identifying "useful" components / regions / portions 902 and "unuseful" components / regions / portions 904 may be provided to an image processing pipeline, for example by receiving manual input 210 (drawing a custom mask) to facilitate masking one or more of the multiple components, for example by excluding a medical implant 906 captured by a scanned image. Since adjacent scans or slides may include the same or very similar useful and unuseful components in each scan, additional masks for excluding or eliminating unuseful components of other 2D images in the same set of original images may be automatically generated with reference to the first mask generated based on the manual input 210.

[0083] Preferably, the concealment step 306 may include the following steps: applying default concealment (step 306-1), drawing custom concealment (step 306-2), applying concealment to trim useless intensity pixels in DICOM (step 306-3), and finally exporting the trimmed DICOM data (step 306-4).

[0084] To further improve the three-dimensional grid points 212 generated based on the shape of the target object or component, in step 308, the image 202 can be further optimized using other image processing techniques, such as edge enhancement, because outlining the edges of different features or components in multiple images helps the process identify and segment different components in a single image or a set of images representing cross-sections at different locations of the patient. For example, referring to Fig. 10A and 10B , the bone edges have been enhanced to "highlight" the bone component in the image, in which case the system will find it easier to segment bone from soft tissue in the same image and in other images in the same set of images.

[0085] Preferably, the segmentation step 308 may include the following steps: deriving a new segmentation configuration from a specific template (step 308-1), adjusting the threshold to accurately represent the segmentation strength (step 308-2), adding an AI-based image fusion filter (step 308-3), and finally saving the segmentation configuration (step 308-4).

[0086] In alternative examples, other user-defined and / or built-in configurations can be used to declare segmentation parameters in order to separate or segment different components, such as visceral organs, muscles, fat, blood vessels, nerves, etc., from other components in the original image. The segmentation parameters or configurations can be derived from templates stored in a database, and the operator can adjust the threshold of each component to accurately represent the segmentation strength and selectively display or hide unnecessary components as needed. In addition, an AI-based image fusion filter can also be applied to the segmentation step 308.

[0087] Those skilled in the art will appreciate that not every step or process requires manual input, for example, if the AI-based processing engine is well trained and has sufficient training examples. Figure 2 and 3 In this example, processes such as providing raw data, concealing useless components in the image, and segmentation based on one or more user definitions can be identified as "manual" or "semi-automatic" processing because it may require at least some user input to further improve the efficiency or accuracy of the constructed 3D image. However, these steps or processes may also be fully automated.

[0088] In order to construct a 3D model based on multiple 2D cross-sectional images at different locations, in step 310, the volumes of different segmented regions need to be allocated or updated according to the relationship between the origin and the image coordinate system. In step 310-1, the segmented volume is normalized to an unsigned Int-8 format; in step 310-2, AI-based image fusion can be applied to the segmented volume; then in step 310-3, the volume origin is updated to the image coordinate system; finally, in step 310-4, the segmented DICOM data (i.e., volume / mesh data) can be exported and further used for 3D model generation.

[0089] Using the exported segmented volume data, an original 3D mesh from the point cloud can be generated at step 312, and then the 3D model / image 204 can be constructed by aggregating the generated 3D mesh points 212 extracted from the original image 202 pre-processed by the 2D image processing module 206 as described above, as shown in FIG. Fig.11 As shown, Fig.11A patient's bone structure 1102 is shown near the person's hip. Optionally, to further improve the visual representation of the 3D image 1102 constructed based on the 3D point cloud in step 312-1, one or more optimization methods may be applied, such as 3D mesh repair in step 312-2, mesh optimization in step 312-3, mesh retina inspection in step 312-4, or UV map generation in step 312-5, to improve the surfaces and edges of the 3D image generated by connecting the 3D mesh points together so that these edges or surfaces may be less wrinkled when viewed and / or manipulated by a clinician, as shown in the "enhanced intermediate model", which may be more preferred by a surgeon or medical practitioner because the enhanced model includes less unnecessary texture. Finally, the 3D model 1104 (e.g., image data) may be exported and saved in step 312-6. Fig.11 shown).

[0090] Additionally, in step 314, a film-like rendering filter may be applied to further enhance the representation of the 3D image 204 to further improve the visual appearance of the created 3D model. For example, a studio lighting scene (also referred to as Blender) may be set up in step 314-1 and then the constructed 3D model may be integrated into the scene in step 314-2. By setting the material in step 314-3 and applying the material texture to the surface of the 3D model in step 314-4, when viewed by a practicing physician or surgeon, the 3D model may appear to be a human figure. Fig.11 The final 3D model 1106 shown may be more visually appealing. Texture data may be output at step 314-5. Lighting conditions may also be set in the viewer so that the direction and / or number of light sources may be changed to further change the appearance of the 3D image representing the 3D object of interest, such as Fig.11 As shown in the “final product” 1106 in FIG.

[0091] As previously mentioned, multiple sets of medical diagnostic scan images can be provided as two-dimensional source images for 3D image construction, and the positions / volumes of the individual segments in different scans may be different, so some shifts or offsets may be introduced in the image representation. Preferably, the method includes an image mapping or registration process in step 316, which allows the MRI volume to be manually mapped or registered to the approximate position on the CT volume in step 316-1, and then the image mapping module can map the positions of the corresponding sections along the axis in the multiple two-dimensional source images and the three-dimensional object by calculating the matrix of the registration transformation in step 316-2 and applying the transformation matrix to the MRI volume data in step 316-3. Finally, the transformed MRIDICOM data can be output in step 316-4.

[0092] refer to Fig.12When a cross-sectional image of the 3D model at a selected position is presented to the viewer, the MRI segments and the CT segments are aligned so that there is no misalignment between the two source images in the presented cross-section. Alternatively, the mapping process can be performed by manually mapping the CT volume relative to the MRI volume in order to align the CT and MRI (and / or other source image sets) associated with the same 3D model.

[0093] Preferably, the image viewer installed in the HMD can present a cross-sectional view of the embedded three-dimensional object with a two-dimensional view of the cross-section reproduced based on the medical diagnostic scan image captured at the corresponding position, and optionally, also simultaneously display a source image selected from at least one original image set and the three-dimensional image, so that the surgeon or clinician can simultaneously view the 2D image that is indistinguishable from viewing the medical diagnostic image using a DICOM viewer.

[0094] In an example embodiment, in step 318, DICOM sprites can be constructed by exporting the DICOM slice series to PNG format in step 318-1, so that the DICOM slices, such as CT / MRI scans, do not have to be opened using a DICOM viewer. In step 318-2, the clinician or surgeon can also select only the DICOM slice series of interest or all slices at different positions along the axis of the 3D model. Then, in step 318-3, the selected DICOM can be constructed into a "power of 2" square megapixel image. Fig.14 and Fig.15 An example rendering including a cross section and a DICOM slice of interest is shown in , which will be described later. Preferably, in step 318-4, a sprite sheet associating a PNG file with a 3D model can be constructed and stored as a configuration file, which can be referenced when a 3D image viewer needs to locate and open a matching PNG file so that the clinician can view the corresponding 3D image cross-sectional view. Finally, in step 318-5, the sprite sheet image and configuration are exported.

[0095] Optionally, the image processing flow may also include an additional step 320 for recording patient-related information, such that a file data structure or package may contain all necessary information related to the surgical procedure, such as patient data and operating room information, as well as the constructed 3D image, the source 2D image reformatted in PNG format, and the mapping table, which may be packaged into an "asset bundle" and saved in the local storage of the HMD device. For example, the asset bundle or surgical data may be patched into the Synglular application on the HoloLens device so that the surgeon may view any of the above information when necessary during the surgical procedure.

[0096] See also Fig.13 , showing exemplary operation of an HMD arranged to display or render a three-dimensional image 204 constructed by the previously described image processing pipeline 200. In this example, the display module is also provided with a user interface 1302, which facilitates manipulation of the three-dimensional image 204.

[0097] Preferably, the 3D image 204 is presented at the center of the screen of the head-mounted display device, and the user can operate the three-dimensional image by performing different gestures or interacting with the user interface 1302. The operation of the three-dimensional image includes scaling, rotating, translating the three-dimensional image and / or moving at least one of the cross-sectional planes used to segment the three-dimensional image along the axis. For example, the user can "grab" the control label 1304 located on the axis 1306, and move the control label to different positions along the axis 1306 to move the cross section, and then release the control label 1304 to stop the cross section at the selected position. In this configuration, the cross section embedded in the 3D image 204 at the selected position is displayed to the user. Alternatively, the 3D image 204 can be operated by performing a grab gesture directly on the 3D model 204 and then rotating the model 204 by twisting the wrist.

[0098] Optionally, the user can "mark" the "Toggle DICOM" button 1308, change the view to "2D view", and use a DICOM viewer or other suitable 2D image viewer to read the 2D cross-sectional image 202. Alternatively, the HMD can display one or more two-dimensional source images 202 selected from at least one original image set by directly manipulating the 3D image 204 and manipulating the 3D image 204 simultaneously.

[0099] In this example, a 3D image 204 is constructed by providing a set of CT or MRI scans (raw images 202) of the skeletal structure of interest to an image processing system, which then automatically processes all of the source 2D images 202 and creates a 3D model 204 that is integrated (or embedded) with the associated cross-sectional scans 202 / 202A.

[0100] refer to Fig.14 and 15 , shows an alternative exemplary procedure, wherein the image represents a first-person view of a surgeon performing a medical procedure on an ankle 1402 of a patient 1404. In this example, the HMD superimposes a three-dimensional image 204 on a three-dimensional object 1402 when viewed by a user of the display module, however, the 3D image / model 204 is not aligned with the actual position of the 3D object, i.e., the ankle 1402 of the patient 1404, as shown in FIG. Fig.14Alternatively, the HMD may identify one or more anchor points (not shown) marked on the three-dimensional object so that the 3D image / model 204 can be precisely aligned with the object in a particular display mode. For example, when viewed by a surgeon wearing the HMD, the 3D image 204 can be automatically positioned at a fixed distance from the ankle 1402 of the patient 1404, so that the 3D image 204 is close to the surgeon's focus, but it does not obstruct the viewing of the ankle 1402 during surgery.

[0101] In this example, the 3D model 204 of the ankle 1402 is displayed on the left side of the AR / MR display, while the original 2D CT / MRI image 202 / 202A is displayed on the right side of the AR / MR display. Preferably, when the surgeon moves the cross-sectional plane to change the cross-sectional view to the 3D model 204, the corresponding source 2D CT / MRI (original) image 202 / 202A is displayed on the right side for simultaneous viewing by the surgeon without requiring the surgeon to manually select the correct 2D image to be opened by a separate DICOM viewer.

[0102] These embodiments can be advantageous because the 3D image processing pipeline is equipped with a 2D preprocessing module that is suitable for preprocessing multiple sets of raw images in a batch and automatic manner using a high-performance image processing pipeline. This can effectively eliminate or at least reduce manual input for optimizing typical medical diagnostic scan images (such as CT, MRI, PET, and X-ray scans), which can then be used as source images for 3D model construction based on segmentation and 3D grid points extracted from these source images. If necessary, the processing pipeline is also capable of manual input, which can further improve the 3D images generated according to the surgeon's preferences.

[0103] In addition, the constructed 3D image is embedded with a source 2D scan that allows the clinician or surgeon to view the original 2D scan when needed, and is also embedded with a 3D model that shows the corresponding cross-sectional views. Because experienced surgeons or medical staff are accustomed to viewing 2D cross-sectional scan images using traditional DICOM viewers, the simultaneous display of the matching 2D source image and 3D cross-sectional views may be favored by some practitioners. By providing 2D / 3D image mapping data, there is no need for additional operations to select / open the desired 2D scan image, because the matching 2D scan image will be automatically displayed simply by operating the 3D model / image in the cross-sectional viewing mode. Preferably, in this implementation, the 2D scan image is a sprite in the sprite map created in step (318), which together show different sprites or images related to body parts, tissues, bones, etc. in multiple cross-sections. Therefore, when the surgeon or user performs a "zoom in and out" operation, the sprite image can be presented frame by frame, simulating the surgeon's "zoom in and out" action to display the cross-sections of relevant body parts, tissues, bones, etc., thereby enabling the surgeon or other users to easily and intuitively access the cross-sectional images in the 3D model and facilitate them to operate the 3D model.

[0104] Although not required, the embodiments described with reference to the accompanying drawings may be implemented as an application programming interface (API) or a series of libraries used by developers, or may be included in another software application, such as a terminal or personal computer operating system or a portable computing device operating system. Generally, since program modules include routines, programs, objects, components, and data files that help perform specific functions, those skilled in the art will understand that the functionality of a software application may be distributed among multiple routines, objects, or components to achieve the same functionality required herein.

[0105] It should also be understood that where the methods and systems of the present disclosure are implemented in whole or in part by a computing system, any suitable computing system architecture may be used. This would include stand-alone computers, network computers, and dedicated hardware devices. Where the terms "computing system" and "computing device" are used, these terms are intended to encompass any suitable configuration of computer hardware capable of implementing the described functionality.

[0106] It will be appreciated by those skilled in the art that various changes and / or modifications may be made to the present disclosure as shown in the specific embodiments without departing from the spirit or scope of the present disclosure as broadly described. Therefore, the present embodiments are to be considered in all aspects as illustrative and not restrictive.

[0107] Unless otherwise indicated, any reference to prior art contained herein should not be taken as an admission that the information is common general knowledge.

Claims

1. A three-dimensional image processing method, characterized in that: include: Preprocessing at least one original image set, each of the original image sets comprising a plurality of two-dimensional source images; wherein each of the two-dimensional source images represents a cross-sectional view of a three-dimensional object at different positions along an axis in a three-dimensional space; and constructing a three-dimensional image representing the three-dimensional object by aggregating the 3D grid points extracted from the at least one original image set in the preprocessing; The three-dimensional image can be read by a first image viewer, and the first image viewer is used to render the three-dimensional image and facilitate operation of the three-dimensional image.

2. The method according to claim 1, characterized in that in, The plurality of two-dimensional source images include medical diagnostic scan images.

3. The method according to claim 2, characterized in that in, Each of the at least one original image set includes a plurality of medical diagnostic scan images obtained by a selected one of computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), and X-ray imaging.

4. The method according to claim 2, characterized in that: in, The medical diagnostic scan image is readable by a second image viewer.

5. The method according to claim 4, characterized in that in, The step of constructing the three-dimensional image comprises the step of embedding the plurality of two-dimensional source images into the three-dimensional image at corresponding positions along the axis.

6. The method according to claim 5, characterized in that in, The first image viewer is for presenting a cross-sectional view of the three-dimensional object embedded with a two-dimensional view of the cross-section reproduced based on the medical diagnostic scan image captured at the corresponding position.

7. The method according to claim 6, characterized in that in, The two-dimensional view of the cross section is substantially equal to a corresponding medical diagnostic scan image read by the second image viewer.

8. The method according to claim 1, characterized in that: Also includes: The step of mapping the positions of the plurality of two-dimensional source images and corresponding cross sections in the three-dimensional object along the axis.

9. The method according to claim 1, characterized in that: Also includes: The step of identifying and segmenting multiple components of different attributes or properties in the three-dimensional object.

10. The method according to claim 9, characterized in that in, The plurality of components may include bone, soft tissue, fluid or an implant.

11. The method according to claim 10, characterized in that in, The three-dimensional object includes at least a portion of an organ, one or more organs, or a combination thereof of a living organism.

12. The method according to claim 9, characterized in that in, Identifying the plurality of components includes facilitating manual concealment of one or more of the plurality of components.

13. The method according to claim 9, characterized in that in, The pre-processing of the at least one original image set comprises: before the step of identifying and segmenting the plurality of components, the step of performing image optimization processing to optimize the at least one original image set.

14. The method according to claim 13, characterized in that in, The image optimization process includes an image fusion process for fusing the multiple two-dimensional source images obtained from different original image sets.

15. The method according to claim 14, characterized in that in, The image optimization process includes an interpolation process for generating one or more supplemented images representing cross-sectional views of the three-dimensional object acquired at locations between two adjacent two-dimensional source images in corresponding sets of original images captured along an axis.

16. The method according to claim 3, characterized in that Also includes: Identify and segment different sets of raw images obtained through different medical diagnostic scanning methods.

17. A method for presenting a three-dimensional image constructed by the method of claim 1, characterized in that: include: The three-dimensional image is presented on a display module; wherein the display module is provided with a user interface for facilitating operation of the three-dimensional image.

18. The method according to claim 17, characterized in that in, The manipulation of the three-dimensional image includes at least one of scaling, rotating, translating the three-dimensional image and / or moving a cross-sectional plane for segmenting the three-dimensional image along the axis.

19. The method according to claim 17, characterized in that in, The presenting the three-dimensional image includes superimposing the three-dimensional image on the three-dimensional object when viewed by a user of the display module.

20. The method according to claim 19, characterized in that in, The step of presenting a three-dimensional image further includes the step of identifying one or more anchor points marked on the three-dimensional object.

21. The method according to claim 17, characterized in that Also includes: The step of simultaneously displaying one or more two-dimensional source images selected from said at least one original image set and said three-dimensional image.

22. The method according to claim 17, characterized in that in, The display module includes a head-mounted display module.

23. An image processing pipeline comprising: a 2D image preprocessing module, which is used to preprocess at least one original image set, each of which includes a plurality of two-dimensional source images; wherein each of the two-dimensional source images represents a cross-sectional view of a three-dimensional object at different positions along an axis in a three-dimensional space; and a 3D model building engine, configured to build a three-dimensional image representing the three-dimensional object by aggregating 3D grid points extracted from the at least one original image set in preprocessing; The three-dimensional image can be read by a first image viewer, and the first image viewer is used to render the three-dimensional image and facilitate operation of the three-dimensional image.

24. The image processing pipeline according to claim 23, characterized in that: in, The plurality of two-dimensional source images include medical diagnostic scan images.

25. The image processing pipeline according to claim 24, characterized in that: in, Each of the at least one original image set includes a plurality of medical diagnostic scan images obtained by a selected one of computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), and X-ray imaging.

26. The image processing pipeline according to claim 24, characterized in that: in, The medical diagnostic scan image is readable by a second image viewer.

27. The image processing pipeline according to claim 26, characterized in that: in, The 3D model building engine is used to embed the plurality of two-dimensional source images into the three-dimensional image at corresponding positions along the axis.

28. The image processing pipeline according to claim 27, characterized in that: in, The first image viewer is for presenting a cross-sectional view of the three-dimensional object embedded with a two-dimensional view of the cross-section reproduced based on the medical diagnostic scan image captured at the corresponding position.

29. The image processing pipeline according to claim 28, characterized in that: in, The two-dimensional view of the cross section is substantially equal to the corresponding medical diagnostic scan image read by the second image viewer.

30. The image processing pipeline according to claim 23, characterized in that: Also includes: An image mapping module is configured to map the plurality of two-dimensional source images and the positions of the corresponding cross sections in the three-dimensional object along the axis.

Images