FORS-enabled image tagging for machine learning models

Abstract

An intervention control method for acquiring an image marker (78) of an object of interest within an imaging space during an intervention procedure based on optical shape sensing. The method involves navigating an interventional device (30) integrated with an optical shape sensor (41) within the imaging space according to the interventional procedure based on registration of the optical shape sensor (41) with the imaging space. The method further involves deriving the image marker (78) of the object of interest within the imaging space according to: based on the pair of interventional devices (30) within the imaging space according to the interventional procedure by the optical sensor (41); sensing of navigation generates encoded image data providing information of a pose of the object of interest within the imaging space.

Description

Technical Field

[0001] The present disclosure generally relates to the generation of training data for machine learning models (e.g., deep neural networks) for use in image segmentation, pose estimation, or image registration during an interventional procedure (e.g., minimally invasive surgery, video-assisted thoracic surgery, vascular procedure, endoluminal procedure, orthopedic procedure), e.g., by tracking the position and / or orientation of an interventional device or anatomical feature relative to a medical imaging system to perform the above operations. Background Art

[0002] Machine learning models (especially deep neural networks) are used in medical imaging in many applications. For example, in image-guided therapy, deep learning methods have been used to (1) identify interventional devices in medical images to assist in guiding the procedure, (2) identify specific anatomical features to be treated in medical images, or (3) learn the registration from one medical imaging modality to another to achieve image fusion. These machine learning models are trained on very large imaging datasets that, in most cases, must be labeled to provide the ground truth on which the training of the machine learning model is based.

[0003] In the context of digital image processing, image segmentation provides the detection and labeling of objects of interest (e.g., anatomical organs / models or interventional devices) depicted in medical images and is implemented to generate very large imaging datasets for machine learning models. Summary of the Invention

[0004] Although machine learning models for image segmentation have proven beneficial, there is still a need for improved techniques to generate very large labeled imaging datasets for training machine learning models. To this end, the present disclosure utilizes optical sensors (e.g., optical (shape) sensors provided by fiber optic real shape (FORS) fibers) to automate the image labeling process or create large datasets for machine learning models for various applications (e.g., identification and tracking of interventional devices, identification and tracking of anatomical features, or registration of multiple imaging modalities).

[0005] More specifically, an optical (e.g., FORS) sensor uses light arranged along a multi-core optical fiber or along a plurality of (single-core) optical fibers to perform device positioning and navigation during an intervention procedure. The principle involved utilizes distributed strain measurement in one or more optical fibers, using characteristic Rayleigh backscattering or a controlled (Bragg) grating pattern to provide information in the reflected light. Optical detection along the optical fiber begins at a specific point (referred to as the launch or z = 0) along the optical sensor, and the subsequent shape, position, strain, and / or orientation of the optical sensor are all relative to this point. For this reason, the optical sensor (usually in an optical fiber) can be integrated into an intervention device to provide the shape, position, and / or orientation of the whole or part of the intervention device relative to this point as well, which provides optical sensing of the intervention device within the image space during the intervention procedure. Optionally, different optical elements of the sensor are arranged to continuously sense the shape along the intervention device.

[0006] Exemplary embodiments of the present disclosure include, but are not limited to, (1) an intervention control system, (2) an image marker controller, and (3) an intervention control method for obtaining an image marker of an object of interest within an imaging space.

[0007] The image marker may represent training data of a machine learning model or ground truth data of an imaging process.

[0008] The object of interest may be an intervention device or an anatomical object.

[0009] The imaging space may be an imaging space delineated by a medical imaging system or a simulated imaging space of a medical imaging system.

[0010] Various intervention control system embodiments of the present disclosure cover systems for obtaining an image marker of an object of interest within an imaging space during an (e.g., live or simulated) intervention procedure.

[0011] The system may employ an intervention device integrated with an optical sensor (or an optical sensing system, optionally an optical shape sensing system or sensor), and the intervention device is operable to be navigated within the imaging space according to the intervention procedure based on the registration of the optical (shape) sensor (or sensing system) with the imaging space.

[0012] The system employs an imaging marker controller configured to derive the image marker of the object of interest within the imaging space according to the following operations: generating encoded image data that provides information on the pose of the object of interest within the imaging space based on sensing the navigation of the intervention device within the imaging space according to the intervention procedure based on the optical (shape) sensing by the optical sensor (or optical shape sensor or optical sensing system or optical shape sensing system).

[0013] Various imaging marker controller embodiments of the present disclosure encompass a controller that employs one or more processors to run or execute instructions stored or encoded on a non-transitory machine-readable storage medium to obtain an image marker of an object of interest within an imaging space that is registered to an optical (shape) sensor integrated with an intervention device that is navigated within the imaging space according to an (e.g., live or simulated) intervention procedure.

[0014] The instructions may be arranged to derive the image marker of the object of interest within the imaging space according to the following operations: generating encoded image data that provides information about the pose of the object of interest within the imaging space based on sensing by the optical (shape) sensing system or optical sensor of the navigation of the intervention device within the imaging space according to the intervention procedure.

[0015] Various intervention control method embodiments of the present disclosure encompass methods executable by an optional device navigation controller and an image marker controller for obtaining an image marker of an object of interest within an imaging space during an (e.g., live or simulated) intervention procedure.

[0016] The method involves: (the optional device navigation controller) navigating an intervention device integrated with the optical (shape) sensor within the imaging space according to the intervention procedure based on the registration of the optical (shape) sensor with the imaging space.

[0017] The method further involves: the image marker controller deriving the image marker of the object of interest within the imaging space according to the following operations: generating encoded image data that provides information about the pose of the object of interest within the imaging space based on sensing by the optical (shape) sensing system of the navigation of the intervention device within the imaging space according to an intervention procedure based on the optical (shape) sensing.

[0018] Based on the following detailed description of various embodiments of the present disclosure read in conjunction with the drawings, the foregoing and other embodiments of the present disclosure and various structures and advantages of the present disclosure will become more apparent. The detailed description and the drawings are merely illustrative of the present disclosure and not limiting, and the scope of the present disclosure is defined by the appended claims and their equivalents. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Illustrates an exemplary embodiment of a medical intervention system according to the present disclosure.

[0020] Figure 2 Illustrates an exemplary embodiment of a medical intervention method according to the present disclosure.

[0021] Figure 3A and Figure 3B illustrates exemplary embodiments of an optical sensing guidewire and catheter-based medical interventions in accordance with the present invention.

[0022] Figure 4A and Figure 4B illustrates exemplary embodiments of two-dimensional ultrasound-based medical interventions in accordance with the present invention.

[0023] Figure 5A and Figure 5B illustrates exemplary embodiments of three-dimensional ultrasound-based medical interventions in accordance with the present invention.

[0024] Figure 6A-6C illustrates exemplary embodiments of optical sensing guidewire-based medical interventions in accordance with the present invention.

[0025] Figure 7 illustrates a first exemplary embodiment of endoscope-based medical interventions in accordance with the present invention.

[0026] Figure 8 illustrates a second exemplary embodiment of endoscope-based medical interventions in accordance with the present invention.

[0027] Figure 9 illustrates exemplary embodiments of anatomical feature detection-based medical interventions in accordance with the present invention.

[0028] Figure 10 illustrates exemplary embodiments of image registration-based medical interventions in accordance with the present invention.

[0029] Figure 11 illustrates exemplary embodiments of dual imaging modality-based medical interventions in accordance with the present invention.

[0030] Figure 12 illustrates exemplary embodiments of an imaging marker controller in accordance with the present disclosure.

[0031] Figure 13 illustrates exemplary embodiments of a user interface in accordance with the present disclosure. Detailed Description

[0032] The present disclosure applies to many and various optical sensing-based intervention procedures arranged to provide location-related information, shape-related information, orientation-related information, and / or strain-related information. These intervention procedures include, but are not limited to, (1) minimally invasive procedures using optical (shape) sensors (e.g., endoscopic hepatectomy, necrotomy, prostatectomy, etc.), (2) video-assisted thoracic procedures using optical (shape) sensors (e.g., lobectomy, etc.), (3) minimally invasive vascular procedures using optical (shape) sensors (e.g., via catheter, sheath, deployment system, etc.), (4) minimally invasive medical diagnostic procedures using optical (shape) sensors (e.g., intraluminal procedures via endoscope or bronchoscope), (5) orthopedic procedures using optical (shape) sensors (e.g., via k-wire, screwdriver, etc.), and (6) data collection procedures (e.g., bench-top / phantom data, pre-clinical, etc.).

[0033] The present invention also applies to: performing optical (shape) sensing-based intervention procedures on a patient or model in vivo, and simulating optical (shape) sensing-based intervention procedures on a patient or model.

[0034] The present disclosure improves the generation of training data for machine learning models (e.g., deep neural networks) for image segmentation or pose estimation during an intervention procedure (e.g., by tracking the position and / or orientation of an intervention device or anatomical feature relative to a medical imaging system).

[0035] To facilitate understanding of the present disclosure, Figure 1 the following description teaches exemplary embodiments of a medical intervention system according to the present disclosure. According to Figure 1 the description, those of ordinary skill in the art of the present disclosure will recognize how to apply the present disclosure to make and use additional embodiments of the medical intervention system according to the present disclosure.

[0036] Referring to Figure 1 , exemplary embodiments of the medical intervention system of the present disclosure employ one or more medical imaging systems 20, which include, but are not limited to, an X-ray imaging system 21, an ultrasound imaging workstation 22, a computed tomography imaging system 23, and a magnetic resonance imaging system 24. As is known in the art of the present disclosure, each of the medical imaging systems 20 in the medical imaging system 20 is operable to delineate a planar imaging space or a volumetric imaging space for imaging an object within the imaging space.

[0037] Still referring to Figure 1, an exemplary medical intervention system of the present disclosure also employs one or more intervention devices 30, and the one or more intervention devices 30 include, but are not limited to, a guide wire 31, a catheter 32, an endoscope 33, and a TEE ultrasound probe 34. As is known in the field of the present disclosure, each of the intervention devices 30 provides surgical functions, treatment functions, and / or diagnostic functions to anatomical objects or models within a patient during (live or simulated) intervention procedures.

[0038] Still referring to Figure 1 , an exemplary medical intervention system of the present disclosure also employs an optical sensing system 40, and the optical sensing system 40 includes an optical sensor 41, an optical sensor controller 42, and an optical sensor interpreter 43.

[0039] The optical sensor 41 is arranged to provide position, orientation, strain, shape-related information or data, and the optical sensor interpreter 43 is configured to retrieve the information or data and provide them to the controller for further processing.

[0040] In one embodiment, the optical (shape) sensor 41 is a fiber Bragg grating-based sensor. As is known in the field of the present disclosure, a fiber Bragg grating (FBG) is a short section of an optical fiber that reflects light of a specific wavelength and transmits light of all other wavelengths. This is achieved by adding a periodic variation in the refractive index to the optical fiber core, which creates a wavelength-specific dielectric mirror. Thus, a fiber Bragg grating can be used as an inline filter to block certain wavelengths or as a wavelength-specific reflector.

[0041] The basic principle behind the operation of a fiber Bragg grating is Fresnel reflection at each interface in the interface where the refractive index changes. For some wavelengths, the reflected light from each period is in phase, such that there is constructive interference for reflection and thus destructive interference for transmission. The Bragg wavelength is sensitive to strain as well as temperature. This means that a Bragg grating can be used as a sensing element in an optical fiber sensor. In an FBG sensor, a measurement (e.g., strain) causes a shift in the Bragg wavelength.

[0042] One advantage of this technology is that various sensor elements can be distributed along the length of the optical fiber. Combining three or more cores with various sensors (gages) along the length of the optical fiber embedded in a structure allows for the precise determination of the three-dimensional form of such a structure, which typically has an accuracy better than 1 mm. Along the length of the optical fiber, at various positions, multiple FBG sensors (e.g., 3 or more optical fiber sensing cores) can be positioned. Based on the strain measurement of each FBG, the curvature of the structure at that position can be inferred. Based on multiple measurement positions, the total three-dimensional form is determined.

[0043] As one of many alternatives to fiber Bragg gratings, the inherent backscattering in conventional optical fibers can be utilized. One such method is to use Rayleigh scattering in standard single-mode communication optical fibers. Rayleigh scattering occurs due to random fluctuations in the refractive index within the fiber core. These random fluctuations can be modeled as a Bragg grating with random variations in amplitude and phase along the grating length. By using this effect in three or more cores traveling within a single multi-core optical fiber of a certain length, the 3D shape and dynamics of the surface of interest can be followed.

[0044] In operation, as is known in the art of the present disclosure, an optical sensing controller 42 (e.g., an optical shape sensing or "OSS" controller) controls the generation of sensor data by an optical (shape) sensor 41 with respect to a reference coordinate system established or synchronized with the controller 42, whereby the sensor data is transmitted in a data stream of coordinate points associated with the strain sensors of the optical (shape) sensor 41 to an optical sensing (or OSS) interpreter 43.

[0045] Also as is known in the art of the present disclosure, the interpreter 43 interprets the sensor data to figure out various operating attributes of the optical (shape) sensor 41, the various operating attributes including but not limited to the shape of the whole or part of the optical sensor 41 and the attitude (e.g., position and / or orientation) of the whole or part of the optical sensor 41 with respect to the reference coordinate system.

[0046] Still referring to Figure 1 , the exemplary medical intervention system of the present disclosure also employs a process controller 50, which includes but is not limited to a path command controller 51, a device navigation controller 52, and an image marking controller 53.

[0047] In one exemplary embodiment known in the art of the present disclosure, the path command controller 51 encompasses the following structural configurations (e.g., a dedicated motherboard or an application-specific integrated circuit), which are used to autonomously control the application of commands for generating: navigating an intervention device 20 according to a planned surgical path, a planned treatment path, and / or a planned diagnostic path within an anatomical region.

[0048] In a second exemplary embodiment known in the art of the present disclosure, the path command controller 51 encompasses the following structural configurations (e.g., a dedicated motherboard or an application-specific integrated circuit), which are used to control the application of commands for generating: navigating an intervention device 20 according to a user's manipulation of a device (e.g., a robot) coupled to the intervention device 20.

[0049] In practice, the path command controller 51 can be installed within the medical imaging system 20, the OSS system 40, and / or any type of standalone workstation or linked to the medical imaging system 20, the OSS system 40, and / or any type of standalone workstation.

[0050] Still referring Figure 1 , as is known in the art of the present disclosure, the device navigation controller 52 encompasses the following structural configurations (e.g., a dedicated motherboard or an application specific integrated circuit) for controlling or managing or monitoring or supervising or processing the application of the following operations: manually navigating or automatically (e.g., robotically) navigating an interventional device 20 integrated with an optical (shape) sensor 41 along a planned surgical path, a planned treatment path, and / or a planned diagnostic path within an anatomical region.

[0051] In one exemplary embodiment, the device navigation controller 52 employs image - based techniques known in the art of the present disclosure to facilitate the display of the manual navigation of an interventional device 20 integrated with an optical (shape) sensor 41 along a planned surgical path, a planned treatment path, and / or a planned diagnostic path within an anatomical region.

[0052] In a second exemplary embodiment, the device navigation controller 52 employs system - based techniques known in the art of the present disclosure to control the robotic navigation of an interventional device 20 integrated with an optical (shape) sensor 41 along a planned surgical path, a planned treatment path, and / or a planned diagnostic path within an anatomical region.

[0053] In practice, the device navigation controller 52 can be installed within the medical imaging system 20, the system 40, and / or any type of standalone workstation or linked to the medical imaging system 20, the system 40, and / or any type of standalone workstation.

[0054] Still referring Figure 1, as will be further described in the present disclosure, the image marking controller 53 encompasses the following structural configurations (e.g., a dedicated main board or an application-specific integrated circuit) for deriving an image mark of an object of interest in the imaging space based on the following operations: generating encoded image data that provides information on the pose of the object of interest in the imaging space based on sensing by the optical (shape) sensor 41 of the navigation of the intervention device 20 in the imaging space according to an intervention process. The image mark of the object of interest in the imaging space represents training data for a machine learning model or a deep learning model. The process of training a deep learning model (e.g., a convolutional neural network (CNN), a recurrent neural network (RNN), a generative adversarial network (GAN), etc.) includes iteratively adjusting the parameters of the model such that when the model is provided with input data, the model accurately provides the corresponding expected output data. In a particular embodiment, the model can be a supervised learning model, where ground truth marks are used to evaluate the accuracy of the output data generated by the model. During training, functions such as mean absolute error, mean squared error, and cross-entropy loss can be used to compare the ground truth marks with the output data generated by the model, thereby calculating the value of the loss function and minimizing the loss function. When the output data generated by the model is within an acceptable range of the ground truth marks, the training terminates, and this acceptable range is defined by the user, or generated based on predefined parameters of acceptability determined by the user, or semi-(automatically) generated based on a parametric framework.

[0055] In practice, the image marking controller 53 can be installed within the medical imaging system 20, system 40, and / or any type of standalone workstation or linked to the medical imaging system 20, system 40, and / or any type of standalone workstation.

[0056] Further in practice, two or more of the controllers 51 - 53 can be integrated within the medical imaging system 20, system 40, and / or any type of standalone workstation.

[0057] Also in practice, the device navigation controller 52 and the image marking controller 53 constitute the intervention control system of the present disclosure.

[0058] To further facilitate understanding of the present disclosure, Figure 2 the following description teaches exemplary embodiments of a medical intervention method according to the present disclosure. According to Figure 2 the description, those of ordinary skill in the art of the present disclosure will realize how to apply the present disclosure to manufacture and use additional embodiments of the medical intervention method according to the present disclosure.

[0059] Referring to Figure 2, exemplary embodiments of the medical intervention method of the present disclosure cover a pre-intervention phase 60 and an intervention phase 70.

[0060] The pre-intervention phase 60 implements a device / sensor integration phase S62, a sensor / imaging space registration phase S64, and a device path planning phase S66 that can be performed sequentially and / or simultaneously in any manner.

[0061] In practice, the device / sensor phase S62 implements techniques known in the field of the present disclosure for integrating an optical (shape) sensor 41 ( Figure 1 ) with an intervention device 20 ( Figure 1 ) in a manner suitable for a specific intervention process.

[0062] In an exemplary embodiment of phase S62, when the optical (shape) sensor 41 is inserted into a guide wire 21 ( Figure 1 ) to form an optical sensing guide wire (which can be an optical shape sensing guide wire or a FORS guide wire), the distal end of the optical sensor 41 is aligned with the distal end of the guide wire 21.

[0063] In a second exemplary embodiment of phase S62, a 3D Hub (catheter tip connector) (Unicath, a catheter with a locking device) is attached to an over-the-wire intervention device 20 (e.g., a catheter or an IVUS probe), and a (FORS) guide wire is inserted through the 3D Hub and the over-the-wire intervention device 20.

[0064] In a third exemplary embodiment of phase S62, for an intervention device 20 that does not cross over a (FORS) guide wire, the (FORS) guide wire is rigidly fixed or integrated into the intervention device 20 at a known location. For example, when tracking a (e.g., transesophageal echocardiogram "TEE") probe, a cap with a known shape template is retrofitted to the probe, and the (FORS) guide wire is integrated into the shape template in the cap. This allows tracking of the position and orientation of the probe in six (6) degrees of freedom.

[0065] Still referring to Figure 2 , in practice, the sensor / imaging space registration phase S64 implements techniques known in the field of the present disclosure for registering an optical (shape) sensor 41 with an imaging space delineated by a medical imaging system 20 ( Figure 1 ) or with a simulated imaging space of the medical imaging system 20 (specifically by identifying the tip and body of the optical sensor 41 in the imaging space and manually aligning the drawing of the optical sensor 41 to match its position in the imaging space).

[0066] In an exemplary embodiment of stage S64, the (FORS) wire or (FORS) catheter is registered to the X-ray C-arm image by acquiring two images of the (FORS) device from different imaging angles offset by at least thirty (30) degrees. The (FORS) device is identified in each fluoroscopic image, and therefrom the registration to the C-arm isocenter and the projection of the device onto the imaging plane are calculated. This allows visualization of the (FORS) device in the imaging space (e.g., fluoroscopy and / or CT / CBCT space).

[0067] In a second exemplary embodiment of stage S64, the (FORS) wire or catheter is directly registered to the ultrasound volume or plane by manually identifying the tip and body of the (FORS) device in an (e.g., ultrasound) image and aligning the optical (shape) sensing (e.g., FORS) fiducials of the (FORS) device to match the position of the device in the image volume. Alternatively, as described above, the (FORS) device is registered to the image system (e.g., fluoroscopy or X-ray or CT), and then the (ultrasound) probe is registered to that imaging space by a 2D-3D registration method, such that the probe position and orientation are identified in the image of the imaging system, thereby enabling real-time visualization of the (FORS) device in the probe volume with both modalities registered to the imaging system.

[0068] Still referring to Figure 2 , in practice, the device path planning stage S66 may be operated by techniques known in the art of implementing the present disclosure for planning surgical, treatment, and / or diagnostic paths through an anatomical region or a model thereof.

[0069] In an exemplary embodiment of stage S66, a path through the anatomical region is generated based on a known algorithm, with the aim of minimizing the device path length and avoiding collisions with critical tissues when approaching the target.

[0070] Still referring to Figure 2 , the intervention stage 70 implements a path command stage S72, a device navigation stage S74, and an image marking stage S76.

[0071] In practice, the path command stage S72 implements techniques known in the art of implementing the present disclosure for generating navigation commands to navigate an intervention device 20 integrated with an optical (shape) sensor 41 within the registered imaging space delineated by the medical imaging system 20 or within the simulated imaging space of the registered medical imaging system 20.

[0072] In practice, the device navigation stage S74 implements techniques known in the art of practicing the present disclosure for executing navigation commands to navigate an interventional device 20 integrated with an optical (shape) sensor 41 within a registered imaging space delineated by a medical imaging system 20 or within a simulated imaging space of a registered medical imaging system 20. Examples of such techniques include, but are not limited to, forward prediction models, inverse prediction models, and imaging prediction models.

[0073] In practice, the image marking stage S76 implements techniques according to the present disclosure for obtaining an image marking 78 of an object of interest within the imaging space. As will be further described in conjunction with the illustrations of FIGS. 3-11, the image marking of the object of interest within the imaging space is derived based on generating encoded image data that provides information on the pose of the object of interest within the imaging space based on sensing the navigation of the interventional device 20 within the imaging space by the optical (shape) sensor 41.

[0074] Still referring to Figure 2 , in practice, the device navigation stage S74 and the image marking stage S74 constitute the interventional control method of the present disclosure.

[0075] Referencing Figure 3A , in Figure 2 an exemplary embodiment of the medical intervention method of Figure 3B , the draw of an optical sensing (e.g., FORS) wire and catheter in a fluoroscopic image 80 is used as a basis for the image marking 81 of the (FORS) wire and the image marking 82 of the catheter in fluoroscopy according to the flowchart 83 shown in

[0076] Specifically, once the (FORS) device is registered to the X-ray space, the projections on the fluoroscopic image 80 are directly used to derive the imaging markings 81 and 82, which have binary-encoded pixel data or voxel data that define the shape, position, and orientation of the (FORS) wire or (FORS) catheter in the fluoroscopic image 80. The imaging markings 81 and 82 are used as training data for a segmentation network to localize the wire or catheter in fluoroscopy. As a particular way of implementing the segmentation network, the segmentation network can be a convolutional encoder-decoder neural network, such as, for example, U-Net, V-Net, or SegNet.

[0077] Additionally, as described above, by attaching the 3D Hub to a catheter to register the catheter to a (FORS) guidewire, an optically sensed (e.g., FORS) guidewire can be used in combination with a non-optically sensed (e.g., non-FORS) catheter (or other device or instrument or deployable tool). This method can be used to detect / mark the ground truth position of any over-the-wire device based on the detected position of the optically sensed guidewire, which is registered to the over-the-wire device on one hand and to a fluoroscopic image (or the imaging system acquiring such an image) on the other hand. The over-the-wire device can be provided with a working channel compatible with the size of the FORS guidewire (which is slidably inserted into the over-the-wire device) and / or with a Luer lock connector at the proximal end. Such an over-the-wire device can be any navigation / support device (e.g., catheter, sheath,...), an imaging catheter (e.g., IVUS, OCT) or a therapeutic catheter (e.g., atherectomy, angioplasty balloon, aspiration).

[0078] If the end of the (FORS) guidewire is aligned with or extends beyond the end of the catheter, the visualization of the catheter on fluoroscopy can be directly used as an imaging marker. If the guidewire is retracted within the catheter, there will be no information about the shape of the distal end of the catheter. In this case, a region of interest extending from the distal end of the guidewire can be drawn, and image processing techniques known in the art of the present disclosure (e.g., edge detection and / or nearest neighbor detection) can be applied to segment and mark the distal end of the catheter.

[0079] In practice, if the virtual representation of the (FORS) device has some jitter that may result in small offset errors in the marked positions, stabilization algorithms known in the art of the present disclosure can be applied to the (FORS) device to minimize the error. Alternatively, an image marker wider than the actual device can be created to define a region of interest on the fluoroscopic image within which the device can be located. Then, image processing techniques known in the art of the present disclosure can be performed on the region of interest to segment the actual device.

[0080] Reference Figure 4A , in Figure 2 a second exemplary embodiment of the medical intervention method of Figure 4B , according to the flowchart 94 shown in

[0081] Specifically, as described above, an optical sensing (e.g., FORS) device or a non-optical sensing (e.g., non-FORS) catheter used in combination with a 3D Hub is registered to an ultrasound volume. The monolithic exchangeable device can be provided with a working channel compatible with the size of a FORS guidewire (which is slidably inserted into the monolithic exchangeable device) and / or be provided with a Luer lock connector at the proximal end. Such a monolithic exchangeable device can be any navigation / support device (e.g., catheter, sheath, …), an imaging catheter (e.g., IVUS, OCT), or a therapeutic catheter (e.g., atherectomy, angioplasty balloon, aspiration). Once the registration is complete, the representation of the (FORS) device in the ultrasound images 90 and 91 is directly used as the corresponding imaging markers 92 and 93 to define the shape, position, and orientation of the (FORS) guidewire or catheter in 2D ultrasound. The imaging markers 92 and 93 are encoded binary masks that identify which pixels or voxels represent the device in the ultrasound images 90 and 91. The (FORS) reconstruction occurs at a high frame rate (>50 Hz), thus enabling real-time manipulation and marking of the device in the ultrasound volume.

[0082] Reference Figure 5A , in Figure 2 the third exemplary embodiment of the medical intervention method according to Figure 5B the flowchart 102 shown, the 3D ultrasound image 100 is used as the basis for the image marker 101.

[0083] Specifically, as described above, an optical sensing (e.g., FORS) device or a non-optical sensing (e.g., non-FORS) catheter used in combination with a 3D Hub is registered to an ultrasound volume. The monolithic exchangeable device can be provided with a working channel compatible with the size of a FORS guidewire (which is slidably inserted into the monolithic exchangeable device) and / or be provided with a Luer lock connector at the proximal end. Such a monolithic exchangeable device can be any navigation / support device (e.g., catheter, sheath, …), an imaging catheter (e.g., IVUS, OCT), or a therapeutic catheter (e.g., atherectomy, angioplasty balloon, aspiration). Once the registration is complete, the representation of the (FORS) device in the 3D ultrasound image 100 is directly used as the corresponding imaging marker 101 to define the shape, position, and orientation of the (FORS) guidewire or catheter in 3D ultrasound. The imaging marker 101 is an encoded binary mask that identifies which pixels or voxels represent the device in the ultrasound image 100. Similarly, the (FORS) reconstruction occurs at a high frame rate (>50 Hz), thus enabling real-time manipulation and marking of the device in the ultrasound volume.

[0084] Reference Figure 6A , in Figure 2In a fourth exemplary embodiment of the medical intervention method, as described in flow chart 110, an optical sensing (e.g., FORS) guide wire is fixed to a more complex intervention device 20, and the optical sensing (e.g., FORS) guide wire is tracked in ultrasound to automatically generate device segmentation markers for a large image dataset. Similarly, segmentation markers can be obtained from a convolutional neural network (e.g., U-Net, V-Net, or SegNet).

[0085] One such intervention device can be a mitral valve repair device 111a deployed under TEE and fluoroscopy guidance. Nevertheless, this example should not be limited to such a specific device, which can be, for example, any edge-to-edge valve repair device for the tricuspid or mitral valve as an exemplary application. As Figure 6B shown, the position and orientation of the mitral valve repair device 111a can be identified by rigidly fixing an optical sensing (e.g., FORS) guide wire 112 to the distal end of the deployment system and along one wing of the clip. Although this type of tracking will be infeasible when the clip is fully deployed in a clinical case, this type of tracking can be used to generate labeled datasets in phantoms and preclinical settings.

[0086] A model of the mitral valve repair device can be positioned in the ultrasound volume 113 at the position and orientation of the mitral valve repair device as Figure 6C shown (which is determined by the (FORS) device). The angle of wing deployment can also be determined from the (FORS) device, since the (FORS) device extends along the wing of the mitral valve repair device. The model of the mitral valve repair device can be displayed in real time in the ultrasound volume, and the model can be used as an image segmentation marker 114.

[0087] Annotating such a device and its articulation in ultrasound is challenging because the articulation mechanism is non-linear due to the tendon-driven nature of the system. In this case, the optical sensing (e.g., FORS) fibers as shown in the image can greatly benefit the annotation process by providing accurate pose information as well as wing status.

[0088] Referring Figure 7 to Figure 2 In a fifth exemplary embodiment of the medical intervention method, an endoscope or bronchoscope generates RGB images of anatomical structures and often includes a working channel (through which intervention tools are deployed). It is important to be able to identify the trajectory of the intervention tool in the intraluminal image in order to be able to accurately deliver treatment or perform a biopsy. A deep learning model can be created to automatically identify the device and its trajectory in endoscopic images, which requires a large labeled image dataset that defines the position of the device in the image.

[0089] An optical sensing (e.g., FORS) guidewire can be rigidly fixed within the lumen of an interventional tool, or the interventional tool can be placed inside an optical sensing (e.g., FORS) catheter to track the position and orientation of the device. Once the device indicates that it has been registered to the image space, device markers can be automatically generated while manipulating the device within the field of view of the endoscope. The optical sensing (e.g., FORS) device within the field of view of the endoscope image 120 as shown in Figure 7 can be directly used to create device markers along the optically sensed shape (yellow) attached to the interventional tool, or an area around the optically sensed shape (yellow) can be defined to create a bounding box marker around the relevant part of the interventional tool in the image. Then, by using, for example, a convolutional neural network (CNN) for image segmentation (e.g., U-Net, V-Net, or SegNet), the centerline of the optical sensing device can be used to train the segmentation process. Then, by using, for example, YOLO or a region-based convolutional neural network (R-CNN), the defined bounding box region can be used to train the object detection network.

[0090] Similarly, an endobronchial ultrasound (EBUS) probe has a side-firing ultrasound array and a working channel through which a biopsy device or other interventional tool can be deployed. An optical sensing (e.g., FORS) guidewire registered with the EBUS image can be used to track the interventional tool, and the angular position of the tool can be detected and marked. This can then be used as ground truth training data on which an appropriate model (e.g., using a neural network) can be trained to automatically detect the tool and / or the tool trajectory in the EBUS image, thereby providing improved intraoperative guidance. The model can be a model for segmenting the image to identify the interventional tool in the image space.

[0091] Refer to Figure 8 , in Figure 2In a sixth exemplary embodiment of the medical intervention method, for the bronchoscope (or endoscope, laparoscope, colonoscope, etc.) guided procedure, it is beneficial to be able to track the endoscope itself as it is navigated through the patient's anatomy. For example, during an endobronchial biopsy procedure, bronchoscope tracking can help determine whether the physician is navigating towards the target. Several deep learning techniques or other artificial intelligence-based techniques have been explored to learn the relative motion between bronchoscope frames through image analysis. That is, by observing how image features change across multiple frames, a network or model can be trained to estimate the relative motion of the bronchoscope camera between these frames. However, obtaining ground truth data for such training is challenging because typical tracking devices are too large to be attached to the distal end of the endoscope (i.e., where the camera is located). Instead, a tracker (e.g., an optical tracker, an electromagnetic tracker, etc.) is attached to the handle of the endoscope. While this can provide somewhat accurate tracking information with some offset for rigid devices such as endoscopes and laparoscopes, for flexible devices such as bronchoscopes and colonoscopes, the tracking information recorded at the device handle does not reflect the pose of the device camera.

[0092] By fixing an optical sensing (e.g., FORS) device to the endoscope, accurate camera tracking information can be obtained by computing the point-to-point (relative) transformation between the (FORS) representations captured at time intervals corresponding to the captured camera frames. Absolute tracking information can also be obtained by acquiring a 3D image (e.g., CBCT) of the endoscope at the start of the navigation. Using the pose of the endoscope in image coordinates as the initial pose and continuously adding the relative pose from the FORS device to the initial endoscope pose, the absolute pose of the endoscope at each frame can be obtained.

[0093] The pose parameters extracted with the assistance of the (FORS) device and paired with the corresponding bronchoscope images can be used as ground truth labels for a pose estimation neural network or model that aims to predict the pose of the device based on the appearance of the bronchoscope view, as Figure 8 shown. By rigidly fixing an optical sensing (e.g., FORS) guidewire to the intervention tool, similar 3D pose ground truth labels can also be generated for the intervention tool inserted into the working channel of the endoscope (see the above section depicted by Figure 7 ). These labels can be used to train a pose estimation neural network or other model to predict the pose of the intervention tool based on the appearance of the intervention tool in the bronchoscope field of view. The pose estimation neural network can include a convolutional neural network (CNN) that regresses pose parameters (camera position and orientation) from each frame and / or a recurrent neural network (RNN) that uses multiple frames to regress pose parameters.

[0094] Another approach is to predict the absolute position and orientation of the intravascular imaging device relative to some other global coordinate system associated with the patient (e.g., a 3D model or a preoperative 3D imaging scan). For example, in order to present the position of the intravascular device on a preoperative 3D model, registration of the intravascular image with the preoperative 3D model is required. Given the registration between the preoperative 3D imaging data and the intraoperative imaging containing the FORS device (e.g., intraoperative 3D scan), the FORS device can be attached to or integrated into the intravascular imaging device to generate absolute position and / or orientation markers associated with the preoperative and / or intraoperative images. A machine learning model can utilize the markers to digest images from the intravascular device and 3D images and attempt to associate the structures visible in the intravascular image with the desired markers. Exemplary embodiments further explaining such marker learning in the context of IVUS are described below.

[0095] Reference Figure 9 , in Figure 2 the seventh exemplary embodiment of the medical intervention method, a deep neural network is often used to automatically detect anatomical features in 2D and 3D images (e.g., Figure 9 image 130 shown). Such automatic detection can be used for many applications, such as optimizing or guiding image acquisition, enhancing the visualization of certain features, or guiding a robotic system to automatically maneuver an imaging probe to an optimal imaging view or deliver treatment. Training a deep segmentation / detection network requires a large amount of data in which individual voxels / pixels are assigned classes or bounding boxes that are placed around regions of interest, and it is laborious to manually obtain these classes or bounding boxes. Optical (shape) sensing (e.g., FORS) can be used to minimize the need for manual segmentation of large anatomical datasets by alternatively marking a single 3D volume and then using an optical sensing (e.g., FORS) device to track an interventional imaging probe that images the pre-segmented volume. If images from two different modalities are registered with each other, the marking of anatomical features can be transferred from one imaging modality to another.

[0096] It may be difficult to accurately generate anatomical image markers in intravascular images (e.g., ultrasound images obtained via, for example, an IVUS catheter, or optical images obtained via, for example, an OCT catheter), while segmenting image volumes (obtained via, for example, CT or MR imaging) is often more straightforward. As an example, if a network is to be trained to automatically identify the aortic lumen and branch vessels during an IVUS pullback, this typically requires ground truth annotation on each 2D image acquired during the pullback for those anatomical features. Alternatively, ground truth labeling can be performed automatically by identifying the aortic wall and branch vessels in the image volume. Then, an optical sensing (e.g., FORS) wire is registered to the image volume, the (FORS) wire is placed within the lumen of an IVUS probe attached to a 3DHub, and the position of the IVUS probe relative to the image volume can be tracked in real time, as Figure 9 shown. When the IVUS probe is pulled back through the vasculature, anatomical markers from the image volume can be automatically transferred to the intravascular (in this example, IVUS) image at the corresponding location, as Figure 10 described in flowchart 140 in. Then, this labeling can be used to train a convolutional neural network (CNN) or other machine learning models for image segmentation (e.g., U-Net, V-Net, or SegNet).

[0097] Similar methods can also be used to transfer labels to images in other modalities, such as bronchoscopy. In practice, post-processing steps may be required to compensate for any registration errors between the image volume and the probe.

[0098] One challenge that may arise when tracking an intravascular (e.g., IVUS) probe is that optical (shape) sensing (e.g., FORS) can provide five degrees of freedom when tracking a coaxial device, which makes the rotation along the long axis of the device unknown. This problem can be alleviated for data collection purposes by creating a small bend in the lumen of the intravascular probe such that the FORS wire can detect the bend and encode the rotation of the intravascular probe.

[0099] As Figure 11As shown, when using an external transthoracic echo (TTE) probe or an internal TEE probe, anatomical image markers can also be transferred from an image volume (e.g., CT or MR imaging) to the luminal space (via an ultrasound or optical catheter). If an optical sensing (e.g., FORS) device 152 is registered to the image volume or space 150 and the intraluminal probe 151 (e.g., an ultrasound probe) is tracked by rigidly fixing the (FORS) device 152 to the probe, then when the probe is manipulated to view the anatomical structure from various angles, the markers generated in the image volume can be plotted in real time in the probe volume. If imaging the cardiac anatomy, it may be necessary to employ cardiac monitoring (e.g., ECG gating) to ensure that the image markers transferred to the probe space are acquired at the same cardiac phase as the image volume.

[0100] In both cases, optical (shape) sensing (e.g., FORS) of the probe image can be used to assist dense template-based segmentation in place of the manually annotated ground truth images for the training purposes of a convolutional neural network (CNN) or other machine learning models for image segmentation (e.g., U-Net, V-Net, or SegNet). This type of annotation can be generated in large quantities without manual input.

[0101] In the eighth exemplary embodiment of stage S62, it is useful to train a deep neural network or other machine learning models to automatically register one imaging modality to another and thereby achieve image fusion. (Deep) neural networks or other machine learning models are well-suited for the task of learning the non-linear similarity metrics or spatial relationships between different imaging modalities. A 2D-3D registration method can be utilized, where the pose of the interventional imaging probe is identified in the image generated from another imaging system. Alternatively, mutual information can be used to register two 3D volumes containing the same anatomical features to each other. The pose estimation neural network can include a convolutional neural network (CNN) that regresses the pose parameters (device position and / or orientation) from each frame and / or a recurrent neural network (RNN) that uses multiple frames to regress the pose parameters.

[0102] More specifically, deep learning models or other machine learning models can be trained that use 2D images to learn the 3D pose of the interventional imaging probe. To train such a model, a large set of 2D images visualizing the interventional imaging probe and the known 3D pose of the interventional imaging probe must be used to define the ground truth.

[0103] The optical sensing (e.g., FORS) device can be fixed to the probe (as previously in Figure 11shown) (e.g., a TEE or an external ultrasound probe (TTE)), to track the probe in six degrees of freedom. If an optical sensing (e.g., FORS) device is registered to an imaging system (e.g., an X-ray system), the pose of the probe relative to the imaging system is known. It is possible to generate a ground truth transformation matrix between the optical sensing device space and the imaging space and sensing data from the optical sensing device corresponding to each image of the distal portion (head) of the probe and use these items to provide a ground truth pose image pair (as an input for the model). Thus, the markers can be used to assist in estimating the 3D pose of the probe from 2D images based on the 2D images of the probe and the corresponding ground truth pose and transformation matrix from optical sensing, and thus the pose is in the image space.

[0104] An optical sensing (e.g., FORS) device can also be used to track an overall exchangeable imaging probe in five degrees of freedom to learn the registration between the imaging probe and the image space of an imaging system (e.g., X-ray imaging). If an optical sensing (e.g., FORS) guidewire is placed inside the lumen of a standard overall exchangeable imaging probe, the rotation about the long axis of the device is unknown. This challenge can be addressed by either of two methods: modifying the imaging probe such that there is a small bend in the lumen near the distal end, such that the (FORS) guidewire can encode the rotation of the probe; or rigidly fixing or integrating the (FORS) guidewire to the (e.g., external) surface of the probe in a known orientation.

[0105] The pose image pairs automatically generated by the (FORS) device at image acquisition can be used to train a pose estimation neural network. The pose estimation neural network can include a convolutional neural network (CNN) that regresses pose parameters (device position and / or orientation) from each frame and / or a recurrent neural network (RNN) that uses multiple frames to regress pose parameters.

[0106] The above embodiments can also improve probe pose estimation that is crucial for image fusion of ultrasound and X-ray. There are many views where shortening of a TEE or ICE or other probe or endoscopic catheter can lead to pose estimation failure or return with large errors. In such cases, knowing as much as possible about the pose (using 5DOF of an optical sensing fiber (e.g., FORS)) can help in registration in challenging views such as for transgastric views.

[0107] In addition to automatic marker generation, optical sensing (e.g., FORS)-assisted imaging can also be used to generate a synthetic dataset that can be used to train a deep learning model in settings where real training data is scarce.

[0108] In the ninth exemplary embodiment of stage S62, it may be valuable to display a simulated X-ray image to the interventional physician during the procedure even when X-rays are not actively used. Due to the large parameter space involved in sampling all possible anatomical and device configurations, accurate device simulation in generating simulated X-ray images is challenging. Data recorded from an optical (shape) sensing (e.g., FORS) guidewire can help constrain the possible configurations that a depth generation model, variational autoencoder, and / or diffusion model or other machine learning models will need to learn before being able to simulate accurate imagery, as these configurations will be those from previous procedures. A possible model can take as input the desired device configuration (which can be extracted from the last real (X-ray) image) and the background anatomical image (which can be a preoperative CT image), and generate a realistic (X-ray) image (DRR) based on these configurations. The model will be trained on image configuration pairs labeled by the (X-ray) image under the (FORS) device.

[0109] When it is difficult to obtain actual data, synthetic data generation is commonly used. Then, this synthetic data can be used to train deep learning models and is a way to build a large training dataset. Optical (shape) sensing (e.g., FORS) can be used to help build a synthetic dataset (e.g., for deep learning). An example of it is using X-ray images and predicting future X-ray images. The value of optical (shape) sensing (e.g., FORS) is that it enables the physician to navigate the interventional device without using X-rays. However, there may be procedures where FORS is not used and then the physician must rely on X-rays. It may be advantageous to develop a method of continuously showing the physician what the device is doing in the body as an "X-ray image" without actually acquiring X-ray images. In this way, X-ray images can be obtained, but future imaging frames can be predicted. To predict future X-ray image frames, a training dataset of known X-ray images (with anatomical structures and devices in the images) must be constructed. However, to build this training dataset, many X-ray images will need to be acquired. In procedures where optical (shape) sensing (e.g., FORS) is used and the (FORS) device is registered to the X-ray image space, a single X-ray image can be acquired. When navigating the (FORS) device, a synthetic image of the X-ray view can be saved by projecting the (FORS) device onto the X-ray image to essentially recreate what the X-ray view would look like. Then, this new synthetic X-ray image dataset can be used for training purposes.

[0110] Optical (shape) sensing (e.g., FORS) can also be used to label data for training a model that provides intraoperative advice or transfers knowledge from expert clinicians to less experienced clinicians. This can be, for example, a network for suggesting an optimal C-arm angle to simultaneously visualize specific anatomical features and the device. Training data can be collected while an expert clinician is navigating a FORS device that is registered to the C-arm and CT spaces. The model takes as input the position and orientation of the device relative to the anatomical features and predicts the C-arm angle of the expert user. Depending on the example, the type of neural network (e.g., CNN) to be selected can be different. In the case of the C-arm angle, pose estimation can be applied. Examples of pose estimation neural networks include a convolutional neural network (CNN) that regresses pose parameters (C-arm position and / or orientation) from each frame and / or a recurrent neural network (RNN) that uses multiple frames to regress pose parameters.

[0111] To facilitate further understanding of the various inventions of the present disclosure, Figure 12 the following description teaches exemplary embodiments of the imaging labeling controller of the present disclosure. From this description, one of ordinary skill in the art will recognize how to apply various aspects of the present disclosure to make and use additional embodiments of the image labeling controller of the present disclosure.

[0112] Referring Figure 12 , the image labeling controller 200 includes one or more processors 201, a memory 202, a user interface 203, a network interface 204, and a storage device 205 interconnected via one or more system buses 206.

[0113] As known in the art of the present disclosure or contemplated hereinafter, each processor 201 can be any hardware device capable of executing instructions stored in the memory 202 or the storage device or otherwise processing data. In a non-limiting example, the (one or more) processors 201 can include a microprocessor, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or other similar devices.

[0114] As known in the art of the present disclosure or contemplated hereinafter, the memory 202 can include various memories, including but not limited to L1, L2, or L3 cache memories or system memory. In a non-limiting example, the memory 202 can include static random access memory (SRAM), dynamic RAM (DRAM), flash memory, read only memory (ROM), or other similar memory devices.

[0115] As is known in the art of the present disclosure or contemplated hereinafter, the user interface 203 may include one or more devices for enabling communication with a user such as an administrator. In a non-limiting example, the user interface may include a command line interface or a graphical user interface that may be presented to a remote terminal via the network interface 204.

[0116] As is known in the art of the present disclosure or contemplated hereinafter, the network interface 204 may include one or more devices for enabling communication with other hardware devices. In a non-limiting example, the network interface 204 may include a network interface card (NIC) configured to communicate according to the Ethernet protocol. Additionally, the network interface 204 may implement a TCP / IP stack for communicating according to the TCP / IP protocol. Various alternative or additional hardware or configurations for the network interface 204 will be apparent.

[0117] As is known in the art of the present disclosure or contemplated hereinafter, the storage device 205 may include one or more machine-readable storage media, including but not limited to read-only memory (ROM), random access memory (RAM), disk storage media, optical storage media, flash devices, or similar storage media. The storage device may be located locally or remotely (e.g., in a cloud database). In various non-limiting embodiments, the storage device 205 may store instructions executed by the processor(s) 201, or data operated on by the processor(s) 201. For example, the storage device 205 may store a basic operating system for controlling various basic operations of the hardware. The storage device 205 also stores application modules in the form of executable software / firmware for implementing various functions of the controller 200 as previously described in the present disclosure (including but not limited to the various functions of the controller 200 as previously described in the present disclosure).

[0118] In practice, the controller 200 may be installed within a medical imaging system 20, an OSS system 40, or a stand-alone workstation (e.g., a client workstation or a mobile device such as a tablet). Alternatively, the components of the controller 200 may be a distributed medical imaging system 20, an OSS system 40, or a stand-alone workstation. Alternatively, the components of the controller 200 may be located locally and / or remotely (e.g., in a cloud database).

[0119] In practice, the present disclosure may provide a user interface for visualization of an intervention process. In Figure 13In an exemplary embodiment of the user interface of the present disclosure as shown, the user interface 300 may include various components to visualize real-time connected optical (shape) sensing (e.g., FORS) data / information and imaging data in such a way, visualize the transformation between various devices or imaging modalities, make specifications, visualize the image markers (masks) created during acquisition, and specify where to save the image markers.

[0120] Reference Figure 1-13 , those of ordinary skill in the art of the present disclosure will recognize numerous benefits of the present disclosure, including but not limited to applying the present disclosure to deep learning models developed for numerous intervention procedures in vascular, cardiac, pulmonary, orthopedic, or other clinical spaces.

[0121] The present disclosure has been described with reference to the preferred embodiments. After reading and understanding the foregoing detailed description, others may make modifications and variations. The invention is intended to be construed to include all such modifications and variations as long as they fall within the scope of the appended claims or their equivalents.

[0122] In addition, in view of the teachings provided herein, those of ordinary skill in the art should understand that the features, elements, components, etc. disclosed and described in the present disclosure / specification and / or depicted in the drawings and / or recited in the claims can be implemented in various combinations of hardware and software, and provide functions that can be combined in a single element or multiple elements. For example, the functions of the various features, elements, components, etc. shown / illustrated / depicted in the drawings and / or recited in the claims can be provided by using dedicated hardware and hardware capable of executing software associated with appropriate software. When the function is provided by a processor, the function can be provided by a single dedicated processor, a single shared processor, or multiple individual processors (some of which can be shared and / or reused). Further, the explicit use of the terms "processor" or "controller" should not be construed as specifically referring to hardware capable of executing software, and can implicitly include but not be limited to digital signal processor ("DSP") hardware, memory (e.g., read-only memory ("ROM") for storing software, random access memory ("RAM"), non-volatile storage devices, etc.), and almost any module and / or machine capable of (and / or configurable to) execute and / or control processes (including hardware, software, firmware, combinations thereof, etc.).

[0123] Moreover, all statements in this document that describe the principles, aspects, and exemplary embodiments of the present disclosure, as well as specific examples thereof, are intended to cover their structural and functional equivalents. Additionally, this document is intended to cover such equivalents including currently known equivalents and equivalents developed in the future (e.g., any elements developed that perform the same or substantially similar functions regardless of structure). Thus, for example, given the teachings provided herein, those of ordinary skill in the art will understand that any block diagrams presented herein can represent a conceptual diagram of illustrative system components and / or circuits embodying the principles of the invention. Similarly, given the teachings provided herein, those of ordinary skill in the art should understand that any flowcharts, flow diagrams, etc. can represent various processes that can be substantially represented in a computer-readable storage medium and thus executed by a computer, processor, or other device having processing capabilities, regardless of whether such computer or processor is explicitly shown.

[0124] The preferred and exemplary embodiments of the present disclosure have been described, and these embodiments are intended to be illustrative rather than restrictive. It should be noted that those of ordinary skill in the art can make modifications and variations based on the teachings provided herein (including the drawings and claims). Thus, it should be understood that changes can be made to the preferred and exemplary embodiments of the present disclosure, and such embodiments are within the scope of the present disclosure and the exemplary embodiments disclosed, described, and taught herein.

[0125] Furthermore, corresponding and / or related systems that are expected to be incorporated into and / or implemented in a device or that can be used / implemented, for example, in a device according to the present disclosure are also expected and considered to be within the scope of the present disclosure. Additionally, corresponding and / or related methods for manufacturing and / or using a device and / or system according to the present disclosure are also contemplated and considered to be within the scope of the present disclosure.

Claims

1. An interventional control system for obtaining an image marker (78) of an object of interest in an imaging space during an interventional procedure, the interventional procedure involving an interventional device being navigated in the imaging space, the interventional device integrating an optical sensing system arranged to provide optically-based sensing data, the optical sensing system being registered to the imaging space, the interventional control system comprising: An imaging marker controller (53) configured to: receive the optically-based sensing data provided by the optical sensing system navigated according to the interventional procedure, and derive the image marker (78) of the object of interest in the imaging space according to the following operations: generate encoded image data providing information on the pose of the object of interest in the imaging space based on the provided optically-based sensing data in the imaging space.

2. The intervention control system according to claim 1, wherein, The image marker (78) represents training data of a machine learning model or ground truth data of an imaging process.

3. The interventional control system according to claim 1, Among them, wherein the object of interest is one of the interventional device (30) or an anatomical object; and wherein the imaging space is one of an imaging space delineated by a medical imaging system (20) or a simulated imaging space of the medical imaging system (20).

4. The intervention control system according to claim 1, wherein, The imaging marker controller (53) is further configured to: transfer the image marker (78) of the object of interest in the imaging space from an imaging modality associated with the imaging space to a different imaging modality.

5. The interventional control system according to claim 1, wherein, The imaging marker controller (53) is further configured to: register an imaging modality associated with the image space to a different imaging modality based on the image marker (78) of the object of interest in the imaging space.

6. The interventional control system according to claim 1, further comprising an interventional device (30) integrated with an optical sensor (41) of the optical sensing system, the interventional device being operable to be navigated in the imaging space according to the interventional procedure based on the registration of the optical sensor (41) to the imaging space, and the imaging marker controller (53) is further configured to perform the derivation of the image marker (78) based on sensing the navigation of the interventional device (30) in the imaging space according to the interventional procedure by the optical sensor (41).

7. The intervention control system according to claim 6, wherein, The optical sensor is an optical shape sensor (41), and the optical shape sensor is optionally arranged to continuously sense the shape along the interventional device (30).

8. The intervention control system according to any one of the preceding claims, wherein, The optically-based data includes optically-based shape data and / or optically-based position data and / or optically-based orientation data of at least a part of the interventional device.

9. An imaging marker controller (53) comprising: At least one processor arranged to execute such instructions received from a non-transitory machine-readable storage medium encoded with instructions to obtain an image marker (78) of an object of interest within an imaging space registered to an optical sensing system integrated with an intervention device (30) navigated within the imaging space according to an intervention procedure; wherein the executed instructions are arranged to derive the image marker (78) of the object of interest within the imaging space according to the following operations: generating encoded image data providing information on the pose of the object of interest within the imaging space based on provided optical-based sensing data within the imaging space, the provided optical-based sensing data being obtained according to an intervention procedure based on optical shape sensing.

10. The imaging marker controller (53) according to claim 9, wherein, The image marker (78) represents training data of a machine learning model or ground truth data of an imaging process. wherein the object of interest is one of the intervention device (30) or an anatomical object.

11. The imaging marker controller (53) according to claim 9, Among them, the object of interest is one of the intervention device (30) or an anatomical object; and wherein the imaging space is one of an imaging space delineated by a medical imaging system (20) or a simulated imaging space of the medical imaging system (20).

12. The imaging marker controller (53) according to claim 9, wherein, The non-transitory machine-readable storage medium further includes instructions for transferring the image marker (78) of the object of interest within the imaging space from an imaging modality associated with the imaging space to a different imaging modality.

13. The imaging marker controller (53) according to claim 9, wherein, The non-transitory machine-readable storage medium further includes instructions for registering an imaging modality associated with the image space to a different imaging modality based on the image marker (78) of the object of interest within the imaging space.

14. An intervention control method for obtaining an image marker (78) of an object of interest within an imaging space during an intervention procedure, the intervention method comprising: navigating an intervention device (30) integrated with the optical sensor (41) within the imaging space according to the intervention procedure based on registration of the optical shape sensor (41) with the imaging space; and deriving the image marker (78) of the object of interest within the imaging space according to the following operations: generating encoded image data providing information on the pose of the object of interest within the imaging space based on sensing of the navigation of the intervention device (30) within the imaging space according to the intervention procedure by the optical sensor (41).

Images