System and method for generating initial transformation matrix for registering intra-operative imaging data with pre-operative imaging data

Abstract

Various systems (100) and methods (500) are provided for generating an initial transformation matrix for registering intra-operative imaging data with pre-operative imaging data. Position data of a tracked instrument moving along a trajectory corresponding to a predefined shape relative to a patient may be received (510). A first set of coordinates of the position data corresponding to the predefined shape may be determined (520). Preoperative imaging data of the patient may be received (530). A second set of coordinates of the pre-operative imaging data corresponding to the predefined shape may be determined (540). An initial transformation matrix (550) may be generated based on the first set of coordinates and the second set of coordinates. The intra-operative imaging data may be registered with the pre-operative imaging data based on the initial transformation matrix (950).

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to systems and methods for generating an initial transformation matrix for registering intraoperative imaging data with preoperative imaging data. BACKGROUND

[0002] Registration of intraoperative imaging data with preoperative imaging data has a variety of applications. For example, intraoperative imaging data acquired using a real-time imaging modality (e.g., ultrasound) can be registered with preoperative imaging data acquired using a high-resolution imaging modality (e.g., computed tomography (CT)) to enable a surgeon to guide an interventional device to a region of interest of a patient during a procedure, thereby assisting the surgeon in evaluating the region of interest or a similar region. In some cases, registration of intraoperative imaging data with preoperative imaging data can require placement of a fiducial relative to the patient during acquisition of both the intraoperative imaging data and the preoperative imaging data. Such a technique can prove to be time consuming, error prone, and / or impossible. In other cases, registration of intraoperative imaging data with preoperative imaging data can require matching anatomical landmarks in the intraoperative imaging data and the preoperative imaging data. Such a technique can also prove to be time consuming, computationally expensive, error prone, etc. SUMMARY

[0003] This summary introduces concepts that are more fully described in the detailed description. It is not intended to determine the essential features of the claimed subject matter nor is it intended to limit the scope of the claimed subject matter.

[0004] In one aspect, a method can include receiving position data of a tracked instrument moving along a trajectory corresponding to a predefined shape relative to a patient; determining a first set of coordinates of the position data corresponding to the predefined shape; receiving preoperative imaging data of the patient; determining a second set of coordinates of the preoperative imaging data corresponding to the predefined shape; generating an initial transformation matrix based on the first set of coordinates and the second set of coordinates; and registering intraoperative imaging data with the preoperative imaging data based on the initial transformation matrix.

[0005] In another aspect, an apparatus can include a memory configured to store instructions; and one or more processors configured to execute the instructions to perform operations comprising: receiving position data of a tracked instrument moving along a trajectory corresponding to a predefined shape relative to a patient; determining a first set of coordinates of the position data corresponding to the predefined shape; receiving preoperative imaging data of the patient; determining a second set of coordinates of the preoperative imaging data corresponding to the predefined shape; generating an initial transformation matrix based on the first set of coordinates and the second set of coordinates; and registering intraoperative imaging data with the preoperative imaging data based on the initial transformation matrix.

[0006] In yet another aspect, a non-transitory computer-readable medium can store instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising: receiving position data of a tracked instrument moving along a trajectory corresponding to a predefined shape relative to a patient; determining a first set of coordinates of the position data corresponding to the predefined shape; receiving pre-operative imaging data of the patient; determining a second set of coordinates of the pre-operative imaging data corresponding to the predefined shape; generating an initial transformation matrix based on the first set of coordinates and the second set of coordinates; and registering intra-operative imaging data with the pre-operative imaging data based on the initial transformation matrix. BRIEF DESCRIPTION OF DRAWINGS

[0007] Figure 1 is a diagram of an example system for generating an initial transformation matrix for registering intra-operative imaging data with pre-operative imaging data based on position data of a tracked instrument moving along a trajectory corresponding to a predefined shape relative to a patient and metadata of the pre-operative imaging data.

[0008] Figure 2 is a diagram of example components of a system.

[0009] Figure 3 is Figure 1 a diagram of an example device of an ultrasound system of

[0010] Figure 4 is Figure 1 a diagram of an example device of a tracking system of

[0011] Figure 5 is a flowchart of an example process for generating an initial transformation matrix for registering intra-operative imaging data with pre-operative imaging data based on position data of a tracked instrument moving along a trajectory corresponding to a predefined shape relative to a patient.

[0012] Figure 6 is a diagram of an example movement of a tracked instrument moving along a trajectory corresponding to a predefined shape relative to a patient and example coordinates of position data of the tracked instrument.

[0013] Figure 7 is a diagram of a set of example coordinates of pre-operative imaging data corresponding to the Figure 6 predefined shape in

[0014] Figure 8 is a diagram of a process of generating an initial transformation matrix based on position data of a tracked instrument moving along a trajectory corresponding to a predefined shape relative to a patient in Figure 6 and a set of example coordinates of pre-operative imaging data corresponding to the Figure 6 predefined shape in Figure 7 ​

[0015] Figure 9 is a flowchart of an example process for displaying images for guiding a procedure using intraoperative imaging data that uses an initial transformation matrix generated by a system of Figure 1 is registered with preoperative imaging data using an initial transformation matrix generated by a system of

[0016] Figure 10 is a diagram of an example process for displaying images for guiding a procedure using intraoperative imaging data that uses an initial transformation matrix generated by a system of Figure 1 is registered with preoperative imaging data using an initial transformation matrix and a final transformation matrix generated by a system of

[0017] Figure 11 is a diagram of training, deploying, and updating an artificial intelligence (AI) model to generate an initial transformation matrix. DETAILED DESCRIPTION

[0018] As described above, techniques for registration of intraoperative imaging data with preoperative imaging data can prove to be time consuming, error prone, impossible, etc. Moreover, these techniques can require use of various fiducials, can require medical personnel to perform specific tasks, can require precise alignment of a patient with respect to an imaging device, etc.

[0019] Some embodiments herein relate to a technique for generating an initial transformation matrix for registering intraoperative imaging data with preoperative imaging data using position data of a tracked instrument moving along a trajectory corresponding to a predefined shape with respect to a patient and metadata of the preoperative imaging data. In this way, some embodiments herein provide a fast, efficient, and easy technique for generating an initial transformation matrix. Moreover, in this way, some embodiments herein can improve robustness of a final registration algorithm, thereby improving accuracy of registration of intraoperative imaging data with preoperative imaging data.

[0020] Figure 1 is a diagram of an example system 100 for generating an initial transformation matrix for registering intraoperative imaging data with preoperative imaging data based on position data of a tracked instrument moving along a trajectory corresponding to a predefined shape with respect to a patient and metadata of the preoperative imaging data. As shown in Figure 1 System 100 can include a registration system 110, an intraoperative imaging system 130, a tracking system 150, a preoperative imaging system 170, and a network 190.

[0021] The registration system 110 can be configured to generate an initial transformation matrix for registering intraoperative imaging data with preoperative imaging data based on position data of a tracked instrument moved along a trajectory corresponding to a predefined shape relative to a patient and metadata of the preoperative imaging data. For example, the registration system 110 can be a server, a workstation, a medical device, a computer, etc.

[0022] The intraoperative imaging system 130 can be configured to acquire intraoperative imaging data of a region of interest of a patient. For example, the intraoperative imaging system 130 can be an ultrasound system, such as a 2D ultrasound system, a 3D ultrasound system, a 4D ultrasound system, a Doppler ultrasound system, etc. Or, the intraoperative imaging system can be a system of an imaging modality different from ultrasound, such as a photoacoustic imaging system, a thermoacoustic imaging system, etc. The intraoperative imaging data can be acquired, generated, or the like during an occurrence of a medical procedure involving the patient.

[0023] The tracking system 150 can be configured to acquire tracking data. For example, the tracking system 150 can be an electromagnetic tracking system, an optical tracking system, an acoustic tracking system, an inertial tracking system, etc.

[0024] The preoperative imaging system 170 can be configured to acquire preoperative imaging data of a region of interest of a patient. For example, the preoperative imaging system 170 can be a CT system, a magnetic resonance imaging (MRI) system, an ultrasound system, an X-ray system, a positron emission tomography (PET) device, etc. The preoperative imaging data can be acquired, generated, or the like before an occurrence of a medical procedure involving the patient.

[0025] The network 190 can allow for communication between the registration system 110, the intraoperative imaging system 130, the tracking system 150, and the preoperative imaging system 170. For example, the network 190 can be a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a cellular network, a private network, an ad hoc network, an intranet, the Internet, a fiber-based network, etc., and / or a combination of these or other types of networks.

[0026] The number and arrangement of systems of the system 100 are provided as examples. In practice, the system 100 can include additional systems, fewer systems, different systems, or differently arranged systems than those shown. Figure 1 Additionally or alternatively, a set of systems (e.g., one or more systems) of the system 100 can be integrated into a single system and / or perform one or more functions described as being performed by another system or set of systems of the system 100.

[0027] Figure 2is a diagram of example components of a system 200. The system 200 can correspond to the registration system 110, the intraoperative imaging system 130, the tracking system 150, and / or the preoperative imaging system 170. As shown, the system 200 can include a bus 210, a processor 220, a memory 230, a storage component 240, an input component 250, an output component 260, and a communication interface 270. Figure 2

[0028] The bus 210 includes a component that permits communication among the components of the system 200. The processor 220 can be implemented in hardware, firmware, or a combination of hardware and software. The processor 220 can be a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or another type of processing component.

[0029] The processor 220 can include one or more processors capable of being programmed to perform functions. The processor 220 can include one or more processors 220 configured to perform the operations described herein. For example, a single processor 220 can be configured to perform all of the operations described herein. Alternatively, a plurality of processors 220 can be collectively configured to perform all of the operations described herein, and each processor 220 of the plurality of processors 220 can be configured to perform a subset of the operations described herein. For example, a first processor 220 can perform a first subset of the operations described herein, a second processor 220 can be configured to perform a second subset of the operations described herein, and so on.

[0030] The memory 230 can include random access memory (RAM) including a dynamic or static RAM, read-only memory (ROM), and / or another type of dynamic or static memory (e.g., flash memory, magnetic memory, and / or optical memory) that stores information and / or instructions for use by the processor 220.

[0031] The storage component 240 can store information and / or software related to the operation and use of the system 200. For example, the storage component 240 can include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optic disk, and / or a solid state disk), a compact disc (CD), a digital versatile disc (DVD), a flash drive, a memory stick, a floppy disk, a cassette, a tape, and / or another type of non-transitory computer-readable medium, along with a corresponding drive.

[0032] ​Input component 250 can include a component that permits the system 200 to receive information, such as via user input (e.g., a touch screen display, a keyboard, a keypad, a mouse, a button, a switch, a camera, and / or a microphone). Additionally, or alternatively, input component 250 can include a sensor (e.g., a global positioning system (GPS) component, an accelerometer, a gyroscope, and / or an actuator) for sensing information. Output component 260 can include a component that provides output information from the system 200 (e.g., a display, a speaker, and / or one or more light-emitting diodes (LEDs)).

[0033] Communication interface 270 can include a transceiver-like component (e.g., a transceiver and / or a separate receiver and transmitter) that enables the system 200 to communicate with other systems, such as via wired connections, wireless connections, or a combination of wired and wireless connections. Communication interface 270 can allow the system 200 to receive information from another system and / or provide information to another system. For example, communication interface 270 can include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a Bluetooth® interface, and / or a cellular network interface.

[0034] The system 200 can perform one or more processes described herein. The system 200 can perform these processes based on processor 220 executing software instructions stored by a non-transitory computer-readable medium, such as memory 230 and / or storage component 240. A computer-readable medium can be defined as a non-transitory memory device. A memory device can include memory space within a single physical storage device or memory space spread across multiple physical storage devices.

[0035] The software instructions stored in memory 230 and / or storage component 240, when executed, can cause the processor 220 to perform one or more processes described herein. Additionally or alternatively, hardwired circuitry can be used in place of or in combination with software instructions to perform one or more processes described herein. Thus, the specific implementations described herein are not limited to any specific combination of hardware circuitry and software.

[0036] Figure 2 The number and arrangement of components shown in FIG. 1 are provided as an example. In practice, system 200 can include additional components, fewer components, different components, or differently arranged components than those shown in FIG. 1. Additionally, or alternatively, one or more components of system 200 can perform one or more other functions of another one or more components. Figure 2The components of system 200 can be configured to perform one or more functions described herein. For example, one or more components of system 200 can be configured to perform one or more functions described herein as being performed by another component of system 200. As another example, one or more components of system 200 can be configured to perform one or more functions described herein as being performed by a component of system 100.

[0037] Figure 3 is Figure 1 a diagram of an example device of intraoperative imaging system 130. As shown in Figure 3 intraoperative imaging system 130 can be an ultrasound system and can include ultrasound probe 131, transmit beamformer 132, transmitter 133, elements 134, receiver 135, receive beamformer 136, user input device 137, processor 138, display 139, memory 140, and communication interface 141. The aforementioned components can be connected via wired or wireless connections.

[0038] Ultrasound probe 131 can be configured to acquire ultrasound data. For example, ultrasound probe 131 can be a linear probe, a phased array probe, a curved linear probe coupled with a position tracking system, a mechanically steered linear array transducer, a phased array transducer, a curved linear array transducer, an electronically steered 2D transducer array, an electronic 3D (e3D) probe, an electronic 4D (e4D) probe, a thin-profile wearable patch version of any of the aforementioned probes, etc. According to one embodiment, ultrasound probe 131 can be configured to generate an ultrasound signal, transmit the ultrasound signal toward a region of interest of a subject, receive echo ultrasound signals backscattered from the region of interest of the subject, generate ultrasound data based on the echo ultrasound signals, and output the ultrasound data. The region of interest can be any region of the subject’s anatomy. The subject can be a human, an animal, a phantom, etc.

[0039] Transmit beamformer 132 can be configured to apply a delay time to an electrical signal provided to elements 134 to focus a corresponding ultrasound signal at the region of interest. Transmitter 133 can be configured to transmit an electrical signal to elements 134 to drive elements 134 to transmit an ultrasound signal toward the region of interest. Elements 134 can be configured to receive the electrical signal from transmitter 133, convert the electrical signal to an ultrasound signal, and transmit the ultrasound signal toward the region of interest. Elements 134 can be configured to receive echo ultrasound signals backscattered by the region of interest, convert the echo ultrasound signals to electrical signals, and provide the electrical signals to receiver 135. Receiver 135 can be configured to receive the electrical signals from elements 134 and provide the electrical signals to receive beamformer 136. Receive beamformer 136 can apply a delay time to the electrical signals received from elements 134.

[0040] The user input device 137 can be configured to receive user input and provide the user input to the processor 138. For example, the user input device 137 can be a touchscreen display, a keyboard, a keypad, a mouse, a button, a switch, a microphone, etc. Additionally or alternatively, the user input device 137 can be configured to sense information. For example, the user input device 137 can sense information from an electromagnetic positioning system, an inertial measurement system, an accelerometer, a gyroscope, an actuator, etc.

[0041] The processor 138 can be configured to perform the operations described herein. For example, the processor 138 can be a CPU, a GPU, an APU, a microprocessor, a microcontroller, a DSP, a FPGA, an ASIC, etc. The processor 138 can be implemented in hardware, firmware, or a combination of hardware and software. The processor 138 can include one or more processors 138 configured to perform the operations described herein. For example, a single processor 138 can be configured to perform all of the operations described herein. Alternatively, multiple processors 138 can be collectively configured to perform all of the operations described herein, and each processor 138 of the multiple processors 138 can be configured to perform a subset of the operations described herein. For example, a first processor 138 can perform a first subset of the operations described herein, a second processor 138 can be configured to perform a second subset of the operations described herein, etc.

[0042] The processor 138 can be configured to control the ultrasound probe 131 to acquire ultrasound data. The processor 138 can be configured to control which elements 134 are active and to control the shape of the beams emitted from the ultrasound probe 131. The processor 138 can generate ultrasound images for display. For example, the processor 138 can generate B-mode images, color Doppler images, M-mode images, color M-mode images, etc. The ultrasound images can be 3D images, 2D images, single plane images, biplane images, triplane images, multiplane images, etc. The ultrasound images can correspond to various anatomical planes of interest (e.g., sagittal, coronal, and transverse).

[0043] The display 139 can be configured to display information. For example, the display 139 can be a monitor, an LED display, a cathode ray tube, a projector display, a touchscreen, a tablet, a mobile phone, etc. The display 139 can display ultrasound images based on ultrasound data in real time. For example, the display 139 can display ultrasound images within one second, two seconds, five seconds, etc. of ultrasound data acquired by the ultrasound probe 131.

[0044] The memory 140 can be configured to store information and / or instructions for use by the processor 138. The memory 140 can be a non-transitory computer-readable medium. For example, the memory 140 can be RAM, ROM, flash memory, magnetic storage, optical storage, etc. The memory 140 can be configured to store instructions that, when executed by the processor 138, cause the processor 138 to perform the operations described herein.

[0045] The communication interface 141 can be configured to enable the processor 138 to communicate with other systems, such as via a wired connection, a wireless connection, or a combination of a wired connection and a wireless connection. For example, the communication interface 141 can include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, an RF interface, a USB interface, a Wi-Fi interface, a cellular network interface, etc.

[0046] Figure 3 The number and arrangement of components shown in the intraoperative imaging system 130 are provided as an example. In practice, the intraoperative imaging system 130 can include additional components, fewer components, different components, or differently arranged components than those depicted in the example. Figure 3 Additionally or alternatively, a set of components (e.g., one or more components) of the intraoperative imaging system 130 can perform one or more functions described as being performed by another set of components of the intraoperative imaging system 130.

[0047] Figure 4 is a diagram of an example device of the tracking system 150. As Figure 4 shown, the tracking system 150 can be an electromagnetic tracking system, and can include an emitter 151, a receiver 152, a user input device 153, a processor 154, a display 155, a memory 156, and a communication interface 157.

[0048] The emitter 151 can be configured to generate a magnetic field. The receiver 152 can be configured to output a signal in response to the magnetic field generated by the emitter 151. The processor 154 can receive the output signal from the receiver 152, and acquire tracking data that identifies a position and / or orientation of the receiver 152. The receiver 152 can be attached to, integral with, disposed in, etc. a tracked instrument. For example, according to one embodiment, the receiver 152 can be attached to the ultrasound probe 131 to track a position and / or orientation of the ultrasound probe 131. Alternatively, the receiver 152 can be attached to a scan gun or other handheld device to track a position and / or orientation of the scan gun or other handheld device. Alternatively, the receiver 152 can be attached to an interventional device to track a position and / or orientation of the interventional device. The interventional device can be a catheter, a needle, etc.

[0049] The user input device 153 can be configured to receive user input and provide the user input to the processor 154. For example, the user input device 153 can be a touchscreen display, a keyboard, a keypad, a mouse, a button, a switch, a microphone, etc. Additionally or alternatively, the user input device 153 can be configured to sense information. For example, the user input device 153 can sense information from an electromagnetic positioning system, an inertial measurement system, an accelerometer, a gyroscope, an actuator, etc.

[0050] The processor 154 can be configured to perform the operations described herein. For example, the processor 154 can be a CPU, a GPU, an APU, a microprocessor, a microcontroller, a DSP, a FPGA, an ASIC, etc. The processor 154 can be implemented in hardware, firmware, or a combination of hardware and software. The processor 154 can include one or more processors 154 configured to perform the operations described herein. For example, a single processor 154 can be configured to perform all of the operations described herein. Alternatively, multiple processors 154 can be collectively configured to perform all of the operations described herein, and each processor 154 of the multiple processors 154 can be configured to perform a subset of the operations described herein. For example, a first processor 154 can perform a first subset of the operations described herein, a second processor 154 can be configured to perform a second subset of the operations described herein, etc.

[0051] The processor 154 can be configured to control the transmitter 151 to acquire tracking data. The processor 154 can be configured to control excitation of the transmitter 151 to generate a magnetic field. The processor 154 can acquire tracking data based on controlling the transmitter 151.

[0052] The display 155 can be configured to display information. For example, the display 155 can be a monitor, an LED display, a cathode ray tube, a projector display, a touchscreen, a tablet, a mobile phone, etc. The display 155 can display tracking data in real time. For example, the display 155 can display tracking data within one second, two seconds, five seconds, etc. of acquired tracking data.

[0053] The memory 156 can be configured to store information and / or instructions for use by the processor 154. The memory 156 can be a non-transitory computer-readable medium. For example, the memory 156 can be a RAM, a ROM, a flash memory, a magnetic storage, an optical storage, etc. The memory 140 can be configured to store instructions that, when executed by the processor 155, cause the processor 155 to perform the operations described herein.

[0054] The communication interface 157 can be configured to enable the processor 154 to communicate with other systems, such as via a wired connection, a wireless connection, or a combination of a wired connection and a wireless connection. For example, the communication interface 157 can include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, an RF interface, a USB interface, a Wi-Fi interface, a cellular network interface, etc.

[0055] Figure 4 The number and arrangement of components of the tracking system 150 shown are provided as examples. In implementation, the tracking system 150 may include additional components, fewer components, different components, or components that are different from the components shown. Figure 4 Additionally or alternatively, one set of components (eg, one or more components) of tracking system 150 may perform one or more functions described as being performed by another set of components of tracking system 150 .

[0056] although Figure 4 Tracking system 150 is depicted as an electromagnetic tracking system, but it should be understood that embodiments herein are applicable to other types of tracking systems, such as optical tracking systems, acoustic tracking systems, ultrasonic tracking systems, and the like.

[0057] Figure 5 is a flow chart of an example process 500 for generating an initial transformation matrix for registering intraoperative imaging data with preoperative imaging data based on position data of a tracked instrument moving along a trajectory corresponding to a predefined shape relative to a patient. Figure 6 is a diagram 600 of example movements of a tracked instrument along a trajectory corresponding to a predefined shape relative to a patient and example coordinates of position data for the tracked instrument. Figure 7 is with Figure 6 700 is an illustration of a set of example coordinates of preoperative imaging data corresponding to a predefined shape in FIG.

[0058] like Figure 5 As shown, process 500 may include receiving position data for a tracked instrument moving along a trajectory corresponding to a predefined shape relative to a patient (operation 510 ).

[0059] The tracked instrument can be any instrument capable of being tracked. For example, the tracked instrument can be the ultrasound probe 131 of the intraoperative system 130. Additionally or alternatively, the tracked instrument can be the ultrasound probe 131 including the receiver 152 of the tracking system 150. Alternatively, the tracked instrument can be a device including the receiver 152 of the tracking system 150. Alternatively, the tracked instrument can be a device that does not include the receiver of the tracking system 150. Alternatively, the tracked instrument can be a device such as a scanner, a handheld object, or the like.

[0060] The predefined shape can be any suitable shape. For example, the predefined shape can be an L-shape, a cross, a square, a rectangle, a line, an arc, etc. According to one embodiment, the predefined shape can be a two-dimensional shape. For example, the predefined shape can extend along two axes. Alternatively, the predefined shape can be a three-dimensional shape. For example, the predefined shape can extend along three axes.

[0061] The tracked instrument can be moved along a trajectory corresponding to a predefined shape relative to the patient. For example, the operator can move the tracked instrument along a trajectory corresponding to a predefined shape relative to the patient. The trajectory can be along the sagittal, coronal, and / or transverse planes of the patient. The trajectory can be up-down, down-down, medial-lateral, medial-lateral, anterior-posterior, posterior-anterior, etc.

[0062] For example, Figure 6 As shown, the predefined shape may be an L-shape. In this case, the operator may move the tracked instrument along a trajectory corresponding to the L-shape by moving the tracked instrument up and down 610 and sideways 620 .

[0063] When the operator moves the tracked instrument along the trajectory relative to the patient, the tracking system 150 can track the tracked instrument. The tracking system 150 can send position data of the tracked instrument to the registration system 110. The registration system 110 can receive position data of the tracked instrument moving along the trajectory corresponding to the predefined shape relative to the patient from the tracking system 150.

[0064] According to one embodiment, the registration system 110 may display guidance information that guides the operator to move the tracked instrument along a trajectory corresponding to the predefined shape. For example, the registration system 110 may display a visual representation of the patient and a visual representation of the predefined shape relative to the patient. Additionally or alternatively, the registration system 110 may display a real-time view of the tracked instrument and may display a guidance indicator that guides the direction of movement of the tracked instrument. According to one embodiment, the registration system 110 may determine that the predefined shape has been moved along a trajectory corresponding to the predefined shape relative to the patient based on the position data. The registration system 110 may display information indicating that the operator has successfully moved the tracked instrument along the trajectory corresponding to the predefined shape relative to the patient.

[0065] like Figure 5 As further shown, process 500 may include determining a first set of coordinates for position data corresponding to a predefined shape (operation 520 ).

[0066] The first set of coordinates of the position data corresponding to the predefined shape may include one or more coordinates of the position data corresponding to the predefined shape. For example, the first set of coordinates may include coordinates of one or more points along a trajectory corresponding to the predefined shape relative to the patient. For example, Figure 6As shown, the first set of coordinates can include coordinates 630 of a point Ml corresponding to a first position of the trajectory, coordinates 640 of a point M2 corresponding to a last point of the trajectory, and coordinates 650 of a point M3 corresponding to a point where the movement of the tracked instrument changes from a superior-inferior direction to a lateral direction.

[0067] The first set of coordinates can be coordinates of an intraoperative coordinate system. The intraoperative coordinate system can be a coordinate system of the intraoperative imaging system 130 and / or the tracking system 150. For example, the intraoperative coordinate system can be a coordinate system used for intraoperative images generated during surgery.

[0068] According to one embodiment, the registration system 110 can fit a plane to a set of 3D points representing a predefined shape. The registration system 110 can project each point onto the plane. The registration system 110 can apply a piecewise linear function having two segments to the projected points. In this way, the registration system 110 can identify a main break point in the piecewise linear function as well as a start point and an end point of the function. The registration system 110 can determine the first set of coordinates using these points.

[0069] As shown, the first set of coordinates can include coordinates 630 of a point Ml corresponding to a first position of the trajectory, coordinates 640 of a point M2 corresponding to a last point of the trajectory, and coordinates 650 of a point M3 corresponding to a point where the movement of the tracked instrument changes from a superior-inferior direction to a lateral direction. Figure 5 As further shown, the process 500 can include receiving preoperative imaging data of the patient (operation 530) and determining a second set of coordinates of the preoperative imaging data corresponding to the predefined shape (operation 540).

[0070] The registration system 110 can receive preoperative imaging data of the patient from the preoperative imaging system 170 and determine a second set of coordinates of the preoperative imaging data corresponding to the predefined shape. The second set of coordinates can be coordinates of a preoperative coordinate system. The preoperative coordinate system can be a coordinate system of the preoperative imaging system 170 and / or a tracking system 150 associated with the preoperative imaging system 170.

[0071] The registration system 110 can determine the second set of coordinates using metadata of the preoperative imaging data. For example, the preoperative imaging data can be in a Digital Imaging and Communications in Medicine (DICOM) format. In this case, the registration system 110 can determine the second set of coordinates using a patient position (0018, 5100) attribute that specifies a position of the patient relative to an imaging device space of the preoperative imaging system 170. Additionally or alternatively, the registration system 110 can determine the second set of coordinates using an image orientation (0020, 0037) attribute that specifies an orientation of image rows and columns relative to the patient. Additionally or alternatively, the registration system 110 can determine the second set of coordinates using an image position (0020, 0032) attribute that specifies x, y, and z coordinates of a top-left corner of the preoperative image data. Additionally or alternatively, the registration system 110 can determine the second set of coordinates using a pixel spacing (0028, 0030) attribute that specifies a position and orientation of image layers relative to a patient-based coordinate system. Additionally or alternatively, the registration system 110 can determine the second set of coordinates using any other attribute.

[0072] The registration system 110 can use any of the aforementioned attributes to determine the second set of coordinates in the preoperative imaging data in a manner that uses a center of the preoperative imaging data as a reference point. Alternatively, the registration system 110 can use another portion of the preoperative imaging data as the reference point. The registration system 110 can determine the second set of coordinates based on the reference point. For example, the registration system 110 can determine a first point in the preoperative imaging data that is spaced a first predetermined distance from the reference point in a first direction, determine a second point in the preoperative imaging data that is spaced a second predetermined distance from the reference point in a second direction, determine a third point in the preoperative imaging data that is spaced a third predetermined distance from the reference point in a third direction, etc.

[0073] The second set of coordinates of the surgical imaging data corresponding to the predefined shape can include one or more coordinates of the surgical imaging data corresponding to the predefined shape. For example, the second set of coordinates can include coordinates of one or more points along the opposing predefined shape. For example, as shown in FIG. 7, the second set of coordinates can include a coordinate 710 of a point M1’ corresponding to a first location of the predefined shape, a coordinate 720 of a point M2’ corresponding to a last point of the predefined shape, and a coordinate 730 of a point M3’ corresponding to a point where a direction of the predefined shape changes from a superior-inferior direction to a lateral direction. Figure 7

[0074] It will be appreciated that the predefined shape of the second set of coordinates of the preoperative imaging data corresponds to the predefined shape of the first set of coordinates of the location data. In other words, referring to FIGS. 6 and 7, Figure 6 and Figure 7 the predefined shape defined by the coordinates 630, 640, and 650 corresponds to the same predefined shape defined by the coordinates 710, 720, and 730.

[0075] According to one embodiment, the registration system 110 can determine the predefined shape based on the first set of coordinates, and determine the second set of coordinates based on determining the predefined shape. Additionally or alternatively, the registration system 110 can determine distances between the first set of coordinates, and determine the second set of coordinates based on the distances. For example, the registration system 110 can determine the second set of coordinates to include distances that are the same or similar to the distances between the first set of coordinates.

[0076] According to one embodiment, the registration system 110 can determine an orientation of the patient relative to the predefined shape formed by the trajectory of the tracked instrument, and determine an orientation of the patient relative to the preoperative imaging data. The registration system 110 can determine the second set of coordinates based on the aforementioned orientations. That is, the registration system 110 can determine the second set of coordinates to include the same orientation relative to the patient as the first set of coordinates.

[0077] As Figure 5 ​Further, process 500 can include generating an initial transformation matrix based on the first set of coordinates and the second set of coordinates (operation 550).

[0078] The initial transformation matrix can transform coordinates of the pre-operative coordinate system to coordinates of the intra-operative coordinate system. The registration system 110 can generate the initial transformation matrix using the first set of coordinates and the second set of coordinates. For example, the registration system 110 can generate the initial transformation matrix using a transformation matrix generation technique.

[0079] The initial transformation matrix can represent a rigid transformation that defines an alignment between the two sets of corresponding 3D point data derived from the pre-operative source and the intra-operative source. The initial transformation matrix can describe how the pre-operative imaging data should be rotated and translated to best match the intra-operative imaging data.

[0080] The registration system 110 can identify corresponding points between the pre-operative imaging data and the position data of the tracked instrument. For example, the registration system 110 can identify corresponding points between the second set of coordinates and the first set of coordinates. Subsequently, the registration system 110 can determine an alignment between the second set of coordinates and the first set of coordinates by computing the initial transformation matrix. The registration system 110 can minimize or reduce the least-squares error between each point from the second set of coordinates and the first set of coordinates. For example, the registration system 110 can compute the least-squares error between each point from the second set of coordinates and the first set of coordinates between the pre-operative imaging data and the position data of the tracked instrument using the following equation:

[0081]

[0082] In the above equation, err represents the least-squares error, n is the number of coordinates in each set, M and M' are the first set of coordinates and the second set of coordinates, respectively, R is a 3x3 rotation matrix, and t is a translation vector. Subsequently, the alignment parameters (translation, rotation) are adjusted using an iterative optimization algorithm to minimize these errors. The initial transformation matrix can include both rotation and translation and serves as a basis for aligning the pre-operative imaging data and the intra-operative imaging data.

[0083] Figure 8 is based on position data of the tracked instrument moving along a trajectory corresponding to the predefined shape in the pre-operative image and position data of the tracked instrument moving along a trajectory corresponding to the predefined shape in the intra-operative image. Figure 6 the position data of the tracked instrument moving relative to the predefined shape in the pre-operative image and the position data of the tracked instrument moving relative to the predefined shape in the intra-operative image. Figure 6 the position data of the tracked instrument moving relative to the predefined shape in the pre-operative image and the position data of the tracked instrument moving relative to the predefined shape in the intra-operative image. Figure 7 A diagram of a process 800 of generating an initial transformation matrix from a set of example coordinates of pre-operative imaging data in the pre-operative image and position data of a tracked instrument moving relative to a predefined shape in the intra-operative image. As Figure 8As shown, the registration system 110 can use the initial transformation matrix 810 to transform coordinates of the preoperative coordinate system to coordinates of the intraoperative coordinate system. For example, the registration system 110 can transform the coordinates 710, 720, and 730 to the coordinates 630, 640, and 650, respectively. As shown by reference number 820, after applying the transformation matrix 810, the coordinates 710, 720, and 730 can be more closely aligned with the coordinates 630, 640, and 650, respectively.

[0084] Although process 500 depicts a particular set of operations and a particular order of operations, it will be appreciated that, in other implementations, process 500 can include fewer operations, more operations, or operations arranged in a different manner.

[0085] Figure 9 is a flowchart of an example process 900 for displaying images for guiding a procedure using intraoperative imaging data that uses an initial transformation matrix generated by Figure 1 the system of is registered with preoperative imaging data.

[0086] As Figure 9 shown, process 900 can include receiving an initial transformation matrix (operation 910). The initial transformation matrix can be the initial transformation matrix generated in operation 550 of Figure 5 the system of .

[0087] As Figure 9 further shown, process 900 can include transforming preoperative imaging data of a patient using the initial transformation matrix (operation 920). For example, the registration system 110 can transform coordinates of a preoperative coordinate system to coordinates of an intraoperative coordinate system.

[0088] As Figure 9 further shown, process 900 can include receiving intraoperative imaging data of the patient (operation 930). For example, the registration system 110 can receive intraoperative imaging data of the patient from the intraoperative imaging system 130.

[0089] As Figure 9Further, process 900 can include generating a final transformation matrix (operation 940). Based on the initial transformation matrix being generated, the operator begins scanning the patient using the ultrasound probe 131 being tracked by the tracking system 150. After the ultrasound data acquisition is complete, the registration system 110 can perform a three-dimensional segmentation of the same underlying anatomy within both the intraoperative imaging data and the preoperative imaging data. The registration system 110 can transform the 3D segmented volume from the preoperative imaging data using the initial transformation matrix. This technique ensures that both the preoperative coordinate system and the intraoperative coordinate system are close to each other, thereby simplifying and improving the final registration process. According to one embodiment, the registration system 110 can use a coherent point drift (CPD) algorithm for the final registration. The CPD algorithm is a probabilistic method of aligning a set of points, such as segmented structures from the preoperative imaging data and the intraoperative imaging data. The CPD algorithm utilizes segmented structures from both the intraoperative imaging data and the preoperative imaging data in the form of point clouds to determine the final registration. In this way, the CPD algorithm compares the spatial relationship between points in both the preoperative imaging data and the intraoperative imaging data to find the best or improved alignment.

[0090] In summary, the concept of utilizing an initial transformation is to align the coordinate systems within both the preoperative imaging data and the intraoperative imaging data, thereby enhancing the robustness during registration using a final algorithm. Without employing this initial transformation, the difference between the preoperative coordinate system and the intraoperative coordinate system can be quite large, which can result in algorithm failure. However, by computing an initial transformation matrix to ensure that both the preoperative coordinate system and the intraoperative coordinate system are closely aligned, the embodiments herein improve the resiliency of the algorithm.

[0091] As Figure 9 Further, process 900 can include registering the intraoperative imaging data of the patient with the preoperative imaging data of the patient using the final transformation matrix (operation 950). For example, the registration system 110 can register the intraoperative imaging data with the preoperative imaging data received from the preoperative imaging system 170 using the final transformation matrix.

[0092] As Figure 9 Further, process 900 can include displaying an image for guiding a procedure using the intraoperative imaging data and the preoperative imaging data (operation 960). For example, the registration system 110 can display an image for guiding a procedure using the intraoperative imaging data and the preoperative imaging data. For example, the image can be an image in which the intraoperative imaging data is fused with the preoperative imaging data. For another example, the image can be an image in which a virtual representation of an interventional device (e.g., a needle, a catheter, a stent, etc.) is displayed in relation to the intraoperative imaging data and / or the preoperative imaging data.

[0093] Figure 10 is a diagram of an example process for displaying an image for guiding a procedure using intraoperative imaging data using a final transformation matrix generated byFigure 1 The system generates an initial transformation matrix and a final transformation matrix to register the preoperative imaging data. For example, as shown, the registration system 110 can receive preoperative imaging data 1010 from the preoperative imaging system 170 and intraoperative imaging data 1020 from the intraoperative imaging system 130. The registration system 110 can generate and display images 1030 for guiding the procedure using the preoperative imaging data and the intraoperative imaging data. For example, the images 1030 can be images that include a fusion of intraoperative ultrasound data and preoperative CT data. However, it should be understood that any combination of data of different imaging modalities can be utilized to generate the images 1030.

[0094] Figure 11 is a diagram of training, deploying, and updating an artificial intelligence (AI) model to generate an initial transformation matrix. The registration system 110 can generate, store, train, and / or use the AI model 120. According to one embodiment, the registration system 110 can include the AI model 120 and / or instructions associated with the AI model 120. For example, the registration system 110 can include instructions for generating the AI model 120, training the AI model 120, using the AI model 120, etc. According to one embodiment, the AI model 120 can be generated and / or trained using a system or device other than the registration system 110. For example, the system or device can include instructions for generating the AI model 120 and / or instructions for training the AI model 120. The system or device can provide the resulting trained AI model 120 to the registration system 110 for use.

[0095] As shown in Figure 11 According to one embodiment, the process 1100 can include a training phase 1102, a deployment phase 1108, and a monitoring phase 1114. In the training phase 1102, at operation 1106, the process 1000 can include receiving and processing training data 1104 to generate a trained AI model 120. The training data 1104 can be generated, received, or otherwise obtained from internal and / or external resources.

[0096] Generally, the AI model 120 can include a set of variables (e.g., nodes, neurons, filters, etc.) that are tuned (e.g., weighted, biased, etc.) to different values via application of the training data 1104. According to one embodiment, the training process at operation 1106 can employ supervised, unsupervised, semi-supervised, and / or reinforcement learning processes to train the AI model 120. According to one embodiment, a portion of the training data 1104 can be reserved during training and / or used to validate the trained AI model 120.

[0097] For supervised learning processes, the training data 1104 can include labels or scores that can facilitate the training process by providing ground truth. For example, the labels or scores can indicate an output of the AI model 120. The training can continue by inputting a training dataset including the training data 1104 into the AI model 120. The AI model 120 can have variables that are set to initialization values (e.g., randomly, based on Gaussian noise, based on pre-trained values, etc.). The AI model 120 can generate an output based on the training dataset being input into the AI model 120. The output can be compared to a corresponding label or score (e.g., ground truth) indicating a known output, which can then be backpropagated through the AI model 120 to adjust the values of the variables. This process can be repeated for multiple samples at least until a determined loss or error is below a predefined threshold. According to one embodiment, some of the training data 1104 can be reserved and used to further validate or test the trained AI model 120.

[0098] For unsupervised learning processes, the training data 1104 can not include pre-assigned labels or scores to help the learning process. Instead, unsupervised learning processes can include clustering, classification, etc., to identify naturally occurring patterns in the training data 1104. For example, the training data can be clustered into groups based on identified similarities and / or patterns. K-means clustering or K-nearest neighbors can also be used, which can be supervised or unsupervised. A combination of K-nearest neighbors and unsupervised clustering techniques can also be used. For semi-supervised learning, a combination of training data 1104 with pre-assigned labels or scores and training data 1104 without pre-assigned labels or scores can be used to train the AI model 120.

[0099] When reinforcement learning is employed, an agent (e.g., an algorithm) can be trained to make decisions from the training data 1104 through trial and error. For example, based on making a decision, the agent can then receive feedback (e.g., a positive reward in the case of a prediction above a predetermined threshold), adjust its next decision to maximize the reward, and repeat until a loss function is optimized.

[0100] After being trained, the trained AI model 120 can be stored and subsequently applied by the registration system 110 during a deployment phase 1108. For example, during the deployment phase 1108, the trained AI model 120 executed by the registration system 110 can receive input data 1110. During the deployment phase 1108, the trained AI model 120 can perform one or more operations as described in connection with Figure 5 and Figure 8 .

[0101] After deployment, the trained AI model 120 can be monitored during a monitoring phase 1114. For example, during the monitoring phase 1114, the AI model 120 can generate monitoring data 1116 for monitoring the trained AI model 120. The monitoring data 1116 can include data identifying outputs as determined by an operator. During a process 1118, the monitoring data 1116 can be analyzed along with the predicted output data 1112 and the input data 1110 to determine the accuracy of the trained AI model 120. Based on the analysis, the process 1100 can return to the training phase 1102, where at operation 1106, the values of one or more variables of the model can be adjusted to improve the accuracy of the AI model 120, according to one embodiment.

[0102] The example process 1100 described above is provided by way of example only, and can include additional, fewer, different, or differently arranged aspects than those depicted in Figure 11 the aspects described above in connection with the trained AI model 120.

[0103] Figure 11 Training, deployment, and monitoring associated with the trained AI model 120 is described. According to one embodiment, one or more other trained AI models 120s can be applied. Each of the trained AI models 120s can include similar training, deployment, and / or monitoring phases as described above for the trained AI model 120 in Figure 11 the aspects described above in connection with the trained AI model 120.

[0104] The embodiments of the disclosure illustrated in the figures and described above are exemplary embodiments only, and are not intended to limit the scope of the claims of this application, including any equivalents of the claims. Various modifications are possible and will be apparent to those skilled in the art. Any combination of the features described herein, which are not mutually exclusive, is intended to fall within the scope of the present application. That is, features of the described embodiments can be combined with any appropriate aspects described above, and optional features of any one aspect can be combined with any other appropriate aspect. Similarly, features recited in dependent claims can be combined with features of other dependent claims, particularly where the dependent claims are from the same independent claim. In jurisdictions where single claims can be used to define a plurality of combinations of the features of the claimed application, it is intended that the dependent claims can be used to define any of the combinations of the features of the claimed application.

Claims

1. A method (500), the method comprising: receiving (510) position data of a tracked instrument moving along a trajectory corresponding to a predefined shape relative to a patient; determining (520) a first set of coordinates of the position data corresponding to the predefined shape; receiving (530) preoperative imaging data of the patient; determining (540) a second set of coordinates of the preoperative imaging data corresponding to the predefined shape; generating (550) an initial transformation matrix based on the first set of coordinates and the second set of coordinates; and registering (950) intraoperative imaging data with the preoperative imaging data based on the initial transformation matrix.

2. The method (500) of claim 1, wherein determining the second set of coordinates comprises determining the second set of coordinates using metadata of the preoperative imaging data.

3. The method (500) of claim 2, wherein the metadata comprises at least one of a patient position attribute, an image orientation attribute, an image position attribute, and a pixel spacing attribute.

4. The method (500) of claim 1, wherein the predefined shape is an L-shape.

5. The method (500) of claim 1, wherein the first set of coordinates comprises a first coordinate and a second coordinate corresponding to superior-inferior movement of the tracked instrument relative to the patient, and the second coordinate and a third coordinate corresponding to lateral movement of the tracked instrument relative to the patient.

6. The method (500) of claim 1, the method further comprising: generating a final transformation matrix based on the initial transformation matrix, wherein registering the intraoperative imaging data with the preoperative imaging data comprises registering the intraoperative imaging data with the preoperative imaging data using the final transformation matrix.

7. The method (500) of claim 6, the method further comprising: displaying an image for guiding a procedure using the intraoperative imaging data and the preoperative imaging data.

8. An apparatus (110), the apparatus comprising: a memory (230) configured to store instructions; and one or more processors (220) configured to execute the instructions, the instructions for performing operations comprising: receiving (510) position data of a tracked instrument moving along a trajectory corresponding to a predefined shape relative to a patient; determining (520) a first set of coordinates of the position data corresponding to the predefined shape; receiving (530) preoperative imaging data of the patient; determining (540) a second set of coordinates of the preoperative imaging data corresponding to the predefined shape; generating (550) an initial transformation matrix based on the first set of coordinates and the second set of coordinates; and registering (950) intraoperative imaging data with the preoperative imaging data based on the initial transformation matrix.

9. The apparatus (110) of claim 8, wherein determining the second set of coordinates comprises determining the second set of coordinates using metadata of the preoperative imaging data. ​ ​ 10. The device (110) of claim 9, wherein the metadata includes at least one of a patient position attribute, an image orientation attribute, an image position attribute, and a pixel spacing attribute.

11. The device (110) of claim 8, wherein the predefined shape is an L-shape.

12. The device (110) of claim 8, wherein the first set of coordinates includes a first coordinate and a second coordinate corresponding to superior-inferior movement of the tracked instrument relative to the patient, and the second coordinate and a third coordinate corresponding to lateral movement of the tracked instrument relative to the patient.

13. The device (110) of claim 8, wherein the operations further comprise: generating a final transformation matrix based on the initial transformation matrix, wherein registering the intraoperative imaging data with the preoperative imaging data includes using the final transformation matrix to register the intraoperative imaging data with the preoperative imaging data.

14. The device (110) of claim 13, wherein the operations further comprise: displaying an image for guiding a surgery using the intraoperative imaging data and the preoperative imaging data.

15. A non-transitory computer-readable medium (230) storing instructions that, when executed by one or more processors (220), cause the one or more processors (220) to perform operations comprising: receiving (510) position data of a tracked instrument moving along a trajectory corresponding to a predefined shape relative to a patient; determining (520) a first set of coordinates of the position data corresponding to the predefined shape; receiving (530) preoperative imaging data of the patient; determining (540) a second set of coordinates of the preoperative imaging data corresponding to the predefined shape; generating (550) an initial transformation matrix based on the first set of coordinates and the second set of coordinates; and registering (950) intraoperative imaging data with the preoperative imaging data based on the initial transformation matrix. ​

Images