Provides user guidance on techniques for registering patient image data with surgical tracking systems.

Abstract

This invention provides a method comprising: acquiring first patient image data of a patient's body; determining registration instructions indicating where to acquire registration points relative to the surface of the body; acquiring second patient image data of the body already acquired by an augmented reality (AR) device; determining a transformation between the coordinate systems of the first and second patient image data; and, based on the transformation, triggering the display of the registration instructions on the display of the AR device, such that an augmented view is presented to a user of the AR device, wherein the registration instructions are superimposed on the patient's body, and the augmented view guides the user on where to acquire registration points. This invention also discloses a computing system, a surgical navigation system, and a computer program product.

Description

Technical Field

[0001] This invention generally relates to a method for providing user guidance for obtaining transformations between the coordinate system of patient image data and the coordinate system of a surgical tracking system. A computing system, a surgical navigation system, and a computer program product are also provided. Background Technology

[0002] Surgical navigation has been used to support medical personnel for some time. For example, displaying the current pose of surgical instruments relative to patient image data can help surgeons move surgical instruments to the desired anatomical location on the patient.

[0003] As is known in the art, the current orientation of surgical instruments in a tracking coordinate system can be tracked using optical or electromagnetic surgical tracking systems. To determine the current orientation of the surgical instruments relative to patient image data, a transformation is required between the tracking coordinate system and the coordinate system of the patient image data. This transformation is sometimes referred to as registration between the surgical tracking system and the patient image data.

[0004] To achieve the aforementioned registration, surgeons or other medical personnel can use a registration probe, tracked by a surgical tracking system, to acquire one or more registration points on the surface of the patient's body. These acquired registration points can then be compared with patient image data to obtain the registration.

[0005] Certain areas of the body surface are more suitable for acquiring registration points than others. Therefore, it would be beneficial to provide surgeons or other medical personnel with guidance on where to acquire registration points on the patient's body surface to achieve accurate registration between the surgical tracking system and the patient's image data.

[0006] In some cases, it may be advantageous to avoid transformations between the tracking coordinate system used to provide user guidance and the coordinate system of the patient image data. In other cases, guidance may be required before the surgical tracking system is activated or made available. Summary of the Invention

[0007] A technique is needed to solve one or more of the above-mentioned or other problems.

[0008] According to a first aspect, a method is provided for providing user guidance to obtain a transformation between a coordinate system of patient image data and a coordinate system of a surgical tracking system. The method includes: acquiring first patient image data of at least a portion of a patient's body; determining at least one registration instruction based on the first patient image data, the at least one registration instruction indicating where to acquire at least one registration point relative to a surface of the patient's body; acquiring second patient image data of at least a portion of the patient's body, the second patient image data having been acquired by an augmented reality (AR) device; determining a first transformation between a first coordinate system of the first patient image data and a second coordinate system of the second patient image data; and triggering, based on the at least one registration instruction and the first transformation, the display of the at least one registration instruction on a display of the AR device, such that an augmented view is presented to a user of the AR device, wherein the at least one registration instruction is superimposed on at least a portion of the patient's body, and the augmented view guides the user on where to acquire at least one registration point.

[0009] The first patient image data can be, for example, medical image data already acquired by a medical imaging device (e.g., preoperative or intraoperative). The first patient image data can include computed tomography (CT) data, magnetic resonance (MR) data, ultrasound image data, etc. A portion of the patient's body can include a surface of the body. This portion can include at least a portion of the patient's face.

[0010] A specific example for determining the first transformation will now be described. The first transformation can be determined based on first patient image data and second patient image data. The first transformation can be determined by matching the first patient image data with the second patient image data, and vice versa. The first transformation can be determined by correlating the first patient image data with the second patient image data. The first transformation can be determined by matching a portion of the patient's body represented by the first patient image data with a corresponding (e.g., identical) portion of the patient's body represented by the second patient image data. The first transformation can be determined by comparing the positions of similar landmarks in a first coordinate system and a second coordinate system. This specific example for determining the first transformation may not require an AR device to be tracked by the tracking system.

[0011] At least one registration instruction may include an indication of a sub-part of the surface of at least a portion of the patient’s body, in which at least one registration point will be acquired.

[0012] A subpart can be a portion, region, or segment of the surface of at least one part of the patient's body. Indication of a subpart can include a visualization of at least one of the subpart's outline and content.

[0013] The method may include: receiving information indicating at least one acquired registration point; and processing the information indicating at least one acquired registration point to obtain a second transformation between a first coordinate system of the first patient image data and a third coordinate system of the surgical tracking system.

[0014] Information indicating at least one acquired registration point may include an indication of the position of at least one acquired registration point in a third coordinate system.

[0015] In one example, information indicating at least one registration point has been acquired by a surgical tracking system that tracks the registration probe.

[0016] Processing information indicating at least one acquired registration point may include matching the location of at least one acquired registration point (e.g., in a third coordinate system) with first patient image data (e.g., in a first coordinate system).

[0017] The location of at least one acquired registration point can be matched to a body surface described by the first patient image data. The body surface can be described by or extracted from the first patient image data, for example, as a 3D point cloud, surface, or shape. The location of at least one registration point in the third coordinate system can be matched to a surface in the first coordinate system.

[0018] The method further includes, for example, acquiring information indicating at least one acquired registration point by obtaining at least one position of the registration probe in a third coordinate system; and determining a second transformation by matching the information indicating at least one acquired registration point to first patient image data.

[0019] The method may further include: obtaining tracking data describing the posture (e.g., at least one of position and orientation) of a surgical instrument in a third coordinate system; transforming the posture of the surgical instrument to a second coordinate system based on a first transformation and a second transformation; determining at least one navigation command associated with the posture of the surgical instrument in the second coordinate system; and triggering the display of the at least one navigation command on a display of an AR device.

[0020] At least one navigation instruction may include an indication of at least one of the type of surgical instrument, the position of the surgical instrument in the second coordinate system, and the orientation of the surgical instrument in the second coordinate system.

[0021] The patient's body portion may include one or more parts of the patient's face, and determining the first transformation may include: matching a generic facial model to at least one of first patient image data and second patient image data. In this case, determining the first transformation may include: determining an initially deformed facial model by matching the generic facial model to one of the first and second patient image data; and comparing the initially deformed facial model to the other of the first and second patient image data to determine the first transformation. The generic facial model may be matched to one of the first and second patient image data such that each of a plurality of predefined landmarks of the generic facial model is located on a corresponding landmark of the patient's body portion in one of the first and second patient image data.

[0022] Comparing the initially deformed facial model with another of the first and second patient image data may include comparing the location of a landmark identified by the deformed facial model with the location of a corresponding landmark in the other of the first and second patient image data.

[0023] In one example, the corresponding landmark is the patient's physical biometrics.

[0024] Biometric features can be at least one of distinctive anatomical features, patient-specific features, and surface features of the patient's body.

[0025] The generic facial model can be a three-dimensional deformable model (3D-MM). The generic facial model can be matched to the surface of the patient's body (e.g., the surface described by at least one of the first patient image data and the second patient image data).

[0026] In the first variant, comparing the initially deformed facial model with another of the first patient image data and the second patient image data includes: determining a secondary deformed facial model by matching a generic facial model or the initially deformed facial model to another of the first patient image data and the second patient image data; and comparing the initially deformed facial model (e.g., the location of a predefined landmark) with the secondary deformed facial model (e.g., the location corresponding to a predefined landmark).

[0027] In a second variant, comparing the initially deformed facial model with another of the first and second patient image data includes: performing image analysis on the other of the first and second patient image data to determine the location of at least one corresponding landmark in the other of the first and second patient image data; and comparing the determined location of at least one of the corresponding landmarks in the other of the first and second patient image data with the location of one of a plurality of predefined landmarks of the initially deformed facial model (e.g., a matching facial model) located on the same corresponding landmark in the first and second patient image data.

[0028] The method may include: determining whether the result of a comparison between (i) an initially deformed facial model and (ii) a second deformed facial model or another of first patient image data and second patient image data meets one or more predefined acceptance criteria; and if the result meets one or more predefined acceptance criteria, triggering the display of the at least one registration instruction.

[0029] One or more predefined acceptance criteria may define at least one of the following: maximum (e.g., point-by-point or average) spatial deviation, maximum (e.g., point-by-point or average) translational deviation, and maximum (e.g., point-by-point or average) rotational deviation.

[0030] In one example, the method includes instructing the surgical tracking system to begin tracking after at least one registration instruction has been triggered for display.

[0031] The method may include instructing the surgical tracking system to begin tracking in response to the display of at least one registration instruction being triggered. The tracking system may be inactive or unavailable before the at least one registration instruction to be displayed is triggered.

[0032] The second patient image data may indicate (e.g., include, describe, or constitute) a two-dimensional image captured by a camera of an AR device. The second patient image data may have been captured by a sensor of the AR device (e.g., a camera of the AR device, such as a 2D camera, a time-of-flight (TOF) camera, or a stereo camera). The second patient image data may include, describe, or contain at least one of two-dimensional image data, depth data, and three-dimensional image data.

[0033] According to a second aspect, a computing system is provided. The computing system includes at least one memory and at least one processor, the at least one memory storing instructions that, when executed on the at least one processor, cause the at least one processor to perform the method according to the first aspect.

[0034] According to a third aspect, a surgical navigation system is provided. The surgical navigation system includes a computing system as described in the second aspect and at least one component selected from an AR device (optionally configured as a head-mounted display (HMD) and a surgical tracking system). The surgical navigation system can be configured such that at least one object selected from the AR device and the patient's body cannot be tracked by the surgical tracking system.

[0035] According to a fourth aspect, a computer program product is provided. The computer program product includes a program code portion for performing the method according to the first aspect when the computer program product is executed on at least one processor (e.g., at least one processor of the computing system according to the second aspect). The computer program product may be stored on one or more computer-readable recording media, such as at least one memory of the computing system according to the second aspect. Attached Figure Description

[0036] Further details, advantages, and aspects of the invention will become apparent from the following embodiments, taken in conjunction with the accompanying drawings, in which:

[0037] Figure 1 An exemplary embodiment of the surgical navigation system according to the present invention is shown;

[0038] Figure 2 An exemplary embodiment of the method according to the present invention is shown;

[0039] Figure 3 A schematic diagram of the facial model in its initial deformation according to the present invention is shown; and

[0040] Figure 4 A schematic example of an enhanced view according to the present invention is shown. Detailed Implementation

[0041] In the following description, exemplary embodiments of a method and surgical navigation system will be explained with reference to the accompanying drawings. The same reference numerals will be used to denote the same or similar structural features.

[0042] Figure 1 An exemplary embodiment of a surgical navigation system 100 according to the present invention is shown. The surgical navigation system 100 includes an augmented reality (AR) device 2, a surgical tracking system 4, and a computing system 6.

[0043] The augmented reality device 2 includes a camera 12 configured to acquire images with a second coordinate system 14. The camera 12 can be configured to acquire two-dimensional images, depth images, or three-dimensional images.

[0044] Augmented reality device 2 also includes a display 15. Display 15 may be a heads-up display, such as a screen that is at least partially transparent as a projection surface for displaying visualizations overlaid on objects visible through the screen. Alternatively, display 15 may include a screen configured to display an image of a patient's body acquired by camera 12, the image including visualizations overlaid on the image of the patient's body. Display 15 may have a known, preferably fixed, pose (i.e., at least one of position and orientation) relative to camera 12. This allows for the display of objects on display 15, where the pose of the objects is given in a second coordinate system. That is, the display of information defined in or associated with second coordinate system 14 (e.g., the pose of the instrument and / or information overlaid on the patient's body) may include transforming the information from second coordinate system 14 to a coordinate system of display 15 using the known pose of display 15 in second coordinate system 14. In one example, second coordinate system 14 may correspond to the coordinate system of display 15.

[0045] For example, display 15 and camera 12 are part of the same (e.g., portable) unit. Augmented reality device 2 can be configured as, for example, a head-mounted display (HMD) in the form of glasses. Examples of such HMDs include Microsoft's HoloLens and HoloLens 2.

[0046] The surgical tracking system 4 in the example shown is an electromagnetic tracking system that includes an electromagnetic field generator 16. Alternatively, the surgical tracking system 4 can be an optical tracking system using a camera or stereo camera for tracking the object.

[0047] The surgical tracking system 4 is configured to track surgical instruments 20 in a third coordinate system 18. Examples of surgical instruments 20 include registration probes, indicators, surgical drill guides, surgical drills, surgical chisels, biopsy needles, deep brain stimulation (DBS) electrodes, etc., which can be used to acquire registration points. In the illustrated example, the surgical instrument 20 has an elongated tool axis 22 defining a longitudinal tool axis 24 and a handle 26 for the user to grip. A tracker 28 (e.g., using biocompatible adhesive) is removably attached to a portion of the patient's body 29. The surgical tracking system 4 also includes a locator 30 configured to determine the current orientation of the tracker 28 in the third coordinate system 18. The locator 30 is further configured to determine the current orientation of the surgical instrument 20 in the third coordinate system 18. The augmented reality device 2 in the illustrated example cannot be tracked by the surgical tracking system 4. This is because the augmented reality device is not equipped with an electromagnetic field sensor that is communicatively coupled to the locator 30. In the case of using an optical tracking system, the augmented reality device 2 may or may not include an optical tracking mark (tracker) that can be tracked by the optical tracking system.

[0048] The computing system 6 includes a memory 32, a processor 34, and a communication interface 36. The memory 32 stores instructions that, when executed on the processor 34, cause the processor 34 to perform the methods disclosed herein. The augmented reality device 2 may be communicatively connected to the communication interface 36 of the computing system 6. Alternatively, the computing system 6 may be part of the AR device 2. The communication interface 36 is communicatively connected to the surgical navigation system 4 (e.g., its locator 30). Note that the functionality of the locator 30 may be implemented by the processor 34. That is, the locator 30 may be part of the computing system 6. In an exemplary variant, the computing system 6 is part of the surgical tracking system 4.

[0049] Figure 1 The document also indicates first patient image data 37 with a first coordinate system 39. Details on how the first patient image data 37 can be used will be referenced below. Figure 2 Describe it.

[0050] Figure 2 An exemplary embodiment of the method according to the present invention is shown. The method can be executed by the processor 34 of the computing system 6. The method can be a computer-implemented data processing method. The method does not require any substantial physical interaction with the patient's body 29. In other words, the method does not include surgical steps.

[0051] In step 200, first patient image data 37 of at least a portion of the patient's body 29 is obtained. The portion of the patient's body 29 may include the skin surface of the patient's body 29, such as the skin surface visible to the surgeon during a planned surgery. The portion of the patient's body 29 may include at least a portion of the patient's head, such as the patient's face. The first patient image data 37 may include at least one of two-dimensional medical image data and three-dimensional medical image data. The first patient image data 37 may include one or more X-ray images of that portion of the patient's body 29. The first patient image data 37 may include a computed tomography (CT) image of that portion of the patient's body 29. The first patient image data 37 may include a magnetic resonance (MR) image of that portion of the patient's body. The first patient image data 37 may be preoperative image data. The first patient image data 37 may be obtained from memory 32 or received via interface 36, for example, from a Picture Archiving and Communication System (PACS). Note that the acquisition of the first patient image data 37 may be performed in advance by a medical imaging device and may not be part of the method described herein. The first patient image data 37 obtained may have been acquired by a medical imaging device, which may be connected to a PACS and / or a computing system 6.

[0052] In step 202, at least one registration instruction is determined based on the first patient image data 37. The at least one registration instruction may be determined automatically, for example, based solely on the first patient image data 37. The at least one registration instruction may be determined without any user input. The at least one registration instruction indicates where at least one registration point will be acquired relative to the surface of the patient's body 29. The at least one registration instruction may include an indication of a sub-region of the surface of at least a portion of the patient's body 29, in which at least one registration point will be acquired.

[0053] Sub-parts can be determined by performing image analysis on the first patient image data 37. Sub-parts can be determined by identifying sub-parts of the surface of the patient's body described by the first patient image data 37, which has one or more predefined attributes. Predefined attributes may include at least one of the following: low mechanical deformability, indicated, for example, by a distance between the surface of the patient's body 29 and the underlying bones being less than a predefined maximum distance; surface curvature exceeding a minimum curvature; surface curvature opposite to the surface curvature of a predefined surface region; surface curvature deviating from the surface curvature of a predefined region (e.g., a region adjacent to or surrounding the sub-part) by more than a predefined amount; surface curvature opposite to or deviating from the surface curvature of the remaining surface of the patient's body 29 (e.g., a portion thereof) by more than a predefined amount; surface curvature deviating from the surface curvature at the location of at least one acquired registration point by more than a predefined amount; and spatial distance to at least one acquired registration point exceeding a predefined amount. Other criteria for determining at least one registration instruction are possible. For example, at least one registration instruction can be determined by matching a predefined mask to the first patient image data 37, the matched predefined mask defining a sub-part of the surface of at least a portion of the patient's body 29. At least one registration instruction can be determined according to the description in European Patent Application No. 20198095.0, filed on 24 September 2020, the entire contents of which are incorporated herein by reference. Referring to claim 1 of the filed European Patent Application, the image data may correspond to the first patient image data 37 described herein. Referring again to claim 1 of the filed European Patent Application, the visualization of priority values ​​of at least one surface point or the visualization of information derived from priority values ​​may correspond to at least one registration instruction described herein.

[0054] In step 204, second patient image data of at least a portion of the patient's body 29 is obtained, which has been acquired by the augmented reality (AR) device 2, such as camera 12. The second patient image data may be two-dimensional or three-dimensional image data indicating (e.g., describing, including, or constituting) at least a portion of the patient's body 29. The second patient image data may be acquired after or before steps 200 and / or 202. In one example, camera 12 acquires video images, and AR device 2 or computing system 6 determines whether a face is depicted in one of the video images by performing image analysis on the video images. Images in which the presence of a face is detected in this way can then be used as second patient image data. The second patient image data may be received by processor 34, for example, via communication interface 36.

[0055] In step 206, a first transformation is determined between the first coordinate system 39 of the first patient image data 37 and the second coordinate system 14 of the second patient image data. As described above, the portion of the patient's body 29 may include one or more portions of the patient's face. In this case, determining the first transformation may include matching a generic facial model with at least one of the first and second patient image data. In an advantageous example, the generic facial model is matched with the second patient image data already acquired by the camera 12 of the AR device 2. The generic facial model may be a deformable facial model, particularly a three-dimensional deformable facial model. The generic facial model may include a mesh, for example, composed of nodes connected by edges. Some or all nodes may be designated with or associated with predefined landmarks.

[0056] Determining the first transformation may include: determining an initially deformed facial model by matching a generic facial model with one of the first patient image data 37 and the second patient image data; and comparing the initially deformed facial model with the other of the first patient image data 37 and the second patient image data to determine the first transformation. In an advantageous example, the generic facial model is matched with the second patient image data to determine the initially deformed facial model, and then the initially deformed facial model is compared with the first patient image data 37.

[0057] A generic facial model can be matched with one of the first patient image data 37 and the second patient image data such that each of the plurality of predefined landmarks of the generic facial model is located on the corresponding landmark of the portion of the patient's body 29 in one of the first patient image data 37 and the second patient image data. In an advantageous example, the generic facial model is matched with the second patient image data such that each of the plurality of predefined landmarks of the generic facial model, such as each of the nodes, is located on the corresponding landmark of the portion of the patient's body 29 in the second patient image data already acquired by camera 12.

[0058] Matching a generic facial model with one of the first patient image data 37 and the second patient image data may include matching the generic facial model with a surface of a portion of the patient's body described by one of the first and second patient image data. This surface may be defined by a three-dimensional point cloud comprising the three-dimensional coordinates of points in a point cloud, or by an algebraic or numerical definition of a three-dimensional surface. Matching the generic facial model with one of the first and second patient image data may include extracting a surface from one of the first and second patient image data. For example, the three-dimensional point cloud may include at least one point for each corresponding landmark. In one example, the generic facial model may be matched with one or more two-dimensional images contained in the second patient image data (e.g., with a first two-dimensional image acquired by a first image acquisition unit and a second two-dimensional image acquired by a second image acquisition unit, wherein the first and second image acquisition units are contained in camera 12. A comparison of each facial model matched with the two-dimensional images may be performed to determine the facial model of the initial deformation.

[0059] A generic facial model can define deformation properties that restrict the deformation of at least a portion of the generic facial model, such as relative to other parts of the generic facial model. These deformation properties ensure that the deformed facial model still represents the true form of a face (e.g., even if the generic facial model matches a two-dimensional image). Matching the generic facial model to one of the first patient image data 37 and the second patient image data can include aligning the generic facial model to one of the first patient image data 37 and the second patient image data (e.g., the surface of a body part of the patient as described). Alignment can include using an alignment optimization algorithm. After alignment, the aligned generic facial model can be deformed to minimize the deviation between the aligned generic facial model and one of the first patient image data 37 and the second patient image. For the alignment and deformation of the generic facial model, the positions of predefined landmarks of the generic facial model can be adjusted to correspond to the positions of corresponding landmarks on the surface of the patient's body described or depicted in one of the first patient image data 37 and the second patient image. The corresponding landmarks can be at least one of anatomical landmarks of the patient's body 29, characteristic surface features of the patient's body 29, and biometric features of the patient's body 29. Biometric features can be patient-specific features. Therefore, the relative positions between biometric features may vary from patient to patient.

[0060] Figure 3A schematic diagram of the initial deformed facial model 38 according to the present invention is shown. In this example, both the general facial model and the initial deformed facial model 38 consist of ten nodes and eleven edges 40. A first node 42 is associated with a node tip, a second node 44 with the left outer corner of the eye, a third node 46 with the right outer corner of the eye, a fourth node 48 with the left inner corner of the eye, and a fifth node 50 with the right inner corner of the eye. Each node 42-50 corresponds to a predefined landmark of the facial model and is matched with a corresponding biometric feature of the second patient image data 52. That is, the biometric feature can be the outer corner of the eye, the inner corner of the eye, the tip of the nose, etc. Figure 3 The image also shows a predefined plane of symmetry 54 for the matched facial model 38, which can be additionally matched with the second patient image data 52 to determine or improve the initially deformed facial model 38. Clearly, Figure 3 For illustrative purposes, and may use other shapes and types of a general facial model or a first-deformation facial model 38.

[0061] Continuing with step 206, in the first variant, comparing the initially deformed facial model with another of the first patient image data 37 and the second patient image data 52 includes determining the second deformed facial model. The second deformed facial model is obtained by matching the generic facial model or the initially deformed facial model 38 with another of the first patient image data 37 and the second patient image data 52, and comparing the initially deformed facial model 38 with the second deformed facial model. Matching the generic facial model or the initially deformed facial model 38 with another of the first patient image data 37 and the second patient image data 52 can involve, similar to the first matching operation described above, extracting surfaces from the other of the first patient image data 37 and the second patient image data 52, and aligning and deforming the generic facial model or the initially deformed facial model 38 to represent the shape of at least a portion of the three-dimensional surface. Note that... Figure 3 The schematic diagrams and descriptions shown can be similarly applied to facial models with secondary deformations as described herein.

[0062] In a second variant, comparing the initially deformed facial model 38 with another of the first patient image data 37 and the second patient image data 52 includes performing image analysis on the other of the first patient image data 37 and the second patient image data 52 to determine the position of at least one of the corresponding landmarks of the portion of the patient's body 29 in the other of the first patient image data 37 and the second patient image data 52. In this case, the method may further include comparing the determined position of at least one of the corresponding landmarks in the other of the first patient image data 37 and the second patient image data 52 with the position of one of a plurality of predefined landmarks of a generic facial model or the initially deformed facial model 38 located on the same corresponding landmark in one of the first patient image data 37 and the second patient image data.

[0063] In an advantageous example, image analysis is performed on first patient image data 37, which may include preoperative CT image data of the patient's body 29, to determine the location of at least one of the corresponding landmarks in the first patient image data 37. This may involve extracting the surface of the body from the first patient image data 37 and analyzing the shape of the extracted body surface to identify the location of the corresponding landmark, such as the location of the tip of the nose, the right outer corner of the eye, the left inner corner of the eye, etc. The location identified in the first patient image data 37 can then be compared with the location of a predefined landmark of a deformed facial model 38 that matches the same landmark in the second patient image data 52. The location of the right outer corner of the eye can be identified in the first patient image 37 and compared with the location of node 46 of the initially deformed facial model 38 obtained by matching a generic facial model with the second patient image data 52. The location of the left inner corner of the eye can be identified in the first patient image 37 and compared with the location of node 48 of the initially deformed facial model 38. This comparison can be performed on some or all of the predefined landmarks or nodes. In other words, multiple locations of landmarks in the first patient image data 37 can be compared with multiple locations of similar landmarks defined by the initially deformed facial model 38 in the second patient image data 52. In other words, the locations of similar landmarks in the first coordinate system 39 and the second coordinate system 14 can be compared with each other.

[0064] Comparison of (multiple) locations can be used to determine the first transformation. The locations of landmarks in the first patient image data 37 are given in the first coordinate system 39, while the locations of predefined landmarks of the generic facial model, after matching corresponding landmarks in the second patient image data 52, are given in the second coordinate system 14. Comparing these two locations yields a transformation between the first coordinate system 39 and the second coordinate system 14. Generally, the more locations in the two coordinate systems 39, 14 are compared, the more accurate the first transformation is obtained. For example, comparing at least three locations may yield a first transformation that may include translation and rotation components. A point-to-point matching algorithm can be used to compare locations, where the result of point-to-point matching can be the first transformation. An optimization algorithm can be used to determine the best match between locations to determine the first transformation. The optimization algorithm may be or include the Iterative Closest Point (ICP) algorithm.

[0065] Refer again Figure 2 In step 208, based on at least one registration instruction and a first transformation, at least one registration instruction is triggered to be displayed on the display 15 of the AR device 2, thereby presenting an enhanced view to the user of the AR device 2, wherein at least one registration instruction is superimposed on at least a portion of the patient's body 29. The enhanced view guides the user on where to acquire at least one registration point on the surface of the patient's body 29.

[0066] As described above, the spatial relationship between camera 12 and display 15 can be fixed and / or known. In other words, the transformation between the second coordinate system and the coordinate system of display 15 can be known. This makes it possible to display these objects on display 15 using the pose described in the second coordinate system. Obviously, this method is applicable to any object, text, or instruction to be displayed. The pose of the object or instruction can be transformed from the second coordinate system 14 to the coordinate system of display 15 by augmented reality device 2 and / or processor 34. For example, step 208 may include determining the pose (i.e., at least one of position and orientation) of the registration instruction in the coordinate system of display 15 by transforming the registration instruction (e.g., pose, as described in the second coordinate system) to the coordinate system of display 15 using a known transformation between the second coordinate system and the coordinate system of display 15.

[0067] Figure 4A schematic example of an augmented view according to the invention is shown. Where the display 15 includes a screen that is at least partially transparent, through which the user of the AR device 2 can see a portion of the patient's body 29, at least one registration instruction can be displayed on the screen. In this case, or if the display 15 is a heads-up display or another type of semi-transparent or holographic display, a portion of the patient's body 29 can be seen through the display 15 of the augmented reality device 2. Alternatively, if the display 15 includes a display screen through which the user of the AR device 2 cannot see a portion of the patient's body 29, at least one registration instruction superimposed on the image of the patient's body 29 captured by the camera 12 (e.g., an image included in the second patient image data 52) can be displayed on the screen. Live video of the patient's body 29 captured by the camera 12 can be displayed on the display 15 to provide an augmented view to the user of the AR device 2.

[0068] The augmented view may include at least one registration instruction superimposed on at least a portion of the patient's body 29, even if the relative position between the camera 12 and the patient's body 29 has changed between the time the second patient image data was acquired and the time the augmented view was displayed. The AR device 2 or computing system 6 may (e.g., continuously or periodically) update the displayed augmented view based on the current pose of the AR device 2 relative to the patient's body 29. For example, steps 204-208 may be performed periodically. That is, the second patient image data 52 may be acquired again, a first transformation may be determined in step 206 based on the most recently acquired second patient image data 52, and the transformation used in step 208 may be the most recently determined first transformation. Alternatively or additionally, the method may include determining a first pose of the AR device 2 (e.g., camera 12) in a world coordinate system when the AR device 2 has acquired the second patient image data. The method may further include (e.g., by locating a portion of the patient's body 29 in a world coordinate system) determining the pose of the patient's body 29 in a world coordinate system. This may involve identifying the locations of landmarks in the second patient image data 52 and, if the two coordinate systems are not equal, transforming the location from the second coordinate system 14 to the world coordinate system using the first pose of the AR device 2. The method may also include determining the second pose of the AR device 2 in the world coordinate system when the display of at least one registration instruction is triggered again (i.e., after the first execution of step 208). The augmented view can then be determined based on the at least one registration instruction, the first transformation, and the result of a comparison between the first and second poses. Specifically, translational and / or rotational shifts of the AR device 2 relative to the patient's body 29 can be compensated for after multiple time points following the acquisition of the second patient image data 52, such that the augmented view always provides the user of the AR device 2 with at least one registration instruction superimposed on the portion of the patient's body 29. In this case, updating the first registration is unnecessary; only updating the second pose of the AR device 2 in the world coordinate system is required. The pose of the AR device 2 in the world coordinate system can be determined using data from one or more sensors included in the AR device 2, such as image data from camera 12 (e.g., the second patient), data from a depth sensor, a time-of-flight camera, an accelerometer, a gyroscope, etc.

[0069] In the illustrated example, the patient's body 29 has a first surface sub-partition 56 on which no registration instructions are displayed. At least one registration instruction includes indications for sub-partitions 58 and 60. Each of the sub-partitions 58 and 60 of the patient surface can be highlighted in a different manner. For example, the indications for sub-partitions 58 and 60 can differ from each other in at least one of color, opacity, or optical pattern.

[0070] Sub-part 56 on the right eyelid of the patient's body 29 is not highlighted. This is likely because acquiring at least one registration point on the right eyelid is disadvantageous, as this body part is highly deformable. For example, when acquiring registration points on such a highly deformable body part, the location of the acquired registration point depends largely on the pressure the registration probe is pressed against the body surface. Sub-parts 58 are more advantageous for acquiring at least one registration point because they mark areas with lower mechanical deformability, represented by a smaller distance between the body surface and the patient's bones beneath it. Sub-parts 60 are the most preferred areas for acquiring at least one registration point because they have a higher surface curvature than sub-parts 60. For details on how sub-parts 56, 58, and 60 can be determined, see the description of step 202 above.

[0071] The display of at least one registration instruction can be conditionally triggered. For example, the method may include determining whether a comparison between (i) the initially deformed facial model 38 and (ii) the second-deformed facial model or another of the first patient image data 37 and the second patient image data 52 meets one or more predefined acceptance criteria. The method may include (e.g., only) triggering the display of at least one registration instruction if the result meets one or more predefined acceptance criteria. For example, one or more predefined acceptance criteria include the maximum deviation of the positions of one or more predefined landmarks or the maximum average deviation of the positions of a set or all predefined landmarks. This deviation may be a translational deviation in one or more predefined spatial directions, a rotational deviation about one or more predefined axes, or a combination thereof. This approach ensures that at least one registration instruction is displayed only if the accuracy of the first transformation is acceptable, thereby avoiding misaligned display of at least one registration instruction.

[0072] For example, a similarity is determined between (i) the initially deformed facial model 38 and (ii) the second deformed facial model or the other of the first patient image data 37 and the second patient image data 52, and it is determined whether the similarity meets one or more predefined acceptance criteria. The method may include, for example, triggering the display of at least one registration instruction only if the similarity meets one or more predefined acceptance criteria. The similarity may be a difference or quotient between at least one geometrical property (e.g., a surface described or extracted from) of the initially deformed facial model 38 and the second deformed facial model or the other of the first patient image data 37 and the second patient image data 52. For example, one or more predefined acceptance criteria may define that the distance between matching node 46 and matching node 44 divided by the distance between matching node 40 and matching node 48 must be similar between the initially deformed facial model 38 and the second deformed facial model and within a predefined tolerance range, such as 5%, 10%, or 12%. The predefined acceptance criteria may define that the initially deformed facial model 38 must match the second deformed facial model to the same patient. At least one registration instruction can be triggered only if the same patient is depicted in the first patient image data 37 and the second patient image data 52. A biometric recognition algorithm, such as at least one of a first-deformation facial model 38 and a second-deformation facial model, can be used to correctly identify the patient based on the first patient image data 37 and the second patient image data 52, respectively.

[0073] When at least one registration instruction is triggered, the surgical tracking system 4 may not track the patient's body 29. For example, the patient's body 29 may not be tracked by the surgical tracking system 4 or may be deactivated before the display of at least one registration instruction has been triggered. The methods disclosed herein may include instructing the surgical tracking system 4 to begin tracking after the display of at least one registration instruction has been triggered, for example, in response to the display of at least one registration instruction being triggered.

[0074] The method may further include obtaining or determining a second transformation between a first coordinate system 39 of the first patient image data 37 and a third coordinate system 18 of the surgical tracking system 4. The chain of the first and second transformations can be said to connect the second coordinate system 14 of the AR device 2 and the third coordinate system 18 of the surgical tracking system 4 via the first coordinate system 39 of the first patient image data 37. For example, the method includes: receiving information indicating at least one acquired registration point; processing the information indicating at least one acquired registration point to obtain the second transformation. The information indicating at least one acquired registration point may have been acquired by the surgical tracking system 4 tracking a registration probe (e.g., surgical instrument 20). In other words, the information indicating at least one acquired registration point may have been acquired using a registration probe tracked by the surgical tracking system 4. Information may be received from the surgical tracking system 4 (e.g., locator 30) via a communication interface 36. Once the information indicating at least one acquired registration point has been received or the second transformation has been determined, the method may again determine at least one registration instruction, this time based on at least one of the first patient image data 37 and the information indicating at least one acquired registration point and the second transformation. At least one registration instruction displayed can be updated based on at least one of the information indicating at least one acquired registration point and a second transformation.

[0075] For example, a user can place the distal end of the registration probe on a contact point on the surface of the patient's body 29. Information indicating at least one acquired registration point may include the position of the contact point in a third coordinate system 18. Information indicating at least one acquired registration point may include the position of at least one acquired registration point in the third coordinate system 18 of the surgical tracking system 4. Alternatively or additionally, information indicating at least one acquired registration point may include the orientation of the registration probe in contact with the surface of the patient's body 29. Based on a known spatial relationship between the registration probe and the surface of the patient's body, such as a predetermined distance between the tracker attached to the registration probe and the distal end of the registration probe, the position of the contact point between the registration probe and the surface of the patient's body 29 can be determined, for example, by a calculation system 6. This contact point may correspond to or serve as at least one acquired registration point. Users can instruct the surgical navigation system 100, particularly the locator 30 or the computing system 6, when to acquire registration points, for example by pressing the foot pedal of the surgical navigation system 100, pressing the button on the registration probe, sending commands to the surgical navigation system 100, or speaking control commands that can be recognized by the audio interface of the surgical navigation system 100.

[0076] Processing information indicating at least one acquired registration point may include matching the position of at least one acquired registration point with the first patient image data 37. The method may include: acquiring information indicating at least one acquired registration point by obtaining at least one position of the registration probe in a third coordinate system 18, and determining a second transformation by matching the information indicating at least one acquired registration point with the first patient image data 37.

[0077] The position of at least one registration point acquired in the third coordinate system 18 can be matched, for example, with the surface of the patient's body 29 described, detected, or extracted in the first patient image data 37 using a point-to-surface matching algorithm. For example, the first patient image data 37 is a three-dimensional CT image of the patient's head. Image analysis algorithms can be used to detect the surface of the body within the CT image, such as outlier removal and surface smoothing using the minimum and maximum Hounsfield values ​​of the CT image voxels. The detected body surface can correspond to the extracted three-dimensional point cloud or surface described above. Other techniques for detecting the surface of the body in the patient image data can also be used. Matching the position of at least one registration point(s) in the third coordinate system 18 with the surface of the patient's body 29 extracted from the first patient image data 37 can include: matching the position of at least one registration point(s) in the third coordinate system 18 with the three-dimensional point cloud in the first coordinate system 39; this method can be called point-to-point matching. Alternatively, matching the location of at least one registration point(s) in the third coordinate system 18 with the surface of the patient's body 29 detected in the first patient image data 37 may include matching the location of at least one registration point(s) in the third coordinate system 18 with a three-dimensional surface in the first coordinate system 39. This method may be referred to as point-to-surface matching.

[0078] The method may further include: obtaining tracking data describing the attitude of the surgical instrument 20 in a third coordinate system 18, and transforming the attitude of the surgical instrument 20 to a second coordinate system 14 based on a first transformation and a second transformation. (See reference) Figure 1It is evident that the pose in the third coordinate system 18 can be transformed to the pose in the second coordinate system 14 using the first and second transformations. That is, the pose can be transformed from the third coordinate system 18 to the first coordinate system 10 using the second transformation already obtained as described above. Then, the pose can be transformed from the first coordinate system 10 to the second coordinate system 14 using the first transformation obtained as described above. Of course, the method may include determining a combined transformation based on the first and second transformations to directly transform the pose from the third coordinate system 18 to the second coordinate system 14. The pose of the surgical instrument 20 can be transformed from the third coordinate system 18 to the world coordinate system using the second transformation, the first transformation, and the transformation between the second coordinate system 14 and the world coordinate system determined based on the first pose of the AR device 2 in the world coordinate system. The pose of the surgical instrument 20 can be transformed from the world coordinate system to the second coordinate system 14 based on the current pose of the AR device 2 determined in the world coordinate system (e.g., a comparison with the first pose of the AR device 2). As described above, the pose of the surgical instrument 20 in the second coordinate system 14 can be transformed to the coordinate system of the display 15 using a known transformation between the second coordinate system and the coordinate system of the display 15.

[0079] The method may include determining at least one navigation instruction associated with the pose of the surgical instrument 20 in a second coordinate system 14 (or, for example, in a third coordinate system or in a world coordinate system). The at least one navigation instruction may include at least one of a type indication, a position indication, and an orientation indication of the surgical instrument 20 in the second coordinate system 14 (or, for example, in a third coordinate system or in a world coordinate system). The method may also include triggering the display of at least one navigation instruction on the display 15 of the AR device 2 (advantageously in an augmented view superimposed on the patient's body 29). Based on a first transformation and a second transformation, the position (e.g., an indication of ) and orientation (e.g., an indication of ) of the surgical instrument 20 in the third coordinate system may be transformed to the world coordinate system or the second coordinate system 14. The position (e.g., an indication of ) and orientation (e.g., an indication of ) of the surgical instrument 20 in the world coordinate system may be transformed to the second coordinate system 14 using the third pose transformation of the augmented reality device 2 when the display of at least one navigation instruction determined in the world coordinate system (particularly the pose of the camera 12) is triggered. That is, as explained above for at least one registration instruction, the method may include determining a third pose of the AR device 2 in the world coordinate system when the display of at least one navigation instruction is triggered. When acquiring second patient image data, the first pose of the AR device 2 in the world coordinate system may be compared with the third pose to compensate for translational and / or rotational offsets of the AR device 2 relative to the patient's body 29 after (multiple) time points following the acquisition of the second patient image data 52. The display 15 may provide the user of the AR device 2 with at least one navigation instruction (e.g., superimposed on a portion of the patient's body 29), regardless of changes in the relative orientation between the AR device 2 and the patient's body 29 over time.

[0080] Refer again Figure 4 The enhanced view includes at least one navigation command. Specifically, the enhanced view includes a pair of crosshairs 64 indicating the position of the tip of the registration probe tracked by the surgical tracking system 4. The enhanced view also includes a line 66 indicating the orientation of the main axis of the registration probe, such as axis 24. Note that a second transformation may be required to display the at least one navigation command. Therefore, Figure 4 At least one navigation instruction shown may be displayed only once the second transformation has been obtained or determined, as disclosed herein.

[0081] Figure 4Further visualization shows the locations of multiple registration points 68 acquired using the registration probe. These locations are visualized overlaid on contact points on the surface of the patient's body 29, where registration points 68 have been acquired using the tracked registration probe. This informs the user where registration points have been acquired on the surface of the patient's body 29. In the illustration shown, the user has moved the registration probe along the surface of the patient's body 29, starting from the philtrum and moving towards the glabella. A portion 62 of the regions 58, 60 covered by the acquired registration points 68 is highlighted in a different manner than regions 58, 60 to inform the user where additional registration points do not need to be acquired. For example, portion 62 may correspond to a portion of regions 58, 60 within a predetermined distance from one or more acquired registration points 58 and / or having a surface curvature (e.g., average) less than a predetermined threshold at which the surface curvature deviates from (e.g., one or more) the acquired registration points 58. Other criteria may be applied to determine portion 62. In this regard, reference is also made to European Patent Application No. 20198095.0 filed on September 24, 2020, particularly to the determination of the fourth and fifth weighting factors described therein.

[0082] As described above, this method can be performed by updating the display of at least one registration instruction, for example, by determining a new at least one registration instruction based not only on the first patient image data 37 but also on the acquired at least one registration point and the second registration, and triggering the display of the new at least one registration instruction if at least one registration instruction is present. In other words, Figure 4 The enhanced view shown can be updated (e.g., periodically or continuously) based on the number and location of the acquired registration points. Updating the enhanced view may include adding newly acquired registration points and correspondingly changing regions 56, 58, 60, and 62. The second transformation may be updated periodically, such as each time a predetermined number of new registration points have been acquired or upon receiving a user command. This can iteratively improve the accuracy of the second transformation, thereby also improving the accuracy of at least one navigation instruction displayed superimposed on the patient's body 29. As described with reference to the fourth and fifth weighting factors in European Patent Application No. 20198095.0 filed September 24, 2020, the display of the second transformation and at least one registration instruction can be continuously updated during the acquisition of registration points.

[0083] Note again that the invention also covers variations in which a first pose of the AR device 2 in the world coordinate system is determined and a transformation between the second coordinate system 14 and the world coordinate system is determined using the first pose and a first transformation. At least one registration instruction and / or at least one navigation instruction can be determined with reference to the world coordinate system. To display at least one registration instruction or at least one navigation instruction on the display 15, a transformation between the world coordinate system and the second coordinate system 14 or the coordinate system of the display 15 can be determined, for example, based on the current, second, or third pose of the AR device 2 in the world coordinate system or a comparison with the first pose of the AR device 2 in the world coordinate system. This allows at least one registration instruction and / or at least one navigation instruction to be displayed in a fixed spatial relationship relative to the patient's body 29, even if the change in the relative position between the AR device 2 and the patient's body 29 after the first transformation has been determined.

[0084] A computer program product including a program code portion is also provided, which, when executed on at least one processor such as processor 34, is used to perform the methods disclosed herein. The computer program product may be stored on one or more computer-readable, non-transitory recording media (e.g., memory 32). A computer program including a program code portion for performing the methods disclosed herein when executed on at least one processor such as processor 34 may also be provided. The computer program may be stored on a recording medium such as memory 32 or transmitted as a digital or analog data carrier signal or signal wave. Many modifications to the embodiments described herein are possible. For example, the order of the method steps described herein may be changed. Not all method steps are necessary or essential.

[0085] This invention provides an advantageous method for providing user guidance to obtain registration between first patient image data 37 and surgical tracking system 4. An enhanced view, including at least one registration instruction superimposed on the patient's body 29, can be presented to the user of the AR device 2. The registration instruction can guide the user on where on the surface of the patient's body 29 to acquire registration points using a registration probe tracked by surgical tracking system 4. This can help the user, for example, acquire registration points in a repeatable manner.

[0086] At least steps 200-208 of the method disclosed herein may not require the presence or activation of the surgical tracking system 4. This may be advantageous if the user wants to check registration instructions before transferring the patient to the operating room where the surgical tracking system 4 is located or before the surgical tracking system 4 has been set up.

[0087] Surgical tracking system 4 may not need to track AR device 2. This is advantageous in cases where surgical tracking system 4 cannot track AR device 2, for example, if AR device 2 does not include a tracker that can be located by surgical tracking system 4 in third coordinate system 18.

[0088] This method can use a facial model to obtain the transformation between a first coordinate system 39 and a second coordinate system 14. This can improve the accuracy of the obtained first transformation and reduce the processing workload. Furthermore, commonly available AR devices can be structurally optimized for image analysis using facial models, meaning that the method described in this paper can be implemented without requiring complex hardware.

Claims

1. A method for providing user guidance on the transformation between a coordinate system for obtaining patient image data and a coordinate system for a surgical tracking system, the method comprising: Obtain first patient image data (37) of at least a portion of the patient’s body (29) (200); Based on the first patient image data (37), at least one registration instruction is determined (202), the at least one registration instruction indicating where at least one registration point is acquired relative to the surface of the patient's body (29); Obtain (204) second patient image data (52) of at least said portion of the patient’s body (29), the second patient image data (52) having been acquired by augmented reality device (2); Determine (206) the first transformation between the first coordinate system (39) of the first patient image data (37) and the second coordinate system (14) of the second patient image data (52); as well as Based on the at least one registration instruction and the first transformation, trigger (208) displays the at least one registration instruction on the display (15) of the augmented reality device (2), such that an augmented view is presented to the user of the augmented reality device (2), wherein the at least one registration instruction is superimposed on at least said portion of the patient’s body (29), and the augmented view guides the user on where to acquire the at least one registration point; The at least one registration instruction includes an indication of a sub-part (58, 60) of the surface of at least a portion of the patient’s body (29), wherein the at least one registration point will be acquired in the sub-part (58, 60), and the sub-part is determined by identifying a sub-part of the surface of the patient’s body described by first patient image data having one or more predefined attributes. Among them, the one or more predefined attributes are: Low mechanical deformability; Surface curvature exceeding the minimum curvature; Surface curvature that is opposite to the surface curvature of a predefined surface region; Surface curvature that deviates from the predefined region by more than a predefined amount; The surface curvature that is opposite to or deviates from the remaining surface of the patient's body (29) by more than a predefined amount; The surface curvature at at least one acquired registration point deviates from the surface curvature by more than a predefined amount; and / or The spatial distance to at least one of the acquired registration points exceeds a predefined amount.

2. The method according to claim 1, comprising: Receive information indicating at least one of the acquired registration points; as well as The information indicating at least one acquired registration point is processed to obtain a second transformation between the first coordinate system (39) of the first patient image data (37) and the third coordinate system (18) of the surgical tracking system (4).

3. The method according to claim 2, wherein, Information indicating that at least one of the acquired registration points has been acquired by the surgical tracking system (4) of the tracking registration probe (20).

4. The method according to claim 2, wherein, Processing information indicating at least one acquired registration point includes matching the location of the at least one acquired registration point with the first patient image data (37).

5. The method according to claim 2, further comprising: Information indicating the at least one registered point is acquired by obtaining at least one position of the registration probe (20) in the third coordinate system (18); as well as The second transformation is determined by matching information indicating at least one acquired registration point with the first patient image data (37).

6. The method according to claim 2, further comprising: Obtain tracking data describing the attitude of the surgical instrument (20) in the third coordinate system (18); Based on the first transformation and the second transformation, the posture of the surgical instrument (20) is transformed to the second coordinate system (14). Determine at least one navigation command associated with the attitude of the surgical instrument (20) in the second coordinate system (14); and The at least one navigation instruction is triggered to be displayed on the display (15) of the augmented reality device (2).

7. The method according to claim 1, wherein, The portion of the patient's body (29) includes one or more portions of the patient's face, and wherein determining the first transformation includes: The generic facial model is matched to at least one of the first patient image data (37) and the second patient image data (52).

8. The method according to claim 7, wherein, Determining the first transformation includes: The initial deformed facial model (38) is determined by matching the general facial model to one of the first patient image data (37) and the second patient image data (52); and The first deformed facial model (38) is compared with another of the first patient image data (37) and the second patient image data (52) to determine the first transformation.

9. The method according to claim 8, wherein, The generic facial model is matched to one of the first patient image data (37) and the second patient image data (52) such that each of the plurality of predefined landmarks of the generic facial model is located on the corresponding landmark in one of the first patient image data (37) and the second patient image data (52) of the part of the patient's body (29).

10. The method according to claim 9, wherein, The corresponding landmark is the biometric feature of the patient's body (29).

11. The method according to claim 8, wherein, Comparing the initially deformed facial model (38) with another of the first patient image data (37) and the second patient image data (52) includes: A secondary deformed facial model is determined by matching the general facial model or the initially deformed facial model (38) to another of the first patient image data (37) and the second patient image data (52); and The first deformed facial model (38) is compared with the second deformed facial model.

12. The method according to claim 9, wherein, Comparing the initially deformed facial model (38) with another of the first patient image data (37) and the second patient image data (52) includes: Perform image analysis on the other of the first patient image data (37) and the second patient image data (52) to determine the location of at least one of the corresponding landmarks in the other of the first patient image data (37) and the second patient image data (52); and The determined position of at least one of the corresponding landmarks in the other of the first patient image data (37) and the second patient image data (52) is compared with the position of one of a plurality of predefined landmarks of the initially deformed facial model located on the same corresponding landmark in the first patient image data (37) and the second patient image data (52).

13. The method of claim 11, comprising: Determine whether the result of a comparison between (i) the initially deformed facial model (38) and (ii) the second deformed facial model, or the other of the first patient image data (37) and the second patient image data (52), meets one or more predefined acceptance criteria; and If the result meets one or more predefined acceptance criteria, then the display of at least one registration instruction is triggered.

14. The method of claim 1, comprising: After the at least one registration instruction has been triggered, the surgical tracking system (4) is instructed to start tracking.

15. The method according to claim 1, wherein, The second patient image data refers to a two-dimensional image captured by the camera (12) of the augmented reality device (2) or a three-dimensional image data captured by the sensor of the augmented reality device (2).

16. A system (6) comprising at least one memory (32) and at least one processor (34), the at least one memory (32) storing instructions that, when executed on the at least one processor (34), cause the at least one processor (34) to: Obtain first patient image data (37) of at least a portion of the patient’s body (29) (200); Based on the first patient image data (37), at least one registration instruction is determined (202), the at least one registration instruction indicating where at least one registration point is acquired relative to the surface of the patient's body (29); Obtain (204) second patient image data (52) of at least said portion of the patient’s body (29), the second patient image data (52) having been acquired by augmented reality device (2); Determine (206) the first transformation between the first coordinate system (39) of the first patient image data (37) and the second coordinate system (14) of the second patient image data (52); as well as Based on the at least one registration instruction and the first transformation, trigger (208) displays the at least one registration instruction on the display (15) of the augmented reality device (2), such that an augmented view is presented to the user of the augmented reality device (2), wherein the at least one registration instruction is superimposed on at least said portion of the patient’s body (29), and the augmented view guides the user on where to acquire the at least one registration point; The at least one registration instruction includes an indication of a sub-part (58, 60) of the surface of at least a portion of the patient’s body (29), wherein the at least one registration point will be acquired in the sub-part (58, 60), and the sub-part is determined by identifying a sub-part of the surface of the patient’s body described by first patient image data having one or more predefined attributes. Among them, the one or more predefined attributes are: Low mechanical deformability; Surface curvature exceeding the minimum curvature; Surface curvature that is opposite to the surface curvature of a predefined surface region; Surface curvature that deviates from the predefined region by more than a predefined amount; The surface curvature that is opposite to or deviates from the remaining surface of the patient's body (29) by more than a predefined amount; The surface curvature at at least one acquired registration point deviates from the surface curvature by more than a predefined amount; and / or The spatial distance to at least one of the acquired registration points exceeds a predefined amount.

17. The system (100) according to claim 16, further comprising: At least one component, said at least one component being selected from: a) The augmented reality device (2) may optionally be configured as a head-mounted display; as well as b) Surgical tracking system (4).

18. The system (100) of claim 17, wherein the system is configured such that the selection from: i) the augmented reality device (2); and ii) The patient's body (29) At least one object cannot be tracked by the surgical tracking system (4).

19. A computer program product comprising a program code portion configured to, when the computer program product is executed on at least one processor, perform a method for providing user guidance to obtain patient image data by transforming a coordinate system between a coordinate system and a surgical tracking system, wherein... The method includes: Obtain first patient image data (37) of at least a portion of the patient’s body (29) (200); Based on the first patient image data (37), at least one registration instruction is determined (202), the at least one registration instruction indicating where at least one registration point is acquired relative to the surface of the patient's body (29); Obtain (204) second patient image data (52) of at least said portion of the patient’s body (29), the second patient image data (52) having been acquired by augmented reality device (2); Determine (206) a first transformation between the first coordinate system (39) of the first patient image data (37) and the second coordinate system (14) of the second patient image data (52); and Based on the at least one registration instruction and the first transformation, trigger (208) displays the at least one registration instruction on the display (15) of the augmented reality device (2), such that an augmented view is presented to the user of the augmented reality device (2), wherein the at least one registration instruction is superimposed on at least said portion of the patient’s body (29), and the augmented view guides the user on where to acquire the at least one registration point; The at least one registration instruction includes an indication of a sub-part (58, 60) of the surface of at least a portion of the patient’s body (29), wherein the at least one registration point will be acquired in the sub-part (58, 60), and the sub-part is determined by identifying a sub-part of the surface of the patient’s body described by first patient image data having one or more predefined attributes. Among them, the one or more predefined attributes are: Low mechanical deformability; Surface curvature exceeding the minimum curvature; Surface curvature that is opposite to the surface curvature of a predefined surface region; Surface curvature that deviates from the predefined region by more than a predefined amount; The surface curvature that is opposite to or deviates from the remaining surface of the patient's body (29) by more than a predefined amount; The surface curvature at at least one acquired registration point deviates from the surface curvature by more than a predefined amount; and / or The spatial distance to at least one of the acquired registration points exceeds a predefined amount.

Images