System and method for augmented reality visualization of surgical procedure steps for training and testing an X-ray image-based surgical navigation system.

Abstract

A system and method are proposed that utilizes a processing unit for running augmented reality devices and computer program products. The augmented reality device can track objects in a room, and based on the processing of tracking data associated with those objects, and based on the processing of simulated X-ray image data, additional information can be visualized within the user's field of view. Such systems and methods are adapted for use in training and testing purposes.

Description

Technical Field

[0001] The present invention relates to the field of augmented reality visualization. Further, the present invention relates to computer-assisted surgery and robot-assisted surgery. The systems and methods described herein can be applied to assist in performing steps of surgical procedures in both training settings and actual surgical procedures. The method can be implemented as a computer program product executable on a processing unit of the system.

Background Art

[0002] Surgical navigation or computer-assisted surgery (CAS) has become a procedure commonly performed to improve patient outcomes and reduce X-ray exposure to patients and operating room staff. However, there may be user-interface issues that can cause misunderstandings. For example, a surgeon, nurse, or X-ray technician may not understand what the CAS system currently expects as the next step. Even when the user understands what the system expects, it may still be a problem to know how to correctly follow the instructions. As an example, C-arm positioning to obtain a desired X-ray image can be cumbersome, resulting in more exposure and more procedure time than necessary. The CAS system can know how to reposition the C-arm to obtain the next X-ray image, but displaying that information may cause misunderstandings.

Summary of the Invention

Problems to be Solved by the Invention

[0003] When using a CAS system, it may be of interest to test how the CAS system functions and whether the information provided by the system can be verified. The problem may be whether the CAS system functions correctly.

Means for Solving the Problems

[0004] At least one or the other of the above-mentioned problems is mitigated or resolved by the subject matter of each independent claim. Further embodiments are described in the respective dependent claims.

[0005] Generally, the system according to the present invention comprises an augmented reality device and a processing unit for executing a computer program product. The augmented reality device can track objects in a room and visualize additional information within the user's field of view based on the processing of tracking data associated with those objects and based on the processing of simulated X-ray imaging data.

[0006] Augmented reality devices will be understood to cover all devices on the augmented reality spectrum and are therefore a superset of virtual reality, augmented reality, and mixed reality devices. This represents all devices that a user can wear and that (1) can alter the user's perception through any sense, including sight, hearing, smell, taste, and touch, and (2) can track its environment through various sensor information, including cameras, depth sensors, accelerometers, and magnetometers. These sensors can track an environment that includes imaging devices and / or at least one object, such as an anatomical model, instrument, or implant. It should be noted that the tracking information allows for the distinction between objects and / or devices. For example, an augmented reality device may comprise a pair of glasses that allows the user to see the environment but is configured to visualize information or virtual representations of objects within the user's field of view.

[0007] A method for augmented reality training or testing using an X-ray imaging-based computer-aided surgical device, when performed by a processing unit of the system, may include the steps of: determining the 3D pose of the X-ray imaging device in a coordinate system; determining the 3D pose of an object in a coordinate system; generating a simulated X-ray projection image of the object, wherein the simulated X-ray projection image is generated based on the determined 3D pose of the X-ray imaging device and the determined 3D pose of the object; interpreting the simulated X-ray projection image; and providing information to a user by an augmented reality device, wherein the augmented reality device is configured to track the 3D pose of the augmented reality device in a coordinate system, and the information provided is based on the interpretation of the simulated X-ray projection image.

[0008] According to one embodiment, a system comprising an augmented reality device and a processing unit connected thereto may be implemented, comprising the steps of: determining the 3D pose of an X-ray imaging device, wherein if the X-ray imaging device is a real X-ray imaging device, the 3D pose of the X-ray imaging device may be based on tracking information received from the augmented reality device; and determining the 3D pose of an object, wherein if the object is a real object, the 3D pose of the object may be based on tracking information received from the augmented reality device. It will be understood that both the X-ray imaging device and the object may be virtual so that their respective 3D poses can be determined by the processing unit of the system without external tracking information. In the context of this disclosure, “3D pose” means a 3D position and orientation determined with respect to a coordinate system.

[0009] Furthermore, a step may be performed to generate a simulated X-ray projection image of an object, which is generated based on the 3D pose of the determined X-ray imaging device and the 3D pose of the determined object. Two or more objects may be visible in the X-ray image, and each of these objects may or may not be tracked by the augmented reality device.

[0010] The method may further include the steps of interpreting a simulated X-ray projection image and providing information to the user, the information being based on the interpretation of the simulated X-ray projection image.

[0011] Part of the interpretation may involve determining the relative spatial positions between objects or between objects and the X-ray imaging device. This spatial relationship may be compared to the spatial relationship expected based on the 3D pose of the objects and / or the 3D pose of the X-ray imaging device. Therefore, it can also be used for verification or testing purposes, for example, to verify an algorithm for interpreting X-ray images based on the assumption that the 3D pose determination performed by an augmented reality device is correct, or to verify the pose determination of objects and / or X-ray imaging devices performed by an augmented reality device based on the assumption that the interpretation of simulated X-ray images is correct. A combination of both is also possible.

[0012] A single X-ray image may be required to verify the accuracy of real-time tracking information, that is, to determine whether the last registration of an object to another object (i.e., the determination of its 3D pose) is still valid. In other words, the proposed system and method (implemented as a computer program product) may be configured to determine the validity of 3D spatial relationships based on a single X-ray image depicting at least a portion of an object.

[0013] In other words, determining the 3D pose of an object yields a simulated X-ray projection image, and based on the interpretation of that projection image, information about the object's spatial position can be provided. For training purposes, such information may be used to guide the user during the procedure. For testing the system, the information may be compared with the determined 3D pose. Based on such a comparison, it can be shown whether the information provided based on the interpretation of the projection image is sufficiently accurate, i.e., whether it falls within a predetermined deviation range of the expected value.

[0014] Interpretation of a simulated X-ray image may include at least one of the following: (i) determining the geometric aspect of an object; (ii) applying a model of the object; (iii) identifying points of interest; (v) measuring dimensions; (vi) measuring distances; (vii) considering imaging characteristics; (viii) considering the characteristics of the object; and (ix) determining relative spatial positioning information between at least one imaged object and / or X-ray imaging device.

[0015] Taking imaging characteristics into consideration may include applying prior knowledge of the physical effects of X-ray image distortion caused by the Earth's magnetic field or pillow distortion. Taking object characteristics into consideration may include prior knowledge of typical mechanical deformations of instruments, such as bending of a drill bit or mechanical play between a drill bit and a drill sleeve.

[0016] The determination of relative spatial positioning information between two or more objects is disclosed in more detail, for example, in the claims of European Patent Application Publication No. 19217245 and European Patent Application Publication No. 21175405. Furthermore, imaging and object characteristics that may be relevant when processing X-ray images are described in more detail in these documents. These detailed disclosures are incorporated herein by reference.

[0017] A determined spatial relationship between at least one imaged object and / or X-ray imaging device may be compared to the 3D pose of the X-ray imaging device and / or the 3D pose of the object, which is generally determined according to one embodiment of the present disclosure.

[0018] Furthermore, the determination of the 3D pose of the X-ray imaging device and / or the 3D pose of the object may be adjusted based on the interpretation of the simulated X-ray projection image. Typically, the information provided to the user may be adjusted accordingly.

[0019] The method may further include the step of determining a representation of an X-ray imaging device and / or an object, the representation of the X-ray imaging device and / or an object based on the tracked 3D pose of an augmented reality device and the determined 3D pose of the X-ray imaging device and / or the determined 3D pose of the object. Furthermore, the representation may be visualized by an augmented reality device.

[0020] In the context of this disclosure, a representation should be understood, for example, as a visualization of at least an aspect of an X-ray imaging device or object. An aspect may be a point or an axis, a contour, or even a rendered surface. In other words, a representation of a virtual device or object may provide the user with sufficient information to enable recognition of the device or object itself, as well as its 3D position and 3D orientation.

[0021] Several different combinations of objects and X-ray devices are included. An object may be at least one of the following: (i) at least a part of a virtual anatomical structure, (ii) at least a part of an actual anatomical structure model, (iii) at least a part of a virtual instrument, (iv) at least a part of an actual instrument, (v) at least a part of an actual implant, and (vi) at least a part of a virtual implant. An X-ray imaging device may be an actual X-ray imaging device or a virtual X-ray imaging device. Note that the actual anatomical structure model mentioned may be an artificially constructed part of a dummy or mockup of an anatomical structure.

[0022] It will be understood that the method may include the step of tracking at least one object and / or an X-ray imaging device by an augmented reality device, where at least one object is a real object and / or the X-ray imaging device is a real X-ray imaging device.

[0023] The information provided to the user may include at least one of the following: (i) advice on how to reposition the X-ray imaging device; (ii) visualization of the position of the virtual image plane of the X-ray image; (iii) visualization of the position of the virtual image plane of a previously simulated X-ray image; and (iv) advice on the next steps in the procedure.

[0024] Information based on the interpretation of simulated X-ray images may be provided to the user by an augmented reality device.

[0025] Regarding the provision of information to the user, it should be noted that information based on the interpretation of simulated X-ray images can be perceived by vision, hearing, smell, taste and / or touch. For example, the system may provide instructions acoustically or provide warnings optically by changing the color of light. Although virtual X-ray imaging devices and / or virtual objects can be visualized on an augmented reality device, it will be understood that some information, such as measurement values, can also be provided on a fixed monitor.

[0026] The method may further include the steps of receiving an input and updating the information provided to the user based on the input.

[0027] In the context of the present disclosure, an input may be understood as any kind of active interaction with the system. For example, the user may change the 3D pose of a device and / or an object, and the augmented reality device tracks the change, and the new tracking information may be considered as an input to the system. On the one hand, the user may change the 3D pose of an actual device and / or an actual object that can be tracked by the augmented reality device. On the other hand, the user may change the 3D pose of a virtual device and / or a virtual object by a tracked hand gesture, a controller, or any other suitable computer input device.

[0028] As described above, the method may be implemented in a complete virtual space where X-ray imaging devices and any objects, i.e., instruments, implants, anatomical structures, are virtually generated and visualized on the screen of an augmented reality device. Such virtual devices and objects may move due to an input or as a result of the interpretation of simulated X-ray images. On the other hand, either the imaging device or the object may be real and may be tracked by the augmented reality device. Such actual devices and objects can be seen through the glasses of the augmented reality device and thus may not be displayed on the screen in the form of a virtual representation.

[0029] However, combinations are also possible. The representation of the actual X-ray imaging device and / or the actual object can be visualized by the augmented reality device, such that the representation appears as an overlay on the X-ray imaging device and / or on the object that is simultaneously visible through the glasses of the augmented reality device.

[0030] Such an overlay may provide the possibility to recognize whether the tracking and / or determination of the 3D pose of the device or object is accurate enough. The deviation between the virtual representation and the actual object can be easily recognized.

[0031] As described above, the tracked 3D pose of the device and / or object may be updated by the processing unit of the system. Such an update may be visualized by a change in the representation of the device and / or object. Thus, the representation may be manipulated due to user input. On the other hand, the system may autonomously manipulate the representation based on the interpretation of the simulated X-ray image. By autonomously manipulating the representation, the system may further provide the user with suggestions on how to proceed. For example, the system may calculate and display a virtual overlay of a desired (other) position of the X-ray imaging device with respect to the current position of the X-ray imaging device based on the interpretation of the simulated X-ray image.

[0032] In addition to using the system to train the user, it is also possible to use the user input to generate training data for further enhancing the system. For example, the system may automatically adapt to the user's preferences.

[0033] It should be noted that the processing units of the systems described herein may be implemented by a single processor that performs all steps of the process, or by a group or multiple processors that do not need to be located in the same place. For example, cloud computing allows processors to be located anywhere. For example, a processing unit may be divided into a first subprocessor that controls user interaction, including an augmented reality device for visualizing results, and a second subprocessor (possibly located elsewhere) that performs all calculations. The first or another subprocessor may also control the movement of, for example, an X-ray imaging device. A processing unit integrated with an augmented reality device may control any tracking of objects and any visualizations on the augmented reality device's screen.

[0034] The system may further include storage means that provide, for example, a database for storing X-ray images. Such storage means may also be provided in a network to which the system may be connected, and data related to the neural network may be received through that network. Furthermore, the device may include an imaging unit for generating at least one 2D X-ray image, and the imaging unit may be capable of generating images from different directions.

[0035] It should be noted that all references to the movement of the C-arm (imaging device) in this disclosure always refer to the relative repositioning between the C-arm and the patient model. Therefore, any C-arm translation or rotation may generally be replaced by the corresponding translation or rotation of the patient / OR table model, or a combination of C-arm translation / rotation and patient model / table translation / rotation. Information regarding positional changes may be provided by a tracking system of an augmented reality device. Alternatively or additionally, such information may be provided by positioning a sensory device on the patient table and / or the robotic model of the robotic-operated imaging device, or by a reference frame attached to the patient table or anatomical model, and / or by an optical navigation system tracking the imaging device. The system may calculate adjustments to the C-arm and / or the patient model. Furthermore, it should be noted that all references to the C-arm may similarly apply to the G-arm, O-arm, etc.

[0036] The methods and techniques disclosed herein may be used in systems assisting a human user or surgeon, or in systems in which some or all of the steps are performed by a robot. Accordingly, references to “user” or “surgeon” in this disclosure may refer to a human user as well as a robotic surgeon, a mechanical assistance device, or similar apparatus. Similarly, whenever a method for adjusting a C-arm is instructed, it should be understood that such adjustment may be performed without human intervention, i.e., automatically, by the robotic C-arm, by the robotic table, or by operating room staff with some form of automated assistance. It should be noted that because robotic surgeons and / or robotic C-arms can operate with greater precision than humans, repetitive procedures may require fewer repetitions, and more complex commands (e.g., combining multiple repetitive steps) may be performed. A key difference between robots and human surgeons is the fact that a robot may keep the instrument completely still between the acquisition of two X-ray images.

[0037] Similar to the automatically controlled movement of real objects and / or devices, augmented reality devices can provide virtual representations of objects and / or devices within the user's field of view. Thus, objects and / or devices may move virtually, and this movement is controlled by a computer program product executed by a processing unit.

[0038] The computer program product may preferably be loaded into the main memory of the data processor during execution. Therefore, a data processor or processing unit of a system according to one embodiment may be equipped to perform at least a portion of the described process. Furthermore, this disclosure may relate to computer-readable media, such as a CD-ROM, that can store the disclosed computer program product. However, the computer program product may also be presented over a network such as the World Wide Web and can be downloaded from such a network to the main memory of the data processor for immediate execution, or downloaded to the non-volatile memory of the data processor for execution at a later time. Furthermore, the computer program product may also be executed on a cloud-based processor, with the results presented over the network.

[0039] It should be noted that prior information about the object (e.g., the size and type of the drill or chisel) can be obtained before or during the method step by simply scanning the package (e.g., barcode) or by any writing to the object itself.

[0040] As is evident from the above description, one embodiment may involve processing X-ray image data, enabling the automatic interpretation of visible objects or the determination of important regions in the X-ray image. The methods described herein should be understood as methods for training users when applying the system or for testing the system. Therefore, the methods do not necessarily involve surgical treatment steps of animals or humans.

[0041] It should be further noted that the embodiments are described with reference to different subject matter. In particular, some embodiments are described with reference to method-type claims (computer program products), and other embodiments are described with reference to apparatus-type claims (systems / devices). However, those skilled in the art will infer from the above and below descriptions that, unless otherwise specified, any combination of features belonging to one type of subject matter, and any combination of features relating to different subject matter, are disclosed in this application.

[0042] Furthermore, it should be noted that the aforementioned system can be used as a game for entertainment or competition.

[0043] The embodiments defined above, as well as further embodiments, features, and advantages of the present invention, can also be derived from the examples of embodiments described below, and will be illustrated with reference to the examples of embodiments shown in the figures, but the present invention is not limited thereto. [Brief explanation of the drawing]

[0044] [Figure 1] This figure shows how a user utilizes the system disclosed herein. [Figure 2] This is a flowchart of the method disclosed herein. [Modes for carrying out the invention]

[0045] Figure 1 illustrates a situation in which the systems and methods disclosed herein may be utilized. The augmented reality device 10 may be a head-mounted device equipped with sensors for tracking objects in a room. Such objects may be instruments such as a drilling machine 20 with a drill bit 22, an operating table 30, a C-arm-based X-ray imaging device 40, and / or parts of a model of a patient 50 or an anatomical structure model. It will be understood that the entirety of each object does not need to be tracked by the sensors of the augmented reality device. For example, it may be sufficient to detect the radiation source and / or radiation detector of the imaging device to determine the 3D pose of the imaging device, i.e., the 3D position and 3D orientation of the imaging device. Otherwise, it may be sufficient to detect the radiation detector along with at least a portion of the C-arm and the base of the imaging device to enable the determination of the 3D pose of the device. Similarly, only a portion of the table 30 or instrument 20 may need to be detected by the tracking sensor.

[0046] One way to interpret Figure 1 is that the dark areas of the object and device, i.e., the parts shown as black surfaces, are outside the tracking area, while the bright areas of the object and device, i.e., the parts shown as dotted outlines, are the parts detected by the augmented reality device's sensors, thus allowing the object and device to be tracked and its 3D pose to be determined.

[0047] Knowing the 3D pose of the imaging device 40 and the 3D pose of the patient model 50 enables the generation of simulated X-ray images. On the one hand, such simulated X-ray images can be used to provide an impression of whether the actual X-ray image visualizes the intended portion of the patient from the intended imaging direction. This may be automatically evaluated by the system. On the other hand, the simulated X-ray images allow for training of the system. Based on such simulated X-ray images, the system may provide information regarding the next steps in the treatment procedure.

[0048] In an alternative scenario, at least one of the objects and devices in Figure 1 is not present in the room but is shown as a virtual representation on the display of the augmented reality device 10. In such a scenario, the user may also position the imaging device relative to an anatomical structure or a model of an anatomical structure, and the user may be provided with simulated X-ray images to enable evaluation of a treatment procedure. For example, the patient 50 may not actually be on the table 30, or instead of the patient, an actual training model of the patient may be on the table, and the user may be provided with a virtual visualization of the patient as an overlay on the actual training model. Thus, the user can position only the instrument 20 as if a patient were present. Nevertheless, the system is configured to provide simulated X-ray images based on which procedures can be trained using the system.

[0049] In another example, the patient model and the drilling machine model can be visualized in augmented reality, for example, while the drilling machine is being tracked by an augmented reality device, resulting in real-time updates to the drilling machine's virtual model pose as the actual drilling machine moves. On the other hand, the patient model may not be tracked by the augmented reality device. Information derived from the interpretation of a simulated X-ray image can be used to determine the patient model's pose relative to the X-ray imaging device's pose at time 1, and the patient model's pose information can be updated based on the assumption that the patient model's position and orientation do not change relative to the coordinate system until the next X-ray image is simulated at time 3. In this scenario, the patient model's pose can be modified to simulate unintended, untracked patient movements for the simulation of this next X-ray image. Thus, navigation information that may be inaccurate at time 2 is updated at time 3 based on the interpretation of the simulated X-ray image and is therefore accurate based on further real-time tracking-based information of the drilling machine's pose provided by the augmented reality device after time 3, unless there are new unintended, untracked patient movements.

[0050] As a further example, a table 30, an imaging device 40, and a patient 50 may be provided as virtual representations on an augmented reality device 10. In such a scenario, the user may, for example, virtually interact with the virtual imaging device to position it relative to a virtual patient on a virtual table. The system then provides a simulated X-ray image based on the 3D pose of the imaging device and the 3D pose of the virtual patient, and the system evaluates the image to provide information about the position and orientation of the instrument, or information about the path, such as the perforation path.

[0051] Another way to interpret Figure 1 is that an object or device is visualized as a virtual representation by a dotted line outlining its contour, or as a black area representing the actual object or device shown in Figure 1. It will also be understood that the virtual representation of an object can be visualized on an augmented reality device as an overlay on the actual object within the user's field of view 12.

[0052] As illustrated by the example, the system provided by this disclosure enables complete virtual training scenarios for patient treatment, as well as any hybrid settings where objects are real and others are virtual. In any case, the system is configured to provide simulated X-ray images and evaluation of the image content.

[0053] As a result, the system may be used to virtually perform a patient's treatment before the patient is actually treated, i.e., when planning the treatment. The system may also be used after the patient's actual treatment with data for learning and teaching purposes. In this case, at least one actual X-ray image of a previous surgical procedure may be interpreted by the system, and the user may be asked to follow system instructions to observe whether and how the next necessary step in the surgery (e.g., repositioning an object or X-ray device) was performed better (e.g., faster) by the human user / surgical robot.

[0054] The method is described below with reference to Figure 2, which shows a flowchart.

[0055] It will be understood that the steps of the methods described herein, in particular, the steps of the methods described in relation to the workflow visualized in the figures, for example, are the main steps, and these main steps may be distinguished or divided into several substeps. Furthermore, there may be additional substeps between these main steps. It will also be understood that only a portion of the whole method may constitute the present invention, i.e., steps may be omitted or summarized.

[0056] When using augmented reality devices, X-ray imaging-based computer-aided surgical procedures may include the following steps:

[0057] In step S1, the 3D pose of the X-ray imaging device is determined. The 3D pose of the X-ray imaging device may be based on tracking information received from an augmented reality device. It will be understood that the 3D pose of the imaging device is generated by the system based on a computer simulation of what would happen if the imaging device were not actually present.

[0058] In step S2, the 3D pose of the object is determined. The 3D pose of the object may be based on tracking information received from an augmented reality device. Alternatively, the 3D pose may be computer-generated for the virtual object.

[0059] In step S3, a simulated X-ray image of the object is generated, and the simulated X-ray image is generated based on both the determined 3D pose of the X-ray imaging device and the determined 3D pose of the object.

[0060] In step S4, the simulated X-ray image is interpreted by the system. A computer program product executed by the system's processing unit may evaluate the information in the simulated X-ray image. Objects may be classified and localized relative to the imaging device based on the simulated X-ray image. In the case of a mock-up bone, a fracture may be identified.

[0061] In step S5, information is provided to the user, based on the interpretation of S4. For example, suggestions on how to treat a fracture may be provided to the user. If an instrument is to be used, the spatial relationship of the instrument to the mock-up bone can be determined, and for example, a desired insertion path for the instrument may be provided, and possible deviation values ​​(such as an angle in degrees or an offset in millimeters) based on the current position of the instrument from this insertion path may be provided.

[0062] In step S6, which is an optional step, the representation of the X-ray imaging device and / or object is determined. The representation of the X-ray imaging device and / or object may be based on the viewpoint of the augmented reality device and the determined 3D pose of the X-ray imaging device and / or the determined 3D pose of the object. As described above, the representation may be visualized on the augmented reality device either alone if the imaging device and / or object is not in the user's field of view, or as an overlay if the imaging device and / or object is present. It should be noted that if the imaging device and / or object is present, such an overlay may have the advantage of providing the user with immediate recognition of whether the tracking of the imaging device and / or object is accurate.

[0063] In step S7, input is received. Information provided to the user may be updated based on the input. The input means may be, for example, a computer keyboard, computer mouse, or touchscreen for controlling a pointing device such as a cursor on a monitor screen, and these may be included in the device. The device may also include a camera or scanner for reading package labels or identifying implants or surgical instruments. The camera may also allow the user to communicate visually with the device by gesture or imitation, for example, by virtually touching (including dragging and dropping) the device displayed on an augmented reality device. The device may also include a microphone and / or speaker for acoustic communication with the user.

[0064] In step S8, the representation of the X-ray imaging device and / or object may be manipulated. On the one hand, the system may automatically adapt the visualization of the representation in response to the user's movements, and therefore the movements of the augmented reality device. On the other hand, the representation may be manipulated based on user input. For example, the user may virtually move the imaging device and / or object.

[0065] Embodiments are illustrated and described in detail in the drawings and the foregoing description, but such examples and descriptions should be considered illustrative or exemplary, not limiting, and the present invention is not limited to the disclosed embodiments.

[0066] Other variations of the disclosed embodiments can be understood and achieved by those skilled in the art in carrying out the claimed invention, based on a study of the drawings, this disclosure, and the appended claims. In the claims, the words “comprising” do not exclude other elements or steps, and the indefinite articles “a” or “an” do not exclude plural. A single processor or other unit may perform the functions of several of the items described in the claims.

[0067] The mere fact that certain means are described in different dependent claims does not mean that combinations of these means cannot be used effectively.

Claims

1. A method for augmented reality training or testing using an X-ray imaging-based computer-assisted surgical device, The steps include determining the 3D pose of the X-ray imaging device in the coordinate system, The steps include determining the 3D pose of the object in the aforementioned coordinate system, A step of generating a simulated X-ray projection image of the object, wherein the simulated X-ray projection image is generated based on the determined 3D pose of the X-ray imaging device and based on the determined 3D pose of the object. The steps include interpreting the simulated X-ray projection image, A step of providing information to a user by an augmented reality device, wherein the augmented reality device is configured to track the 3D pose of the augmented reality device in the coordinate system, and the information provided is based on the interpretation of the simulated X-ray projection image. Methods that include...

2. The method according to claim 1, wherein the step of interpreting the simulated X-ray projection image includes at least one of the following: (i) determining the geometric aspect of the object; (ii) applying a model of the object; (iii) identifying points of interest; (v) measuring dimensions; (vi) measuring distances; (vii) considering the properties of the object; and (viiii) considering imaging properties.

3. The method according to claims 1 and 2, wherein the step of interpreting the simulated X-ray projection image includes determining the spatial relationship between at least one imaged object and / or the X-ray imaging device.

4. The method according to claim 3, further comprising comparing the spatial relationship between the at least one imaged object and / or the X-ray imaging device with the determined 3D pose of the X-ray imaging device and / or the determined 3D pose of the object.

5. The method according to any one of claims 1 to 4, further comprising the step of adjusting the determination of the 3D pose of the X-ray imaging device and / or the determination of the 3D pose of the object based on the interpretation of the simulated X-ray projection image, wherein the information provided to the user is adjusted based on the interpretation of the simulated X-ray image.

6. The method according to any one of claims 1 to 5, further comprising the steps of determining a representation of the X-ray imaging device and / or the object, the representation being based on the tracked 3D pose of the augmented reality device and the determined 3D pose of the X-ray imaging device and / or the determined 3D pose of the object, and visualizing the representation by the augmented reality device.

7. The method according to any one of claims 1 to 6, wherein the object is at least one object from the group consisting of (i) at least a part of a virtual anatomical structure, (ii) at least a part of an actual anatomical structure model, (iii) at least a part of a virtual instrument, (iv) at least a part of an actual instrument, (v) at least a part of an actual implant, and (vi) at least a part of a virtual implant, and the X-ray imaging device is an actual X-ray imaging device or a virtual X-ray imaging device.

8. The method according to claim 7, further comprising the step of tracking the at least one object and / or the X-ray imaging device by the augmented reality device, where the at least one object is a real object and / or the X-ray imaging device is a real X-ray imaging device.

9. The method according to any one of claims 1 to 8, wherein the information provided to the user includes at least one of the following: (i) advice on how to reposition the X-ray imaging device; (ii) a visualization of the position of the virtual image plane of the X-ray image; (iii) a visualization of the position of the virtual image plane of a previously simulated X-ray image; and (iv) advice on the next steps of the procedure.

10. The method according to any one of claims 1 to 9, further comprising the steps of receiving an input and updating the information provided to the user based on the input.

11. The method according to any one of claims 1 to 10, wherein, if the object is a real object and / or the X-ray imaging device is a real X-ray imaging device visible through the augmented reality device, the representation of the X-ray imaging device and / or the object is visualized by the augmented reality device such that the representation appears as an overlay on the X-ray imaging device and / or the object.

12. A computer program product that, when executed on a processing unit, includes a set of instructions causing the processing unit to perform the steps of the method according to any one of claims 1 to 11.

13. A system for augmented reality training or testing using an X-ray imaging-based computer-assisted surgical device, comprising: an augmented reality device configured to provide tracking information to track the 3D pose of the augmented reality device in a coordinate system and to visualize information within the user's field of view; and a processing unit connected to the augmented reality device and configured to execute the computer program product described in claim 12.

14. The system according to claim 13, further comprising an input device.

Images