Techniques for providing a user with guidance on where to arrange an object of interest in an operating room

Abstract

A technique and method of providing guidance to a user on where to arrange an object of interest in an operating room are disclosed. The method comprises obtaining a spatial pose of an item arranged within the operating room and obtaining a requirement on a spatial relationship between the object of interest and the item. The method further comprises determining: (a) a first spatial portion in which the object of interest is recommended to be arranged, the first spatial portion being larger than the object of interest; and / or (b) a second spatial portion in which the object of interest is not allowed to be arranged. Furthermore, the method comprises triggering a visual display of the first spatial portion and / or the second spatial portion to guide the user on where to arrange the first object of interest in the operating room.

Description

Technical Field

[0001] This disclosure generally relates to a computer implementation method for providing a user with guidance on where to place an object of interest in an operating room. This disclosure also relates to a processing system and a computer program. Background Technology

[0002] In a clinical setting, various medical devices may be placed in the operating room in different configurations. These devices include: instrument trays, medical monitoring devices such as electroencephalographs or electrocardiographs, tracking devices (e.g., stereo cameras for optical tracking systems, electric field generators for electromagnetic tracking systems, tracking markers such as active or passive optical markers detectable by optical tracking systems, and sensors configured to detect electromagnetic fields emitted by the electric field generator), and operating tables.

[0003] Currently, user manuals for medical devices typically contain setup information about where and how to position the corresponding medical device in the operating room (e.g., in which orientation). Users therefore need to study all the user manuals and memorize the setup information for each medical device.

[0004] When numerous medical devices are positioned in an operating room, errors can occur due to users' misremembering of setup information. Consequently, medical devices may be placed in the operating room in an incorrect orientation. Correcting such misalignment can take additional time. More importantly, some medical personnel, such as surgeons, may not immediately recognize the incorrect orientation and may begin surgery with one or more medical devices not properly positioned in the operating room. This can hinder the surgical procedure and potentially negatively impact clinical outcomes.

[0005] The aforementioned issues not only arise when placing medical equipment in the operating room, but also apply equally to other items and objects that need to be arranged within the operating room. For example, the patient should be positioned in a certain posture within the operating room. This posture may depend on the surgical procedure the patient is undergoing. Summary of the Invention

[0006] A technique is needed to solve one or more of the above-mentioned problems or other problems.

[0007] According to a first aspect, a computer-implemented method is provided for providing a user with guidance on where to place an object of interest in an operating room. The method includes: obtaining scene data indicating the spatial orientation of at least one object placed in the operating room; obtaining first constraint data indicating one or more requirements for the spatial relationship between a first object of interest and the at least one object placed in the operating room; and determining at least one primary spatial portion based on the scene data and the first constraint data, the primary spatial portion being selected from: (a) a first spatial portion in which the first object of interest is recommended to be placed, the first spatial portion being larger than the first object of interest; and (b) a second spatial portion in which the placement of the first object of interest is not permitted. The method further includes: triggering a visual display of at least one primary spatial portion to guide the user on where to place the first object of interest in the operating room.

[0008] The first spatial portion can be a two-dimensional region or a three-dimensional volume. The same applies to the second, third, and fourth spatial portions described herein. The term "portion" can also be understood as "part" or "segment." At least one principal spatial portion may include only the first spatial portion, or it may include only the second spatial portion. At least one principal spatial portion may include both the first and second spatial portions.

[0009] The first spatial portion may include at least two first sub-parts, each first sub-part including at least one recommended position or pose (i.e., at least one of position and orientation) of the first object of interest. Each of the first sub-parts may be associated with a different degree of recommendation. Visualization of the first spatial portion may include a distinguishable visualization of each first sub-part. The sub-parts described herein may have the same dimensions as the corresponding spatial portion (e.g., a sub-part of a two-dimensional first spatial portion may also be two-dimensional, and a sub-part of a three-dimensional first spatial portion may also be three-dimensional).

[0010] The method may further include: obtaining second constraint data for each of at least one second object of interest. The second constraint data may indicate one or more requirements for the spatial relationship between the second object of interest and the first object of interest. The method may further include: determining at least one secondary spatial portion for each of at least one second object of interest based on the determined primary spatial portion and the second constraint data, the secondary spatial portion being selected from: (a) a third spatial portion, in which the second object of interest is recommended to be placed; and (b) a fourth spatial portion, in which the placement of the second object of interest is not permitted. The method may further include: triggering a visualization of the at least one secondary spatial portion for each of at least one second object of interest to guide the user regarding where to place the at least one second object of interest in the operating room.

[0011] The third space can be larger than the second object of interest.

[0012] The third space portion may include at least two third sub-parts. Each third sub-part may include at least one recommended location of the second object of interest. Each of the third sub-parts may be associated with a different degree of recommendation. The visualization of the third space portion may include a distinguishable visualization of each third sub-part.

[0013] The method may further include: obtaining first alignment data indicating the spatial relationship between a first object of interest arranged in the operating room and at least one item. The method may further include: based on second constraint data and the first alignment data, determining an update instance of at least one secondary spatial portion for each of the at least one second object of interest. The method may further include: based on the update instance of the at least one secondary spatial portion, updating the visualization of the at least one secondary spatial portion for each of the at least one second object of interest.

[0014] An updated instance of at least one secondary spatial portion can be determined in response to (e.g., in response to or triggered by) first alignment data indicating that a first object of interest is arranged within a first spatial portion.

[0015] The method may further include: in response to first alignment data indicating that a first object of interest is arranged within a first spatial portion, triggering the display of an instruction for arranging one or more of at least one second object of interest within a determined third spatial portion for at least one or more of at least one second object of interest.

[0016] The method may further include: obtaining second alignment data indicating the spatial relationship between at least one primary spatial portion and one or more of at least one second object of interest arranged in the operating room. The method may further include: determining an update instance of the at least one primary spatial portion based on the second constraint data and the second alignment data. The method may further include: updating the visualization of the at least one primary spatial portion based on the update instance of the at least one primary spatial portion.

[0017] An updated instance of at least one main spatial portion can be determined in response to second alignment data indicating that one or more of at least one second object of interest are arranged within a third spatial portion determined for one or more of at least one second object of interest.

[0018] The method may further include: obtaining third constraint data for at least one of the at least two objects of interest. The third constraint data may indicate one or more requirements for the spatial relationship between the at least one of the at least two objects of interest and one or more other objects of interest. At least one secondary spatial portion of the at least one of the at least two objects of interest may be determined based on the third constraint data.

[0019] The method may further include: obtaining third alignment data indicating a spatial relationship between (i) at least one secondary spatial portion of at least one of the second objects of interest and (ii) one or more other second objects of interest arranged in the operating room. The method may further include: determining an updated instance of at least one secondary spatial portion of at least one of the second objects of interest based on the third constraint data and the third alignment data. The method may further include: updating the visualization of at least one secondary spatial portion of at least one of the second objects of interest based on the updated instance of the at least one secondary spatial portion.

[0020] The method may further include: triggering the display of an instruction for arranging a first object of interest in a first spatial portion. This instruction may be displayed in response to obtaining scene data.

[0021] For example, at least one of these requirements satisfies at least one of the following requirements: it is patient-specific; it is associated with the type of surgery selected by the user; it is associated with the surgical procedure selected by the user.

[0022] At least one article may include or be composed of a part of the patient's body. The first object of interest may be a medical device. Alternatively, the first object of interest may include a part of the patient's body. In this case, at least one article may be a medical device. As yet another alternative, each of the at least one article, the first object of interest, and at least one second object of interest may be a medical device.

[0023] In one particular variant, one or more of the following conditions may be met: (i) (e.g., stationary) medical devices may be configured to be positioned in a static or fixed location within an operating room; (ii) the medical device may include a clinical tracking system or a component of a clinical tracking system, such as an electromagnetic field generator for an electromagnetic tracking system; (iii) the medical device may be configured to be spatially fixed relative to the patient's body; (iv) the medical device may include or be configured to be a tracking marker tracked by a surgical tracking system, such as an electromagnetic sensor configured to be tracked by an electromagnetic tracking system or an optical tracking marker configured to be tracked by an optical tracking system; (v) the medical device may not be a handheld device, wherein a handheld device may include a pointer, drill, screwdriver, or navigation surgical instrument.

[0024] Visualizations can be triggered to be overlaid on at least a portion of the view of the operating room.

[0025] Visualizations can be triggered to be displayed on the display of an augmented reality device (ARD) (e.g., a head-mounted display (HMD)).

[0026] The HMD may include at least one sensor selected from optical sensors and depth sensors. The method may further include: acquiring sensor data captured by the at least one sensor; detecting at least one component selected from a first object of interest and at least one item based on the sensor data; and determining visualization based on the spatial relationship between the detected at least one component and the HMD.

[0027] According to a second aspect, a processing system is provided. The processing system includes at least one processor configured to perform the method according to the first aspect. For example, (e.g., one or more) the at least one processor is configured to: obtain a spatial pose of at least one item arranged in an operating room; obtain one or more requirements for a spatial relationship between a first object of interest and the at least one item arranged in the operating room; and use the obtained spatial pose and the obtained one or more requirements to trigger the display of a visualization of at least one main spatial portion to guide a user regarding where to arrange the first object of interest in the operating room, the at least one main spatial portion being selected from: (a) a first spatial portion where the first object of interest is recommended to be arranged, the first spatial portion being larger than the first object of interest; and (b) a second spatial portion where the first object of interest is not permitted to be arranged. Another of the at least one processor (e.g., one or more) may be configured to: obtain sensor data captured by at least one sensor selected from optical sensors and depth sensors of the HMD; detect at least one component selected from the first object of interest and at least one item based on the sensor data; determine a spatial relationship between the detected at least one component and the HMD; and optionally, determine a visualization based on the spatial relationship between the detected at least one component and the HMD.

[0028] According to a third aspect, a computer program is provided. The computer program includes instructions that, when executed by at least one processor (e.g., a processor of the processing device according to the second aspect), cause the at least one processor to perform the method according to the first aspect.

[0029] According to the fourth aspect, a data carrier is provided. The data carried is a computer program as described in the third aspect. The data carrier may be a computer-readable storage medium or a data carrier signal.

[0030] According to a fifth aspect, a surgical system is provided. The surgical system includes the processing system described in the second aspect. The surgical system may include a head-mounted display (HMD) including at least one sensor selected from optical sensors and depth sensors, wherein the at least one sensor is configured to capture sensor data enabling the detection of at least one component selected from a first object of interest and at least one article described herein. The surgical system may further include a display unit configured to display at least one piece of information selected from the visualizations and instructions described herein. The display unit may be configured to display at least one piece of information in an augmented manner. The display unit may be an augmented reality display unit. The display unit may be part of an augmented reality display (ARD) described herein. The surgical system may include an ARD. The display unit may be included in an HMD included in the surgical system. The surgical system may further include at least one component selected from: (i) the at least one article described above; (ii) the first object of interest described above; and (iii) one or more of the at least one second object of interest described above. As an alternative or addition to the ARD or HMD, the surgical system may include a tracking system configured to (e.g., in a tracking coordinate system) acquire or track the pose of at least one component. The processing system may access information indicating the relative pose between the ARD and the tracking coordinate system. Attached Figure Description

[0031] Other details, advantages, and aspects of this disclosure will become apparent from the following embodiments, taken in conjunction with the accompanying drawings, in which:

[0032] Figure 1 A surgical system according to this disclosure is shown;

[0033] Figure 2 The method according to this disclosure is shown;

[0034] Figure 3 An exemplary display of content prior to the placement of an object, according to this disclosure, is illustrated schematically;

[0035] Figure 4 An exemplary display content after only placing the first object is shown in accordance with this disclosure;

[0036] Figure 5 An exemplary display content after the placement of the first and second objects is illustrated schematically according to this disclosure;

[0037] Figure 6 An exemplary display content after only placing the first object is shown in accordance with this disclosure;

[0038] Figure 7 An exemplary display of content prior to the placement of an object, according to this disclosure, is illustrated schematically;

[0039] Figure 8 The illustration schematically shows the requirements and uses of this disclosure; and

[0040] Figure 9 The requirements and uses of this disclosure are illustrated schematically. Detailed Implementation

[0041] In the following description, exemplary embodiments of the methods, processing systems, surgical navigation systems, and computer programs will be illustrated with reference to the accompanying drawings. The same reference numerals will be used to denote the same or similar structural or functional features.

[0042] Figure 1 A surgical system 100 according to the present disclosure is shown. The surgical system 100 includes a processing system 200. The processing system 200 includes at least one processor 202, at least one memory 204, and at least one interface 206. The processor 202 is connected to the memory 204 and the interface 206. The memory 204 stores instructions, for example, in the form of a computer program. The instructions can configure the processor 202 to perform the methods described herein. Note that the methods described herein can be performed by multiple processors, which may or may not be arranged in different locations.

[0043] The surgical system 100 also includes an augmented reality device (ARD) 300, which includes a display unit 302 and a sensor 304. The ARD 300 can be configured as a head-mounted display (HMD), such as augmented reality glasses. The ARD 300 may include one or more processors (…). Figure 1 (not shown in the text), configured to perform the methods described herein (e.g., at least a portion thereof).

[0044] Display unit 302 can be configured to display information in an enhanced manner, such as superimposed on a view of the operating room where the ARD 300 is located. Display unit 302 can be configured as a head-up display or a holographic display to display information on an optically transparent component. Alternatively, display unit 302 can be configured to display video or still images captured by the ARD 300's camera and display information superimposed thereon. Display unit 302 can display the information, for example, upon receiving a corresponding instruction (e.g., including: the content to be displayed; and optionally, a trigger instruction indicating when to display the content) from processing system 200 (e.g., from processor 202) via interface 206.

[0045] Note that display unit 302 may be part of a different component, i.e., it may not be included in ARD 300. For example, display unit 302 may be the display screen of a surgical navigation system or a separate display. Unless otherwise indicated, in the context of references to displayed information (e.g., visualizations or instructions) herein, this will be interpreted as relating to the display on display unit 302, regardless of whether display unit 302 is part of ARD 300.

[0046] In some variations, sensor 304 includes a depth sensor (e.g., a time-of-flight sensor, a stereo camera, or a lidar sensor). Alternatively or additionally, sensor 304 includes an optical sensor (e.g., a camera or a stereo camera). Sensor 304 is configured to acquire sensor data representing the surroundings or environment of ARD 300 in a coordinate system 306 of sensor 304. In the case where ARD 300 is located in an operating room in a hospital, the sensor data represents at least a portion of the operating room, such as its spatial information. Sensor data can be transmitted from ARD 300 to processing system 200, for example, via at least one interface 206 (e.g., wirelessly).

[0047] The surgical system 100 also includes a tracking system 400. The tracking system 400 in the illustrated example includes a locator 402, an electromagnetic field generator 404, and an electromagnetic sensor 406. The locator 402 is configured to determine the attitude (i.e., at least one of position and orientation) of the sensor 406 in the tracking coordinate system 408. Figure 1 In the example shown, the electromagnetic sensor 406 is attached (e.g., removably fastened, glued, or pasted) to the head 2 of the patient 500.

[0048] Additional electromagnetic sensors may also be provided. For example, surgical instrument 500 may include additional electromagnetic sensors so that locator 402 can determine the orientation of surgical instrument 500 in tracking coordinate system 408. In the illustrated example, surgical instrument 500 includes a gripping portion 502 and an elongated element 504 attached to the gripping portion 502. The elongated element 504 extends longitudinally along instrument axis 506. Surgical instrument 500 may be a guiding instrument for inserting a biopsy needle along instrument axis 506 into a planned insertion point 4 on the patient's head 2.

[0049] Tracking system 400 may alternatively be an optical tracking system. In this case, tracking system 400 may include a stereo camera configured to track one or more active or passive optical tracking markers. Surgical system 100 may include both electromagnetic tracking system and optical tracking system.

[0050] Figure 1The diagram also schematically illustrates patient image data 600. Patient image data 600 may include medical images of the patient's head 2 acquired preoperatively or intraoperatively. These images may include computed tomography (CT) scans, CT images, or magnetic resonance imaging (MRI) images. Patient image data 600 is associated with an image coordinate system 602.

[0051] Using known registration techniques, transformations between two or more coordinate systems 306, 408, and 602 can be determined. For example, an instrument 500, tracked in coordinate system 408 by locator 402, can acquire several points on the surface of the patient's head 2. These points can be matched to a surface model of the patient's head 2, as described in or detected in patient image data 600. This results in a transformation between coordinate systems 408 and 602. As another example, ARD 300 may include an electromagnetic sensor that can be tracked by locator 402. Alternatively, ARD 300 may be provided with an optical tracking marker that can be tracked by an optical tracking system. This allows the transformation between coordinate systems 306 and 408 to be determined. Other variations in determining the transformation between coordinate systems 306, 408, and 602 are also possible and will be apparent to those skilled in the art. Whenever spatial information (e.g., the pose or spatial relationship between two components) is described in one of coordinate systems 306, 408, or 602, these transformations can be used to transform it to one or more other coordinate systems (e.g., for comparison with another set of spatial information).

[0052] When one of the components of surgical system 100 is mentioned below, it is not necessarily required to provide one or more, or all other, components of surgical system 100. For example, display unit 302 may be referenced below. Then it may not be necessary to provide tracking system 400, patient image data 600, or surgical instrument 500. In other words, surgical system 100 does not necessarily need to include references. Figure 1 All components described. In one variant, the surgical system 100 includes a processing system 200 and a display unit 302.

[0053] Figure 2 A method according to this disclosure is illustrated. This method can be executed by processor 202 of processing system 200. A computer program stored in memory 204 can configure processor 202 to perform the method described herein when executed by processor 202. The method is not limited thereto and can be executed by another processor.

[0054] The method includes step S2: obtaining scene data indicating the spatial orientation of at least one item arranged in the operating room.

[0055] Scene data may be based on data acquired by sensor 304. Scene data may be based on data acquired from tracking system 400. Scene data may include spatial information about the operating room, including at least one part of an object. Spatial information may be associated with one or both of coordinate systems 306 and 408. Although other variations are possible, at least one object may include or consist of the patient's body or a part of the patient's body.

[0056] The method includes: obtaining first constraint data indicating one or more requirements for the spatial relationship between a first object of interest and at least one item arranged in the operating room.

[0057] The first constraint data may be specific to a first object of interest. The first constraint data may be specific to the type of surgery or surgical procedure selected by the user. One or more requirements may include at least one of the following: a minimum distance, a maximum distance, a minimum relative angle, and a maximum relative angle between the first object of interest and at least one item. One or more requirements may include at least one of the following: (i) a minimum distance, a maximum distance, a minimum relative angle, and a maximum relative angle between the first object of interest and (ii) a portion of at least one item, a point on at least one item, or a path about at least one item. A portion, point, or path may be defined by planning data (e.g., in coordinate system 602). A portion may be a tumor within the patient's body. A point may be a planned entry point on the patient's body (e.g., point 4). A path may be a planned trajectory, such as (e.g., a planned insertion trajectory of a biopsy needle or surgical drill). The method may include: obtaining planning data to determine one or more requirements for the spatial relationship between the first object of interest and the patient's body. Although other variations are possible, the first object of interest may be an electromagnetic field generator 408.

[0058] The method includes step S6: determining at least one main spatial portion based on scene data and first constraint data, the main spatial portion being selected from: (a) a first spatial portion, in which it is recommended to place a first object of interest, the first spatial portion being larger than the first object of interest; and (b) a second spatial portion, in which the placement of the first object of interest is not permitted.

[0059] Determining a first spatial portion may include: obtaining object data indicating a surface of first interest; determining multiple poses in which the first interest is recommended to be arranged, based on scene data and first constraint data; and determining virtual surfaces for each of the multiple poses based on the object data, wherein the first spatial portion surrounds all virtual surfaces. The multiple recommended poses may satisfy at least one of the requirements indicated by the first constraint data. Similarly, determining a second spatial portion may include: obtaining object data indicating a surface of first interest (e.g., if object data for determining the first spatial portion has not yet been obtained); determining multiple poses in which the arrangement of the first interest is not permitted, based on scene data and first constraint data; and determining virtual surfaces for each of the multiple poses based on the object data, wherein the second spatial portion surrounds all virtual surfaces.

[0060] Alternatively or additionally, the method may include: once one of the first and second portions has been determined, using an inversion of the determined portion (e.g., all spatial portions that are not part of the determined portion) as the other of the first and second portions. Note that in one variation, the first spatial portion does not necessarily need to be larger than the first object of interest. For example, the first spatial portion may consist of only a virtual surface (e.g., if only one pose in which the first object of interest is recommended to be placed exists). The method includes step S8: triggering the display of visualization of at least one main spatial portion to guide the user about where to place the first object of interest in the operating room.

[0061] The visualization of the first spatial section informs the user which area or volume of the operating room the first object of interest (POI) should be placed in. Conversely, the visualization of the second spatial section informs the user which area or volume of the operating room the POI will not be placed in. This allows the user to place the POI in recommended locations and orientations without having to remember (e.g., settings provided in the POI's user manual) the corresponding configuration information.

[0062] The spatial components described in this article can be two-dimensional or three-dimensional. Therefore, the visualization of each spatial component described in this article can be two-dimensional or three-dimensional.

[0063] The visualizations described herein can be displayed on display 302. The visualizations described herein can be displayed relative to (e.g., overlaid on) an acquired image or a view of the operating room (e.g., at least a portion thereof). The visualizations described herein can be displayed relative to (e.g., overlaid on) patient image data 600. In the latter case, the method may include acquiring patient image data 600. The method may also include: acquiring a transformation between coordinate system 602 and a coordinate system (e.g., coordinate system 306 or 408) in which the spatial pose of at least one object is indicated by scene data. The method may include: (e.g., using one or more transformations as described above) transforming at least one principal spatial portion to one or more of coordinate systems 306, 408, and 602.

[0064] Figure 3 An exemplary display content according to this disclosure is schematically illustrated. It can be seen that the display content includes the patient's head 2 and the planned entry point 4. In this case, the items arranged in the operating room correspond to the patient's head 2. The display content also includes visualizations of a first spatial portion 6 and a second spatial portion 8. The first spatial portion 6 extends around the patient's head 2 in the cranial and lateral regions or volumes. In this example, the first spatial portion represents the volume in which the field generator 404 will be arranged relative to the patient's head 2. In other words, the first object of interest in this example is the field generator 404. The second spatial portion 8 extends close to the patient's head 2 in the tail volume. The second spatial portion 8 informs the user about the volume in which the field generator 404 is not permitted to be arranged. It can be seen that portion 8 does not occupy the entire volume except for portion 6. In other words, there is a volume that is not visualized or visually not highlighted. This may be because the field generator can be arranged in other postures outside portion 6, although this is not recommended. When presented... Figure 3 When the view shown is presented, the user is informed in an easy-to-understand manner where and how to arrange the field generator 404.

[0065] The method may further include (e.g., in response to determining a first spatial portion) triggering the display of instructions for arranging a first object of interest in the first spatial portion. The instructions may include an identifier of the first object of interest (e.g., a name, identification number, or an image representation thereof) to inform the user which object is to be placed in the main spatial portion. The instructions described herein may be displayed on the same display as the visualization, such as display unit 302.

[0066] The method may include: obtaining first alignment data indicating the spatial relationship between a first object of interest arranged in an operating room and at least one item. The instruction may be determined based on the first alignment data. The instruction may indicate whether or to what extent at least one of one or more requirements of the first constraint data and the current pose of the at least one first object of interest are satisfied. In a variation, instead of steps S6 and S8, the method may include: obtaining the first alignment data and determining an instruction based on the first alignment data (e.g., the instruction may indicate whether or to what extent at least one of one or more requirements of the first constraint data and the current pose of the first object of interest are satisfied). The method may then further include triggering the display of the instruction.

[0067] To provide users with further guidance for improvement, the different sub-sections of the identified spatial portion can be ranked or prioritized. This ranking or prioritization can also be visualized. For example, a first spatial portion may include at least two first sub-sections. Each first sub-section includes at least one recommended location for a first object of interest, and each first sub-section is associated with a different degree of recommendation (e.g., a different degree of prioritization). The visualization of the first spatial portion can then include a distinguishable visualization of each first sub-section. For example, the first sub-sections can be visualized differently using different colors or opacities.

[0068] When the first object is placed in a corresponding subsection, the more requirements are met (e.g., as indicated by the first constraint data), the higher the recommendation level of the subsection. Alternatively or additionally, the requirements indicated by the first constraint data can be associated with corresponding priority values, wherein the higher the priority value of the met requirements (e.g., when the first object is placed in a corresponding subsection), the higher the recommendation level of the subsection.

[0069] Refer again Figure 3 The first spatial portion 6 includes sub-parts 10 and 12. Sub-part 12 includes poses of the first object of interest that meet more requirements than all poses of the first object of interest included in sub-part 10. In other words, sub-part 12 is prioritized (e.g., because placing the field generator 404 in the first sub-part 12 enables the most accurate tracking of the biopsy needle at the insertion point 4).

[0070] The method may include: for each of at least one second object of interest, obtaining second constraint data indicating one or more requirements for the spatial relationship between the second object of interest and the first object of interest.

[0071] The second constraint data may be specific to at least one of the following: a first object of interest and at least one second object of interest. The second constraint data may be specific to the type of surgery or surgical procedure selected by the user. One or more requirements may include at least one of the following: minimum distance, maximum distance, minimum relative angle, and maximum relative angle between at least one second object of interest and the first object of interest. At least one second object of interest may include a patient tracker (e.g., an electromagnetic sensor 406 or an optical tracking marker attached to the patient's body).

[0072] The method may further include: based on the determined primary spatial portion and second constraint data, for each of at least one second object of interest, determining at least one secondary spatial portion, the secondary spatial portion being selected from: (a) a third spatial portion, wherein it is recommended to arrange the second object of interest; and (b) a fourth spatial portion, wherein it is not allowed to arrange the second object of interest.

[0073] Determining a third spatial portion of the second object of interest may include: obtaining object data indicating a surface of the second object of interest; determining multiple poses in which the second object of interest is recommended to be arranged, based on the determined principal spatial portion and second constraint data; and determining virtual surfaces for each of the multiple poses based on the object data, wherein the third spatial portion surrounds all virtual surfaces. The multiple recommended poses may satisfy at least one of the requirements indicated by the second constraint data. Similarly, determining a fourth spatial portion of the second object of interest may include: obtaining object data indicating a surface of the second object of interest (e.g., if object data for determining the third spatial portion has not yet been obtained); determining multiple poses in which the second object of interest is not permitted to be arranged, based on the determined principal spatial portion and second constraint data; and determining virtual surfaces for each of the multiple poses based on the object data, wherein the fourth spatial portion surrounds all virtual surfaces. Alternatively or additionally, the method may include: once one of the third and fourth portions has been determined, using an inversion of the determined portion (e.g., all spatial portions that are not part of the determined portion) as the other of the third and fourth portions. The third spatial portion may be larger than the second object of interest. Note that in one variation, the third spatial portion does not necessarily need to be larger than the second object of interest. For example, the third space portion may consist of only a virtual surface (e.g., if only one pose in which a second object of interest is recommended to be arranged exists).

[0074] In one variation, the second constraint data also indicates one or more requirements for the spatial relationship between the second object of interest and at least one item arranged in the operating room. The secondary spatial portion can then be further determined based on scene data. It can be said that the secondary spatial portion in this variation depends not only indirectly on the pose of the item by depending on the determined primary spatial portion (e.g., the first spatial portion), but also directly on the pose of at least one item.

[0075] As described above, the method may include: for each of at least one second object of interest, triggering the display of a visualization of at least one secondary spatial portion to guide the user regarding where to place the at least one second object of interest in the operating room.

[0076] Refer again Figure 3 The visualization of the third spatial portion is indicated by reference numeral 14, and the visualization of the fourth spatial portion is indicated by reference numeral 16. In this example, the second object of interest is sensor 406, which will be attached to the patient's head 2 before the start of the surgical procedure (e.g., before the biopsy needle insertion point 4). It can be seen that the third spatial portion 14 and the fourth spatial portion 16 are each different from the first spatial portion 6 and the second spatial portion 8. This is only a favorable example, and the third spatial portion 14 and the fourth spatial portion 16 may very well overlap with one or both of the first spatial portion 6 and the second spatial portion 8.

[0077] Similar to the first spatial portion, the different sub-parts of the third spatial portion can be prioritized differently. The third spatial portion can therefore include at least two third sub-parts, each including at least one recommended location for a second object of interest, wherein each third sub-part is associated with a different degree of recommendation. The visualization of the third spatial portion can include a distinguishable visualization of each third sub-part.

[0078] When the first object is placed in a corresponding subsection, the more requirements are met (e.g., as indicated by the second constraint data), the higher the recommendation level of the subsection. Alternatively or additionally, the requirements indicated by the second constraint data can be associated with corresponding priority values, wherein the higher the priority value of the met requirements (e.g., when the first object is placed in a corresponding subsection), the higher the recommendation level of the subsection.

[0079] Refer again Figure 3The third spatial portion 14 includes sub-parts 18 and 20 that are visualized in different ways. Sub-part 20 has a higher priority than sub-part 18 because when sensor 406 is arranged in sub-part 20, more requirements or higher priority requirements for the second constraint data are met compared to when sensor 406 is arranged in sub-part 18.

[0080] The visualization can be updated accordingly when the first or second object of interest is placed.

[0081] As described above, the method may include: obtaining first alignment data indicating the spatial relationship between a first object of interest arranged in the operating room and at least one item.

[0082] The first alignment data can be obtained from at least one of the ARD 300 and the tracking system 400. The first alignment data can indicate the relative spatial relationship (e.g., relative pose) between a first object of interest and at least one item captured by sensor 304 or locator 402. For example, the ARD 300 can capture depth images of the patient's head 2 and the field generator 404. The first alignment data can then be determined based on the depth images by detecting both the head 2 and the field generator 303 in the depth images and (e.g., in coordinate system 306) determining their spatial relationship. Other variations are possible. For example, the relative pose between the field generator 404 and the head 2 can be captured by an optical tracking system (e.g., by locating optical tracking markers attached to the head 2 and optical tracking markers attached to the field generator 404).

[0083] The method may include: determining an update instance of at least one secondary spatial portion for each of at least one second object of interest, based on second constraint data and first alignment data. The update instance of the at least one secondary spatial portion may be determined in a manner similar to that described above for the at least one secondary spatial portion, but using the pose of at least one item instead of the primary spatial portion during determination. As a result, in most cases, this will lead to a smaller update of the secondary spatial portion (e.g., using only the current pose of the item as a reference to the requirements of the second constraint data) compared to the secondary spatial portion (e.g., using all poses within the primary spatial portion as a reference to the requirements of the second constraint data). Therefore, it can be said that the placement of the first object of interest affects the third spatial portion (e.g., and the fourth spatial portion).

[0084] The method may include: updating the display of the visualization of at least one secondary spatial portion based on (e.g., by triggering a display) an update instance of at least one secondary spatial portion, for each of at least one second object of interest.

[0085] Figure 4 schematically shown Figure 3 The display content is shown after the field generator 404 has been placed in the first spatial portion 6. In this case, the field generator 404 is placed in the optimal manner, i.e., in the highest priority sub-portion 12. Although not required, visualization of the main spatial portions (e.g., visualization of the first spatial portion 10 and / or the second spatial portion 12) is still possible even though the first object of interest is placed in the first spatial portion 10. In the given example, only the first spatial portion 10 is still visualized, while the second spatial portion 8 is no longer displayed. It can also be seen that by placing the field generator 404 in the first spatial portion 10, the third spatial portion 14 is updated accordingly. That is, instead of determining all recommended poses of the sensor 406 relative to all potential poses of the field generator 404 included in the first spatial portion 10, the updated instance of the third spatial portion 14 is determined only using the actual pose of the field generator 404. Therefore, with Figure 3 Compared to the third space portion 14 shown, Figure 4 The updated third space portion 14 shown is smaller.

[0086] An update instance of at least one secondary spatial portion can be determined in response to first alignment data indicating that a first object of interest is arranged within a first spatial portion, or as a reaction to that first alignment data. Similarly, this response determination can avoid unnecessary recalculation.

[0087] The display of instructions for arranging one or more of at least one second object of interest in a third spatial portion can be triggered in response to first alignment data indicating that a first object of interest is arranged within a first spatial portion. Alternatively, the display of the instructions can be triggered when the third spatial portion is determined.

[0088] The instruction may include identifying at least one second object of interest. In the case of multiple second objects of interest, sequence data may be obtained. The sequence data may indicate a recommended order of placement for the multiple second objects of interest. The recommended order may be associated with a surgical type or procedure selected by the user. The recommended order may be patient-specific. The recommended order may be specific to at least one component selected from the item and the first object of interest. The instruction may determine and include the identifier of the second object of interest recommended to be placed next based on the sequence data.

[0089] The method may include: obtaining second alignment data indicating the spatial relationship between at least one principal spatial portion and one or more of at least one second object of interest arranged in an operating room. The instruction may be determined based on the second alignment data. The instruction may indicate whether or to what extent at least one requirement of the second constraint data and the current pose of the at least one second object of interest is satisfied. The method may include: obtaining first alignment data and determining an instruction based on the second alignment data (e.g., the instruction may indicate whether or to what extent at least one requirement of the second constraint data and the current pose of the at least one second object of interest is satisfied). The method may then further include triggering the display of the instruction.

[0090] As mentioned above, the placement of the first object of interest may affect the secondary spatial portion. In a similar manner, the placement of the second object of interest may affect the primary spatial portion.

[0091] The method may include: determining an update instance of at least one major spatial portion based on second constraint data and second alignment data.

[0092] The second alignment data can be obtained from at least one of the ARD 300 and the tracking system 400. The second alignment data can indicate the relative spatial relationship (e.g., relative pose) between the second object of interest and at least one of the first object and at least one item captured by the sensor 304 or the locator 402. For example, the ARD 300 can capture depth images of the field generator 404 and the sensor 406. The first alignment data can then be determined based on the depth images by detecting both the field generator 404 and the sensor 406 in the depth images and determining their spatial relationship (e.g., in coordinate system 306). Other variations are possible. For example, the relative pose between the field generator 404 and the sensor 406 can be captured by the tracking system 400 (e.g., by the locator 402).

[0093] The method may include updating the display of a visualization of at least one primary spatial portion based on an update instance of the at least one primary spatial portion (e.g., by triggering a display). In other words, the displayed primary spatial portion can respond to the placement of at least one second object of interest. Simultaneously or subsequently, for example, in response to determining an update instance of the primary spatial portion, secondary primary spatial portions may also be updated (e.g., again). As described above, after placing both the first and second objects of interest, it may not be necessary to display visualizations of the primary or secondary spatial portions. However, displaying these visualizations allows the user to move and reposition the first and second objects of interest within the respective first and third spatial portions to optimize their placement (e.g., by arranging each of the objects of interest in a higher-ranking sub-portion).

[0094] Figure 5 It shows the relationship with Figure 4 The displayed content corresponds to the updated displayed content, but only after the second object of interest has been placed. In this example, the second object of interest corresponds to sensor 406. Figure 4 In comparison, it can be seen that the first spatial portion 6 has been reduced in size (e.g., volume). This is because... Figure 5 The first spatial portion 6 has been updated by comparing the requirements indicated by the second constraint data with the capture posture of the patient tracker 406. Figure 4 In contrast, the size of the third space portion 14 has been reduced because the third space portion 14 is updated based on an updated instance of the main space portion (e.g., based on an updated, smaller first space portion 6).

[0095] Instructions indicating whether or to what extent a requirement of at least one of the second constraint data and the current pose of the at least one second object of interest (e.g., and the current pose of the first object of interest) are satisfied can be visualized.

[0096] To avoid unnecessary recalculation of the main spatial portion, an updated instance of at least one main spatial portion can be determined in response to second alignment data indicating that one or more of at least one second object of interest are arranged within a third spatial portion determined for one or more of at least one second object of interest.

[0097] and Figures 3 to 5 The difference lies in the fact that the first object of interest can correspond to a patient tracker, such as sensor 406. The second object of interest can then correspond to field generator 404. Furthermore, in this case, the initial display content before placing the first or second object of interest can correspond to... Figure 3The display content will be different, and the reference numerals in the main spatial regions 6, 8, 10, and 12 and the secondary spatial regions 14, 16, 18, and 20 will need to be interchanged. When the sensor 406 is placed in the first spatial region 6, the third spatial region 14 can be updated. Figure 6 The resulting display content is illustrated exemplarily. In the case where sensor 406 is the second object of interest and is placed within the third spatial portion before field generator 404 is placed in the first spatial portion, the resulting display content can correspond to... Figure 6 The contents shown are as indicated, and reference numerals 14 and 6 in the attached figures will need to be interchanged.

[0098] Note that this disclosure is not limited to a single secondary object of interest. It can guide the user on where to place multiple secondary objects of interest. For each of the secondary objects of interest, a separate instance of the secondary spatial portion, along with optional separate instructions, can be identified and visualized.

[0099] like Figure 7 As shown, the displayed content may include visualizations of multiple third-space portions 14 and 22. Figure 7 In the example, the first spatial region 6 is associated with the field generator 404 (e.g., the field generator 202 will be placed within the first spatial region 6), the third spatial region 14 is associated with the sensor 406, and another third spatial region 22 is associated with another second object of interest (e.g., a surgical instrument 500 configured as a biopsy needle guide).

[0100] Individual secondary spatial portions (e.g., instances thereof), such as individual third spatial portions, may be affected by the placement of one or more second objects of interest. It can be said that secondary spatial portions influence each other.

[0101] The method may further include: obtaining third constraint data for at least one of the second objects of interest (e.g., sensor 406), the third constraint data indicating one or more requirements for the spatial relationship between the at least one of the second objects of interest and one or more other than at least one of the second objects of interest (e.g., instrument 500), wherein at least one secondary spatial portion of the at least one second object of interest (e.g., a third spatial portion 14 associated with sensor 406) is further determined based on the third constraint data. The degree of recommendation of the third spatial portion of the at least one second object of interest may then alternatively or additionally depend on the third constraint data.

[0102] The third constraint data may be specific to one of at least one second object of interest. The third constraint data may be specific to one or more other second objects of interest. The third constraint data may be specific to the surgical type or procedure selected by the user. One or more requirements may include at least one of the following: minimum distance, maximum distance, minimum relative angle, and maximum relative angle between one of at least one second object of interest and one or more other second objects of interest.

[0103] The method may further include: obtaining third alignment data indicating the spatial relationship between (i) at least one secondary spatial portion of at least one second object of interest (e.g., sensor 406) and (ii) one or more other second objects of interest (e.g., instrument 500) arranged in the operating room. The third alignment data may describe the spatial relationship between a third spatial portion (e.g., third spatial portion 14) and one of the second objects of interest not associated with the third spatial portion (e.g., instrument 500 associated with third spatial portion 22).

[0104] The third alignment data can be obtained from the ARD 300 or the tracking system 400. The third alignment data can be obtained in a similar manner to the first and second alignment data described above.

[0105] Based on the third constraint data and the third alignment data, an updated instance of at least one secondary spatial portion (e.g., third spatial portion 14) of one of at least one second object of interest (e.g., sensor 406) can be determined. Based on the updated instance of at least one secondary spatial portion (e.g., third spatial portion 14), the visualization of at least one secondary spatial portion (e.g., third spatial portion 14) of one of at least one second object of interest (e.g., sensor 406) can be updated (e.g., in response to the third alignment data indicating that one or more other objects of interest among at least one second object of interest have been placed within the corresponding third spatial portion).

[0106] As described above, at least one article may include a part of the patient's body (e.g., head 2) or may correspond to the patient's body. Alternatively or additionally, at least one article may correspond to a medical device. The first object of interest may be a first medical device, and at least one second object of interest may include or consist of at least one second medical device. The corresponding medical device may be one of the following: a field generator (e.g., field generator 404), an electromagnetic sensor (e.g., sensor 406), a surgical instrument (e.g., instrument 500), a biopsy needle, a stereo camera of an optical tracking system, an optical tracking marker (e.g., to be) attached to the patient, another tracker, or an implant (e.g., a metal implant such as a surgical screw or bone plate).

[0107] While other variations are possible, this disclosure provides a solution in which a first object of interest should be placed before at least one second object of interest. Optionally, at least one of the at least two objects of interest should be placed before the second of the at least two objects of interest. This order can be conveyed to the user by displaying the aforementioned instructions. This order can be indicated by the sequence data as described above. Multiple consecutive instructions can be displayed, each informing the user which object should be placed next. In response to alignment data indicating that the current object of interest has been correctly placed, subsequent instructions among these instructions can be displayed.

[0108] In the first example, based on the above reference Figures 3 to 5 As described, the first object of interest is the field generator 404, and at least one second object of interest (e.g., its first or second) is a patient tracker (e.g., sensor 406 or an optical tracking tag attached to the patient's body).

[0109] In the second example, the first object of interest is a patient tracker (e.g., sensor 406), the first of at least one second object of interest is a surgical instrument 500 configured as a biopsy guide, and the second of at least one second object of interest is a field generator 404 or a stereo camera of the tracking system.

[0110] In the third example, the first object of interest is a surgical instrument 500 configured as a biopsy guide, the first of at least one second object of interest is a patient tracker (e.g., sensor 406), and the second of at least one second object of interest is a field generator 404 or a stereo camera of a tracking system.

[0111] In the fourth example, the first object of interest is the field generator 404 or the stereo camera of the tracking system, the first of at least one second object of interest is a patient tracker (e.g., sensor 406), and the second of at least one second object is a surgical instrument 500 configured as a biopsy guide.

[0112] In the fifth example, at least one item includes an operating table, and the first object of interest corresponds to the patient's body (e.g., head 2). In this example, the first of at least one second object of interest may be a field generator 404, and the second of at least one second object of interest may be a patient tracker.

[0113] Scene data can be obtained from the ARD 300 (e.g., from sensor 304). The method may include acquiring sensor data captured by at least one sensor 304. Based on the sensor data, at least one component selected from a first object of interest and at least one item can be detected (e.g., using one or more object recognition algorithms). The pose of at least one component relative to sensor 304 (e.g., pose in coordinate system 306) can be determined after at least one component has been detected. In other words, the spatial relationship between the detected at least one component and the ARD 300 (e.g., configured as an HMD) can be determined. The method may include determining visualization based on the spatial relationship between the detected at least one component and the ARD 300.

[0114] For example, sensor 304 can be used to detect the current pose of the patient's head 2 relative to the ARD 300 (e.g., the current pose in coordinate system 306). This can allow for the augmented overlay of primary and secondary spatial portions in a view including head 2. In this case, it may not be necessary to acquire tracking data from the tracking system, which allows for guidance on where and how to position the components of the tracking system (e.g., where to position field generator 404 and sensor 406) even before the tracking system is activated. After the field generator 404 and sensor 406 are placed, locator 402 (e.g., or processor 202, using data obtained from tracking system 400) can determine the relative pose between field generator 404 and sensor 406. This relative pose can be determined with high accuracy and used to update at least one of the primary and secondary spatial portions.

[0115] The techniques disclosed above will be described again in other terms below. Although the methods described above may include one or more features described below, the following paragraphs should not be considered as limitations on the methods described above.

[0116] There may be specific objects (e.g., hardware components) that need to be positioned relative to each other. For example, a tracking device (e.g., a field generator 404 or a stereo camera in an optical tracking system) may need to be positioned relative to the patient's body (e.g., head 2) so that the surgical site is within the optimal tracking volume of the tracking device. There may be certain constraints that need to be met to obtain optimal surgical performance and results, which stem from the type of surgery (e.g., ENT, cranial surgery, etc.) or specific surgical procedures (e.g., insertion of a biopsy needle or instrument along the access path).

[0117] This disclosure can be considered as providing a technique for transmitting these constraints to the user to achieve appropriate user actions.

[0118] Typical objects of interest to be placed are field generator 404 or stereo camera, patient tracker (e.g., sensor 406), or biopsy needle. Furthermore, typical constraints on object placement include the patient's body posture, the posture and type of the operating table, the posture and type of the system cart and monitor, and the posture and type of Mayfield clamps or other components that affect tracking quality.

[0119] In a basic surgical navigation setup, the tracking system (e.g., tracking system 400) can at least measure the orientation of a patient tracker (e.g., sensor 406) rigidly attached to the patient. The exact position of the patient relative to the patient tracker may be unknown during the positioning of navigation system components (e.g., field generator 404 and sensor 406) in the operating room prior to the commencement of the surgical procedure. The registration between patient image data 600 and tracking system 400 (resulting in transformations between coordinate systems 602, 408) may also be unknown. However, guidance for the optimal placement of the tracking system relative to the patient can be provided using the techniques disclosed herein.

[0120] The constraints disclosed in this article indicate one or more requirements that can be categorized into two main types: process information and operating room setup information.

[0121] Process information requirements can be given by the type of surgery and the surgical procedure, and optionally by the process plan (e.g., planning data). Process information requirements can be used to derive the optimal orientation or settings for tracking system components and other tracking-related components (e.g., the orientation of field generator 404). Process information requirements in Figure 8 The instructions may be based on or depend on one or more of the following: entry point 4 (e.g., its orientation), key structures (e.g., a vessel), overhanging areas, permitted tracking system areas in the operating room, target areas or points, existing screws, permitted patient tracker areas, etc. Process information requirements may be based on or depend on at least one of the following: (i) patient image data 600, (ii) planning data, (iii) the type of surgery selected by the user or indicated by the planning data (e.g., functional endoscopic sinus surgery (FESS), cochlear surgery, biopsy, etc.), and (iv) external information regarding the current operating room setup (e.g., spatial mapping or video images acquired, for example, by sensor 304).

[0122] Operating room setup information requirements can depend on: tracking data, object and item attributes, and physical models of the patient's body, objects, and items, etc. Operating room setup information requirements also... Figure 8The requirements may depend on one or more of the following: the measurement area of ​​the tracking system, the tracking characteristics of the tracking system components, the working area of ​​the components (e.g., surgical instruments, or at least one of a first object of interest, at least one second object of interest, and an article), the shape of the components, their physically possible configuration, interfering components that negatively affect tracking accuracy, etc. Operating room setup information requirements may depend on or be determined based on at least one of the following: (i) spatial mapping data, (ii) spatial understanding data, (iii) webcam or depth images acquired, for example, by sensor 304, (iv) marker-based and / or markerless tracking of components (e.g., objects or articles), (v) model information of the components (e.g., as STL mesh data), (vi) the geometric properties of the components, and (vii) tracking data from the tracking system (e.g., tracking system 400).

[0123] Figure 8 It also indicates that the "permitted area" can be combined with the "tracking area". These requirements define, in essence, the observation area or effective tracking area or volume of the tracker (e.g., a stereo camera or electromagnetic field generator) of the tracking system. These requirements may also indicate spatial requirements regarding the entry point and target size and attitude. The suggested location area ("suggested location area") may correspond to the primary spatial portion, first spatial portion, second spatial portion, secondary spatial portion, third spatial portion, and / or fourth spatial portion described above. Object placement can affect the suggested area ("refinement using the setup positions of other components"). Tracking data obtained from the tracking system can be used to determine the precise spatial attitude of one or more objects and update the spatial portions accordingly. Objects should be placed such that they do not interfere with tracking quality ("optimize tracking quality").

[0124] Based on one or more requirements, the method described herein can optimize the spatial relationships between the navigation system 400 and the involved components and recommend positioning regions weighted by their quality. During user placement of components (e.g., a first object of interest and at least one second object of interest), the recommended positioning regions (e.g., a first spatial portion and a third spatial portion) can be dynamically refined.

[0125] The components to be deployed (e.g., a first object of interest and at least one second object of interest) may include, but are not limited to, an operating table, a navigation system, a trolley, a mounting arm, a robot, imaging equipment, surgical tools or instruments, staff, and staff work areas.

[0126] Figure 9 This can be considered as a reference above. Figure 2 Examples and more detailed descriptions of the methods described. Figure 9The "visualization of the suggested location area of ​​the field generator" can correspond to the first spatial portion described herein. The "visualization of the suggested location area of ​​the patient tracker" can correspond to the third spatial portion described herein. "User interaction" can correspond to the placement or movement of the first object of interest (e.g., field generator 404) or the second object of interest (e.g., sensor 406). That is, the visualization can be updated not only when the object of interest is placed, but also in response to the movement of the object of interest.

[0127] Figure 9 The steps shown in the "constraint-based location optimization" can correspond to Figure 8 Steps with similar names. Figure 9 The "OR-Setup Information" can include operating room setup information requirements (e.g., as referenced above). Figure 8 (As described). Figure 9 The “process information” can include process information requirements (e.g., as referenced above). Figure 8 (As described). The combination of "OR-Setup Information" and "Process Information" can be considered to represent the first constraint data, the second constraint data, and the third constraint data.

[0128] The resulting operating room setup proposal can represent the three-dimensional arrangement of the tracking system components relative to the patient (e.g., the patient's head during ENT surgery) or the setup components. This arrangement can be displayed on a monitor of the navigation system (e.g., including tracking system 400) or on an ARD 300 (e.g., Microsoft). )superior

[0129] The ARD 300 (e.g., configured as a head-mounted display (HMD)) can directly visualize guidance information for optimal operating room setup in the real world. This guidance can be based on a pre-registration of the patient, field generator 404, and patient tracker 406 relative to the coordinate space 306 of the head-mounted display. This pre-registration can use a trained model for face and object detection in webcam images from sensor 304 and spatial mapping of the head-mounted display 300.

[0130] The proposed method does not necessarily utilize tracking data from the surgical navigation system. Therefore, it can provide guidance independently of the current state of the surgical navigation system and can help set up the navigation system itself (e.g., tracking system 400).

[0131] If additional sensor inputs from the operating room setup are available, the proposed method could allow for a holistic approach to component placement.

[0132] In potential applications, guided visualizations can be displayed on a head-mounted display. If the patient is tracked by the head-mounted display, the visualization can be shown directly at their actual location. In this case, the user may no longer need to convey information from abstract two-dimensional or three-dimensional visualizations.

[0133] Modifications to the methods, processing systems, surgical systems, and computer programs disclosed herein are possible. For example, operating room setup information requirements may be indicated by at least one of first constraint data, second constraint data, and third constraint data. As another example, process information requirements may be indicated by at least one of first constraint data, second constraint data, and third constraint data. In view of the foregoing, those skilled in the art will appreciate other technical effects and advantages.

Claims

1. A computer-implemented method for providing a user with guidance on where to position an object of interest in an operating room, the method comprising: Obtain scene data, the scene data indicating the spatial orientation of at least one item arranged in the operating room; Obtain first constraint data, the first constraint data indicating one or more requirements for the spatial relationship between a first object of interest and at least one item arranged in the operating room; Based on the scene data and the first constraint data, at least one main spatial portion is determined, the main spatial portion being selected from: (a) a first spatial portion, in which the first object of interest is recommended to be placed, the first spatial portion being larger than the first object of interest; and (b) a second spatial portion, in which the first object of interest is not allowed to be placed; For each of at least one second object of interest, second constraint data is obtained, the second constraint data indicating one or more requirements for the spatial relationship between the second object of interest and the first object of interest; Based on the determined primary spatial portion and the second constraint data, for each of the at least one second object of interest, at least one secondary spatial portion is determined, the secondary spatial portion being selected from: (a) a third spatial portion, wherein the second object of interest is recommended to be arranged; and (b) the fourth space portion, in which the second object of interest is not permitted to be placed; The visualization of at least one main spatial section is triggered to guide the user on where to place the first object of interest in the operating room; as well as For each of the at least one second object of interest, a visualization of the at least one secondary spatial portion is triggered to guide the user on where to place the at least one second object of interest in the operating room.

2. The method according to claim 1, wherein, The first spatial portion includes at least two first sub-parts, each first sub-part including at least one recommended location of the first object of interest, wherein each of the first sub-parts is associated with a different degree of recommendation, and wherein the visualization of the first spatial portion includes a distinguishable visualization of each first sub-part.

3. The method according to claim 1, wherein, The third spatial portion is larger than the second object of interest.

4. The method according to claim 3, wherein, The third spatial portion includes at least two third sub-parts, each third sub-part including at least one recommended location of the second object of interest, wherein each of the third sub-parts is associated with a different degree of recommendation, and wherein the visualization of the third spatial portion includes a distinguishable visualization of each third sub-part.

5. The method according to claim 3, further comprising: Obtain first alignment data, which indicates the spatial relationship between a first object of interest arranged in the operating room and the at least one item; Based on the second constraint data and the first alignment data, for each of the at least one second object of interest, an update instance of the at least one secondary spatial portion is determined; as well as Based on the update instance of the at least one secondary spatial portion, the visualization of the at least one secondary spatial portion is updated for each of the at least one second object of interest.

6. The method according to claim 5, wherein, In response to the first alignment data indicating that the first object of interest is arranged within the first spatial portion, the update instance of the at least one secondary spatial portion is determined.

7. The method according to claim 5, further comprising: In response to the first alignment data indicating that the first object of interest is arranged within the first spatial portion, the display of an instruction is triggered, the instruction being used to arrange one or more of the at least one second object of interest in a third spatial portion determined for one or more of the at least one second object of interest.

8. The method according to claim 3, further comprising: Obtain second alignment data, which indicates the spatial relationship between the at least one main spatial portion and one or more of the at least one second object of interest arranged in the operating room; The update instance of the at least one main spatial portion is determined based on the second constraint data and the second alignment data; as well as The visualization of the at least one main spatial portion is updated based on the updated instance of the at least one main spatial portion.

9. The method according to claim 8, wherein, The update instance of the at least one main spatial portion is determined in response to the second alignment data indicating that one or more of the at least one second objects of interest are arranged within the third spatial portion determined for one or more of the at least one second objects of interest.

10. The method of claim 3, further comprising: For one of the at least one second object of interest, third constraint data is obtained, the third constraint data indicating one or more requirements for the spatial relationship between one of the at least one second object of interest and one or more other second objects of interest. The at least one secondary spatial portion of one of the at least two second objects of interest is further determined based on the third constraint data.

11. The method of claim 10, further comprising: Obtain third alignment data, which indicates (i) the spatial relationship between at least one secondary spatial portion of one of the at least two second objects of interest arranged in the operating room and (ii) one or more of the other two second objects of interest; Based on the third constraint data and the third alignment data, an update instance of at least one secondary spatial portion of one of the at least one second object of interest is determined; as well as Based on the updated instance of the at least one secondary spatial portion, update the visualization of at least one secondary spatial portion of one of the at least one second objects of interest.

12. The method according to claim 1, further comprising: The display of instructions for arranging the first object of interest in the first spatial portion is triggered.

13. The method according to claim 1, wherein, At least one of the requirements satisfies at least one of the following requirements: - It is patient-specific; - It is associated with the type of surgery selected by the user; - It is associated with the surgical procedure selected by the user.

14. The method according to claim 1, wherein, The at least one item includes a part of the patient's body, and wherein the first object of interest is a medical device.

15. The method according to claim 1, wherein, The visualization is triggered to be overlaid on a view of at least a portion of the operating room.

16. The method according to claim 1, wherein, The visualization is triggered to be displayed on the head-mounted display (HMD).

17. The method according to claim 16, wherein, The HMD includes at least one sensor selected from optical sensors and depth sensors, and the method further includes: Obtain sensor data captured by the at least one sensor; Based on the sensor data, at least one component selected from the first object of interest and the at least one item is detected; Determine the spatial relationship between at least one detected component and the HMD; and The visualization is determined based on the spatial relationship between at least one detected component and the HMD.

18. A surgical system comprising at least one processor, said at least one processor being configured to: Obtain scene data, which indicates the spatial orientation of at least one item arranged in the operating room; Obtain first constraint data, the first constraint data indicating one or more requirements for the spatial relationship between a first object of interest and at least one item arranged in the operating room; Based on the scene data and the first constraint data, at least one main spatial portion is determined, the main spatial portion being selected from: (a) a first spatial portion, in which the first object of interest is recommended to be placed, the first spatial portion being larger than the first object of interest; and (b) a second spatial portion, in which the first object of interest is not allowed to be placed; For each of at least one second object of interest, second constraint data is obtained, the second constraint data indicating one or more requirements for the spatial relationship between the second object of interest and the first object of interest; Based on the determined primary spatial portion and the second constraint data, for each of the at least one second object of interest, at least one secondary spatial portion is determined, the secondary spatial portion being selected from: (a) a third spatial portion, wherein the second object of interest is recommended to be arranged; and (b) the fourth space portion, in which the second object of interest is not permitted to be placed; The visualization of at least one main spatial section is triggered to guide the user on where to place the first object of interest in the operating room; For each of the at least one second object of interest, a visualization of the at least one secondary spatial portion is triggered to guide the user on where to place the at least one second object of interest in the operating room.

19. The surgical system of claim 18, further comprising: A head-mounted display (HMD) includes at least one sensor selected from optical sensors and depth sensors, wherein the at least one sensor is configured to capture sensor data that enables the detection of at least one component selected from a first object of interest and the at least one article.

20. A computer-readable storage medium carrying a computer program, the computer program including instructions that, when executed by at least one processor, cause the at least one processor to: Obtain scene data, which indicates the spatial orientation of at least one item arranged in the operating room; Obtain first constraint data, the first constraint data indicating one or more requirements for the spatial relationship between a first object of interest and at least one item arranged in the operating room; Based on the scene data and the first constraint data, at least one main spatial portion is determined, the main spatial portion being selected from: (a) a first spatial portion, in which the first object of interest is recommended to be placed, the first spatial portion being larger than the first object of interest; and (b) a second spatial portion, in which the first object of interest is not allowed to be placed; For each of at least one second object of interest, second constraint data is obtained, the second constraint data indicating one or more requirements for the spatial relationship between the second object of interest and the first object of interest; Based on the determined primary spatial portion and the second constraint data, for each of the at least one second object of interest, at least one secondary spatial portion is determined, the secondary spatial portion being selected from: (a) a third spatial portion, wherein the second object of interest is recommended to be arranged; and (b) the fourth space portion, in which the second object of interest is not permitted to be placed; The visualization of at least one main spatial section is triggered to guide the user on where to place the first object of interest in the operating room; as well as For each of the at least one second object of interest, a visualization of the at least one secondary spatial portion is triggered to guide the user on where to place the at least one second object of interest in the operating room.

Images