MR device and surgical navigation system

Abstract

The invention discloses MR equipment and a surgical navigation system. The MR equipment comprises a sensing device, an imaging device and a control device, a storage device; the display device is used for overlapping the first anchoring view field and the second anchoring view field on a real scene for presentation; the processing device is connected with the storage device, the display device and the sensing device, and is used for calling and executing at least one program from the storage device: displaying a reference body 3D virtual model registered with a reference body and an observation body 3D virtual model registered with an observation body in a first anchoring view field; and displaying a guide window interface in a second anchoring field of view, a local two-dimensional view of the reference body 3D virtual model and a first geometric figure representing a projection distance between a first target position on the observation body 3D virtual model and a second target position on the reference body 3D virtual model are displayed in the guide window interface; the second geometric figure represents an operation direction included angle between the first target position and the second target position.

Description

Technical Field

[0001] The present application relates to the technical field of medical data processing, and in particular to an MR device and a surgical navigation system. Background Art

[0002] The surgical navigation system can assist doctors in performing surgery on patients by providing images of the surgical site during the operation, thereby improving the safety of the operation.

[0003] In the current surgical navigation system, doctors observe the position of the surgical site (e.g., lesion site) in a virtual model (e.g., a spine model, a brain model, etc.) displayed on a display device, and rely on their experience to adjust the posture of the surgical instrument so that the surgical instrument contacts the actual surgical site. However, it is difficult to ensure that the posture of the surgical instrument when contacting the actual surgical site meets the surgical requirements, resulting in long operation time and low surgical accuracy.

[0004] Therefore, how to provide users with intuitive guidance information during surgery to improve surgical efficiency and accuracy is a technical problem that needs to be solved urgently. Summary of the invention

[0005] In view of the shortcomings of the related technologies mentioned above, the purpose of the present application is to provide an MR device and a surgical navigation system to solve the technical problem of how to provide intuitive guidance information to users during surgery to improve surgical efficiency and accuracy.

[0006] To achieve the above-mentioned purpose and other related purposes, the present application provides a MR device in a first aspect, comprising: a sensing device for acquiring sensing data of an observation body and a reference body matched with the observation body; a storage device for storing at least one program; a display device for superimposing a first anchoring field of view and a second anchoring field of view on a real scene for presentation; a processing device connected to the storage device, the display device, and the sensing device, for calling and executing the at least one program from the storage device to realize: displaying a 3D virtual model of a reference body registered with a reference body and a 3D virtual model of a reference body registered with an observation body in the first anchoring field of view. A registered observation body 3D virtual model; and displaying a guide window interface in the second anchoring field of view, wherein the guide window interface displays a local two-dimensional view of the reference body 3D virtual model, a first geometric figure representing the projection distance between a first target position on the observation body 3D virtual model and a second target position on the reference body 3D virtual model, and a second geometric figure representing the operating direction angle between the first target position and the second target position; wherein the projection distance and the operating direction angle are determined by using the postures of the observation body and the reference body obtained based on the sensor data.

[0007] In certain embodiments disclosed in the first aspect of the present application, the processing device is further used to call the reference volume 3D virtual model and the observation volume 3D virtual model.

[0008] In certain embodiments disclosed in the first aspect of the present application, the processing device retrieves the reference body 3D virtual model and the observation body 3D virtual model from a database by identifying a two-dimensional code set on the reference body and / or the observation body.

[0009] In certain embodiments disclosed in the first aspect of the present application, the database is a local database, a local area network database, or a cloud database.

[0010] In certain embodiments disclosed in the first aspect of the present application, the reference body includes a medical device implanted in the surgical site or patient, and the observation body includes a surgical instrument.

[0011] In certain embodiments disclosed in the first aspect of the present application, the processing device is also used to display a selection window interface in the second anchoring field of view before displaying the guide window interface, so as to determine the first target position and the second target position based on the user's operation on the selection window interface.

[0012] In certain embodiments disclosed in the first aspect of the present application, the processing device displays the selection window interface according to a trigger gesture of the user.

[0013] In certain embodiments disclosed in the first aspect of the present application, the guide window interface also includes a control area for adjusting the second target position, so that the processing device adjusts the offset of the second target position based on the user's operation in the control area.

[0014] In certain embodiments disclosed in the first aspect of the present application, the local two-dimensional view is a local enlarged view of the two-dimensional projection of the reference body 3D virtual model corresponding to a projection surface.

[0015] In certain embodiments disclosed in the first aspect of the present application, the projection distance is the distance between the projection point of the first target position on the projection plane and the second target position, or the distance between the projection point of the first target position on the projection plane and the projection point of the second target position on the projection plane, and the operation direction angle is the angle between a first operation direction corresponding to the first target position and a second operation direction corresponding to the second target position.

[0016] In certain embodiments disclosed in the first aspect of the present application, the size of the first geometric figure is used to characterize the size of the projection distance, and the size of the second geometric figure is used to characterize the size of the operating direction angle.

[0017] In certain embodiments disclosed in the first aspect of the present application, the position of the first geometric figure in the local two-dimensional view of the reference body 3D virtual model represents the deviation direction of the projection point of the first target position relative to the second target position, and the position of the second geometric figure in the local two-dimensional view of the reference body 3D virtual model is used to characterize the deviation direction of the operating direction of the second target position relative to the operating direction of the first target position.

[0018] In certain embodiments disclosed in the first aspect of the present application, the first geometric figure is a circle, and the second geometric figure is a ray whose starting point falls on the circumference of the circle.

[0019] In certain embodiments disclosed in the first aspect of the present application, the circumference of the circle passes through the second target position on the local cross-sectional view and the projection point of the first target position, or passes through the projection point of the second target position and the projection point of the first target position, and the starting point of the ray passes through the second target position on the local cross-sectional view or the projection point of the second target position.

[0020] In certain embodiments disclosed in the first aspect of the present application, in the guide window interface, different colors of the first geometric figure are used to represent the projection distance within different distance ranges, and in the guide window interface, different colors of the second geometric figure are used to represent the operation direction angle within different angle ranges.

[0021] In certain embodiments disclosed in the first aspect of the present application, the processing device is further configured to synchronously display the first geometric figure and the second geometric figure on the 3D virtual model of the reference volume in the first anchor field of view.

[0022] In certain embodiments disclosed in the first aspect of the present application, the processing device is further configured to display a local two-dimensional view of the 3D virtual model of the observation body on the guide window interface when it is detected that the observation body passes through the second target position of the reference body.

[0023] A second aspect of the present application provides a surgical navigation system, comprising: an MR device as described in any embodiment disclosed in the first aspect of the present application and a computer device communicatively connected to the MR device.

[0024] In certain embodiments disclosed in the first aspect of the present application, the computer device includes: a storage device for storing at least one program; an interface device for communicatively connecting the MR device to receive an observation body and a reference body posture matched with the observation body sent by the MR device; a processing device connected to the storage device and the interface device, and used to call and execute the at least one program from the storage device to determine the projection distance and the operation direction angle between the first target position on the 3D virtual model of the observation body and the second target position on the 3D virtual model of the reference body based on the posture of the observation body and the reference body.

[0025] In certain embodiments disclosed in the first aspect of the present application, the storage device is further used to store at least one reference body 3D virtual model and an observation body 3D virtual model adapted to the reference body 3D virtual model, for the MR device to call.

[0026] In certain embodiments disclosed in the first aspect of the present application, the computer device further includes a display device, and the processing device is used to display the display data synchronously transmitted by the MR device and received by the interface device on the display device.

[0027] In certain embodiments disclosed in the first aspect of the present application, the surgical navigation system also includes a field of view sharing device communicatively connected to the computer device, and the field of view sharing device is used to synchronously display the reference volume 3D virtual model, the observation volume 3D virtual model, and the guide window interface.

[0028] In summary, the MR device and surgical navigation system provided by the present application display a reference body 3D virtual model aligned with a reference body and an observation body 3D virtual model aligned with an observation body in a first anchoring field of view, and display a local two-dimensional view of the reference body 3D virtual model, a first geometric figure representing the projection distance, and a second geometric figure representing the angle of the operation direction in a guide window interface displayed in a second anchoring field of view, and detect the actual operation process so that the first geometric figure and the second geometric figure can change according to the real-time operation, thereby providing the user with intuitive guidance information / intuitive visual navigation, so that the user can quickly contact the surgical instrument with the real surgical site in a posture that meets the requirements of preoperative preparation or intraoperative surgery, so as to improve the efficiency and accuracy of the operation; further, by synchronously displaying the first geometric figure and the second geometric figure in the reference body 3D virtual model, the user can observe the first geometric figure and the second geometric figure on the reference body 3D virtual model, which is convenient for hand-eye coordination during the operation, and can also observe the larger first geometric figure and the second geometric figure in the guide window interface when the first geometric figure and the second geometric figure in the reference body 3D virtual model are small, thereby improving the efficiency and accuracy of the operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The features and advantages of the inventions of the present application can be better understood by referring to the exemplary embodiments and drawings described in detail below. The drawings are briefly described as follows:

[0030] Figure 1 Shown is a schematic diagram of the structure of a surgical navigation system in one embodiment of the present application.

[0031] Figure 2 Shown is a schematic structural diagram of an MR device in one embodiment of the present application.

[0032] Figure 3 and Figure 4 They are schematic diagrams of the selection window interface when no target to be selected is selected and after the target to be selected is selected in one embodiment of the present application.

[0033] Figure 5 and Figure 6 Schematic diagrams showing projection distances in different embodiments of the present application are shown respectively.

[0034] Figure 7 Shown is a schematic diagram of the operating direction angle in one embodiment of the present application.

[0035] Figure 8 Shown is a schematic diagram of a guide window interface and a reference volume 3D virtual model and an observation volume 3D virtual model displayed in an embodiment of the present application.

[0036] Fig. 9 Show this application Figure 8 A schematic diagram of the first geometric figure after being changed in the illustrated embodiment.

[0037] Fig.10 Show this application Figure 8 A schematic diagram of the second geometric figure after the change in the illustrated embodiment.

[0038] Fig.11 Shown is a schematic diagram of a guide window interface and a reference volume 3D virtual model and an observation volume 3D virtual model displayed in another embodiment of the present application.

[0039] Fig.12 Shown is a schematic diagram of the structure of a computer device in one embodiment of the present application. DETAILED DESCRIPTION

[0040] The following is an explanation of the implementation of the present application by means of specific embodiments. People familiar with the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification.

[0041] The present application is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The technical solutions in the embodiments of the present application are clearly and completely described. Obviously, the described embodiments are only embodiments of a part of the present application, rather than all embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application. "An implementation method", "implementation method" or similar expressions mentioned in the whole text of this specification mean that the specific features, structures or characteristics described together with the implementation method are included in at least one embodiment of the present application. Therefore, in the whole text of this specification, the appearance of the phrases "in an implementation method", "in an implementation method" and similar expressions may (but not necessarily) relate to the same implementation method.

[0042] Although the term first, second etc. is used to describe various elements or parameters in this article in some instances, these elements or parameters should not be limited by these terms.These terms are only used to distinguish an element or parameter from another element or parameter.For example, the first geometric figure can be referred to as the second geometric figure, and similarly, the second geometric figure can be referred to as the first geometric figure, without departing from the scope of the various described embodiments.The first geometric figure and the second geometric figure are both describing a geometric figure, but unless the context clearly indicates otherwise, they are not the same geometric figure.

[0043] In order to more prominently emphasize the independence of the operation of the functional units, many functional units or devices described in this specification have been labeled as modules. For example, the modules can be implemented as hardware circuits, which include custom VLSI circuits, gate arrays, or off-the-shelf semiconductors (such as logic chips, transistors or other discrete components). The modules can also be implemented in programmable hardware devices, such as field programmable gate arrays, programmable array logic, programmable logic devices, or similar devices.

[0044] The modules may also be implemented in software run by different types of processors. For example, a module of executable code may include one or more physical or logical blocks of computer instructions organized as objects, programs, or functions. However, the executable files of the modules do not have to be physically located together, but may include different commands stored in different locations that, when logically connected together, comprise the module and achieve the specified goals of the module.

[0045] Of course, the module of executable code can be one or many instructions, and can even be distributed in several different code segments, distributed in different programs and distributed in multiple memory / device devices. Similarly, operation data can be identified and shown in the module here, and the operation data can be embodied in any suitable form and organized in any suitable type of data structure. The operation data can be collected as a single data set, or can be distributed in different locations (including different storage devices), and can exist at least partially as electrical signals in a system or network. When a module or part of a module is implemented in software, the software part is stored on one or more computer-readable media.

[0046] As described in the background technology, the current surgical navigation system lacks intuitive guidance information, resulting in long operation time and low surgical accuracy. Taking the surgical navigation system based on MR equipment as an example, the user (such as a doctor or a doctor's assistant, etc.) can simultaneously observe the real patient and the virtual model (such as a spine model, a brain model, etc.) that matches the real patient through the MR equipment worn. The position of the real part to be operated on in the real physical space can be clearly determined through the part to be operated on (such as the lesion site in the body) displayed in the virtual model, and then the user can adjust the surgical instrument to contact the real part to be operated on. However, during the operation, due to the lack of intuitive guidance / intuitive visual navigation, the user needs to rely on experience to adjust the posture of the surgical instrument so that the surgical instrument contacts the real part to be operated on, and it is difficult to ensure that the posture of the surgical instrument when in contact with the real part to be operated on meets the surgical requirements, resulting in long operation time and low surgical accuracy.

[0047] In view of this, some embodiments provided in the present application disclose an MR device and a surgical navigation system, which display a 3D virtual model of a reference body aligned with a reference body and a 3D virtual model of an observation body aligned with an observation body in a first anchoring field of view, and display a local two-dimensional view of the reference body 3D virtual model, a first geometric figure representing the projection distance, and a second geometric figure representing the angle of the operation direction in a guide window interface displayed in a second anchoring field of view, and detect the actual operation process so that the first geometric figure and the second geometric figure can change according to the real-time operation, thereby providing the user with intuitive guidance information / intuitive visual navigation, so that the user can quickly bring the surgical instrument into contact with the actual surgical site in a posture that meets the requirements of preoperative preparation or intraoperative surgery, so as to improve the efficiency and accuracy of the surgery.

[0048] In an embodiment of the present application, the surgical navigation system is an electronic system that assists the user in performing surgery by providing guidance information. The guidance information includes: a virtual model and position corresponding to the actual surgical site, a virtual model and position of surgical instruments, and / or a surgical path, etc. The surgical navigation system is exemplified by a surgical navigation system based on image tracking. Among them, the surgical navigation system based on image tracking tracks the location of the surgical site and surgical instruments through real-time captured images to provide real-time guidance information to the user.

[0049] In one embodiment, see Figure 1 , which is a schematic diagram of the structure of a surgical navigation system in one embodiment of the present application, as shown in the figure, the surgical navigation system includes an MR device 1 and a computer device 2. The computer device 2 is connected to the MR device 1 for communication so as to transmit data between the MR device 1 and the computer device 2. In this embodiment, the surgical navigation system realizes the positioning and tracking of the surgical site and surgical instruments through the MR device 1 to provide real-time guidance information for the user.

[0050] In one embodiment, the surgical navigation system further includes a field of view sharing device, which is used to synchronously display the reference volume 3D virtual model, the observation volume 3D virtual model, and the guide window interface. The field of view sharing device is communicatively connected to the computer device. Specifically, the field of view sharing device is connected to the computer device. Figure 1 The MR device 1 shown may be located in the same actual physical space or in different actual physical spaces.

[0051] In the field of view sharing device and Figure 1 In the embodiment shown in FIG. 1 , the MR device 1 is located in different actual physical spaces. The view sharing device obtains the reference volume 3D virtual model, the observation volume 3D virtual model, and the guide window interface displayed in the MR device 1 and directly overlays and displays them with the real scene under the perspective of the view sharing device. In other words, the virtual objects (such as the reference volume 3D virtual model, the observation volume 3D virtual model, and the guide window interface) seen by the user (such as a doctor participating in the operation remotely) through the view sharing device and the virtual objects (such as the reference volume 3D virtual model, the observation volume 3D virtual model, and the guide window interface) seen by the user (such as a doctor in the operating room) through the view sharing device are the same as the virtual objects (such as the reference volume 3D virtual model, the observation volume 3D virtual model, and the guide window interface) seen by the user (such as a doctor in the operating room) through the view sharing device. Figure 1 The virtual objects viewed by the MR system 1 shown are identical.

[0052] In the field of view sharing device and Figure 1 In the embodiment shown in FIG. 1 , the MR device 1 is located in the same actual physical space. The view sharing device determines the reference volume 3D virtual model, the observation volume 3D virtual model, and the guide window interface in its perspective and displays them superimposed with the real scene in its perspective. In other words, the user (e.g., the doctor's assistant in the operating room) uses the view sharing device and the user (e.g., the doctor in the operating room) uses the view sharing device to view the real scene in the operating room. Figure 1 The virtual objects respectively seen by the MR device 1 are virtual objects in their respective viewing angles. Specifically, the reference volume 3D virtual model, the observation volume 3D virtual model, and the guide window interface in the viewing angle of the field of view sharing device can be displayed by the computer device through Figure 1 The relative position relationship between the MR device 1 and the field of view sharing device shown in the figure is determined by the reference volume 3D virtual model, the observation volume 3D virtual model, and the guide window interface displayed in the MR device 1. In one example, Figure 1 The relative position relationship between the MR device 1 and the field of view sharing device shown can be determined by jointly identifying a shared QR code.

[0053] Furthermore, in one embodiment, the vision sharing device can also synchronously display a selection window interface, etc. Figure 1 A virtual object is shown displayed in the MR device 1 .

[0054] In one embodiment, the visual field sharing device is exemplified by a MR device or an AR device. Figure 1 The structures of the MR devices 1 shown are the same or similar and will not be described in detail here.

[0055] Furthermore, the surgical navigation system further includes a server (not shown) that is communicatively connected to the computer device 2 and the MR device 1 .

[0056] In one embodiment, the MR (Mixed Reality Glasses) device 1 can display virtual objects stacked / superimposed on a real scene, that is, the MR device 1 can display virtual and real superposition; further, the user can also interact with the virtual object. Among them, the real scene is composed of various objects in the actual physical space. The MR device 1 is exemplified by a head-mounted MR device (such as MR glasses or MR helmet). In one embodiment, the MR device 1 may include one or more sensors (such as depth sensors, cameras, etc.) for scanning or collecting information from an actual physical space (such as a room, etc.) and a circuit system for transmitting the collected information to another device (such as a server, workstation, desktop computer, laptop, tablet computer, smart phone, etc.). Further, the MR device 1 may include a wireless interface for connecting to the Internet, a local wireless network, or another computing device.

[0057] In one embodiment, the MR device 1 may also be used to receive audio / gesture input from the user. Such gestures or audio input may be spoken voice commands or recognized user gestures, which when recognized by the MR device 1 may cause the MR device 1 to execute the corresponding command, and the gestures may be recognizable gestures described in the embodiments below. In one embodiment, the MR device 1 is preferably lightweight and is constructed to avoid the use of heavy components, which may make the MR device 1 uncomfortable to wear. For example, the MR device 1 is MR glasses, which are relatively light, and the user's head is less weighted when wearing the MR glasses during surgery.

[0058] The MR device 1 may be configured to determine a display area in an actual physical space for placing a virtual object in the display area. In other words, the MR device 1 may be configured to determine an anchoring field of view (e.g., a first anchoring field of view and a second anchoring field of view) for placing a virtual object in the anchoring field of view. Further, the MR device 1 may also provide image data and depth data to a mobile processor and receive display information from the mobile processor, which may be separate from the MR device 1, such as in a computer device in communication with the MR device 1, as described in detail in the accompanying drawings. Figure 1 discussed in more detail.

[0059] In one embodiment, the MR device 1 may receive a user input for indicating an anchoring field of view (e.g., a "second anchoring field of view"). The input may be any of a variety of user inputs, such as a button press, a recognizable gesture performed in the field of view of the camera, a focused gaze of the user (which is recognized as an image that remains relatively still for a predetermined time), or other forms of recognizable input. When the MR device 1 determines a virtual object anchored to a specified anchoring field of view, a processing device coupled to or within the MR device 1 may render the virtual object on a display device, so that when the user views the virtual object through the display device, the virtual object appears to be on the anchoring field of view. The virtual object may be any virtual object, including text, graphics, images, and / or 3D shapes, for example, the virtual object is a reference volume 3D virtual model, an observation volume 3D virtual model, a guide window interface, and a selection window interface. When presented in this manner, a virtual object placed at / on a specified location in an actual physical space can create a virtual reality experience and enable a user to interact with the virtual object. Various embodiments enable natural interaction with virtual objects (e.g., a reference volume 3D virtual model and an observation volume 3D virtual model). Such natural interaction with a virtual object may include gesture control of the virtual object, touch manipulation, highlighting of the touched portion thereof, etc. Among them, the recognizable gestures may be stored in the form of a gesture dictionary accessible by the MR device 1. Such a gesture dictionary may store motion data or patterns for identifying gestures, which may include poking, tapping, knocking, pushing, directing, pinching, flicking, turning, rotating, grabbing, pulling, palming, drawing (e.g., finger painting), forming shapes with fingers (e.g., an "OK" gesture), and swiping, etc., all of which may be performed at the apparent position of the virtual object, adjacent to the apparent position, or in the direction of the apparent position (relative to the user).

[0060] In another embodiment, the MR device 1 may determine the anchor field of view according to a position of an object in an actual physical space relative to the MR device 1 determined in real time.

[0061] In the embodiment of the present application, the pose of the object includes the position (coordinates) and the pose of the object. It should be noted that in the embodiment where the object can be regarded as a point, the pose of the object only includes the position.

[0062] The MR device 1 in the present application can render virtual objects displayed on an anchor field of view (such as the first anchor field of view and the second anchor field of view in the present application) to provide a mixed reality experience, which can facilitate interaction with a computing device and collaboration with other users.

[0063] See also Figure 2 Combined with Figure 1, which is a schematic diagram of the structure of an MR device in one embodiment of the present application. As shown in the figure, the MR device 1 includes a sensing device 10, a storage device 11, a processing device 12, and a display device 13.

[0064] The sensor device 10 is used to obtain sensor data of an observation body and a reference body matched with the observation body. In some embodiments, the sensor device 10 includes at least one sensor, and correspondingly, the sensor data is data detected by the sensor. In an embodiment where the sensor device 10 includes multiple sensors, the multiple sensors can all be sensors of the same type or sensors of different types.

[0065] In one embodiment, the sensing device 10 includes a camera, which is used to provide image data to the processing device 12, and then the processing device 12 can analyze the image data to estimate the position of the object in the image data relative to the MR device by using triangulation analysis; further, the sensing device can also include a depth sensor, which is used to provide depth data to the processing device 12, and then the processing device 12 can determine the distance of the object relative to the MR device based on the depth data.

[0066] In one embodiment, the sensing device 10 includes horizontal orientation sensors, such as accelerometers, gyroscopes, magnetic sensors, optical sensors, mechanical or electronic horizontal sensors, and inertial sensors, which, alone or in combination, can provide data about the up / down / horizontal orientation of the device (for example, by sensing gravity orientation) and thereby provide the user's head position / orientation (and the perspective therefrom) to the device's processor; further, the sensing device 10 can also include rotational orientation sensors, such as electronic compasses and accelerometers, which can provide data about left / right orientation and movement to the processing device 12.

[0067] In some embodiments, the storage device 11 is used to store at least one program, and the at least one program is executable by the processing device 12. Here, the storage device 11 includes, but is not limited to: a read-only memory, a random access memory, and a non-volatile memory. For example, the storage device 11 includes a flash memory device or other non-volatile solid-state storage device. In some embodiments, the storage device 11 may also include a memory away from one or more processing devices 12, such as a network attached memory accessed via an RF circuit or an external port and a communication network, wherein the communication network may be the Internet, one or more intranets, a local area network, a wide area network, a storage area network, etc., or a suitable combination thereof. The memory controller may control access to the memory by other components of the device such as a CPU and a peripheral interface.

[0068] The display device 13 is used to superimpose the first anchoring field of view and the second anchoring field of view on the real scene for presentation, so as to realize virtual-real superposition display, and then the user can see both the virtual object and the real scene through the display device 13 of the MR device 1. Among them, the first anchoring field of view is a display area used to display virtual models (such as reference body 3D virtual model and observation body 3D virtual model) in the actual physical space, and then, when the virtual model is anchored to the first anchoring field of view for display, the user can see that the virtual model is located in the first anchoring field of view through the display device 13 of the MR device. In one example, the first anchoring field of view can be determined according to the reference body 3D virtual model and the observation body 3D virtual model obtained by aligning the reference body 3D virtual model and the observation body 3D virtual model with the corresponding reference body and observation body described later, and the posture of the reference body 3D virtual model and the observation body 3D virtual model displayed in the MR device. The second anchoring field of view is a display area in the actual physical space for displaying a window interface (such as selecting a window interface or guiding a window interface, etc.), and then when the window interface is anchored to the second anchoring field of view for display, the user can see that the window interface is located in the second anchoring field of view through the display device 13 of the MR device. In one example, the second anchoring field of view can be determined according to the trigger gesture described later. For another example, the second anchoring field of view can be determined according to the trigger gesture and the moving gesture described later. It should be noted that the second anchoring field of view in the embodiment of the present application may be independent of the first anchoring field of view, or may have an overlapping area with the first anchoring field of view.

[0069] In one embodiment, the display device 13 is a semi-transparent display device, through which the user can see the real scene, and the display device 13 can display virtual objects (such as a 3D virtual model of an observation volume, a 3D virtual model of a reference volume, a selection window interface, or a guide window interface) so that the virtual object appears to be anchored to an anchoring field of view seen by the user through the display device 13. In this way, the user can use a semi-transparent head-mounted display device to view the real scene in the actual physical space, wherein objects in the real scene (such as the reference volume, the observation volume, and other items in the operating environment) are visible through the head-mounted display device, and the head-mounted display device also places virtual objects in the visible scene. Visible virtual objects can be anchored to or connected to real-world objects. This embodiment provides a mixed reality experience,

[0070] In one embodiment, the display device 13 is a display device based on video synthesis technology. Specifically, the processing device 12 (such as a graphics processor) obtains the image information of the real scene collected by the sensor device 10, and then synthesizes the virtual object into the image of the real scene after information processing, and finally the processing device 12 controls the display device 13 to display the synthesized image to the user. In one example, the display device based on video synthesis technology includes a display.

[0071] In one embodiment, the display device 13 is a display device based on an optical element, for example, the display device is a holographic display device. The processing device 13 controls the display device based on an optical element to first directly project the virtual object onto the optical element unit, and then send the virtual object into the human eye through the reflection of the optical element unit. At the same time, the image in the real scene will directly enter the human eye through the optical element unit, so that the user can see the virtual object while seeing the real scene.

[0072] In one example, the optical element-based display device includes a display and an optical element unit. The optical element unit includes a semi-transparent and semi-reflective mirror, a holographic diffraction waveguide grating, or other suitable optical elements. In this embodiment, the processing device controls the display to display the virtual object at a preset position and in a preset posture, and the optical element unit is used to reflect the virtual object on the display into the field of view of the human eye and to transmit light from the real scene.

[0073] It should be noted that the description of the display device in the present application is limited to examples. In other embodiments, the display device may also be a display device including elements such as optical waveguides.

[0074] In some embodiments, the processing device 12 is connected to the storage device 11 , the display device 13 , and the sensor device 10 , and is used to implement step S110 and step S120 when calling and executing the at least one program from the storage device 11 .

[0075] In step S110 , a 3D virtual model of a reference volume registered with a reference volume and a 3D virtual model of an observation volume registered with an observation volume are displayed in a first anchor field of view.

[0076] The reference body includes a medical device implanted in the part to be operated on or in the patient. The part to be operated on is the lesion site on the human body. For example, the part to be operated on is a brain tumor, and the reference body may correspond to the patient's brain. For another example, the part to be operated on is the stomach, and the reference body may correspond to the patient's abdomen. For another example, the part to be operated on is the thoracic vertebrae, and the reference body may correspond to all the thoracic vertebrae of the patient. The medical device implanted in the patient is a device that requires the user to further operate with a surgical instrument after implantation (such as an intramedullary nail main nail) or a device that requires the user to remove with a surgical instrument (such as a temporary vascular stent).

[0077] The observation body corresponding to the reference body includes a surgical instrument. The surgical instrument is an instrument for performing a corresponding operation on the reference body, and is related to the operation performed on the reference body. For example, if the reference body is the abdomen, and the operation performed on the reference body is an abdominal puncture operation, then the corresponding observation body is a puncture needle. For another example, if the reference body is a main nail of an intramedullary nail, and the operation performed on the reference body is surgical fixation of the main nail of the intramedullary nail, then the corresponding observation body is a guide sleeve.

[0078] In one embodiment, the reference body and the observation body are also configured with corresponding two-dimensional codes. For example, a two-dimensional code is configured at a preset position on the reference body and the observation body. Among them, the two-dimensional code on the reference body is used to call the reference body 3D virtual model, and the two-dimensional code on the observation body is used to call the observation body 3D virtual model. Furthermore, the two-dimensional code on the reference body can also be used to align the reference body 3D virtual model with the reference body, and the two-dimensional code on the observation body can also be used to align the observation body 3D virtual model with the observation body. In one example, the two-dimensional code can be fixed on the reference body and the observation body by sticking, clamping, or screwing.

[0079] It should be noted that in the embodiment where the reference body includes the part to be operated on, the QR code for calling the 3D virtual model of the reference body can be placed directly near the reference body. For example, if the reference body is the abdomen of the patient, the QR code can be placed in the lower right corner of the abdomen of the patient.

[0080] Wherein, the reference body 3D virtual model and the observation body 3D virtual model are 3D virtual models pre-built and stored in a database. In one example, the reference body 3D virtual model is obtained by using a scanning device to scan the reference body before or during surgery to obtain scanning data (such as CT scanning data, MRI scanning data, or ultrasound scanning data, etc.) to construct a three-dimensional model. In an embodiment where the reference body is a medical device implanted in a patient, the reference body 3D virtual model can also be constructed using the design parameters of the reference body. Wherein, the design parameters include but are not limited to geometric parameters, material parameters, and functional parameters. Similarly, the construction method of the observation body 3D virtual model is the same or similar to the construction method of the reference body 3D virtual model, which will not be repeated here. Further, in one embodiment, the reference body 3D virtual model and the observation body 3D virtual model are models after transparency processing to avoid the reference body 3D virtual model and the observation body 3D virtual model completely blocking the real scene, thereby improving the safety of the operation. Further, the reference body 3D virtual model is also marked with surgical paths and surgical incision positions.

[0081] In the embodiment where the reference body and the observation body are respectively configured with two-dimensional codes, the reference body 3D virtual model and the observation body 3D virtual model also include models of their own two-dimensional codes.

[0082] In some embodiments, the database storing the reference volume 3D virtual model and the observation volume 3D virtual model is a local database, a local area network database, or a cloud database. The local database is exemplified by a database of an MR device; the local area network database is exemplified by a database of a MR device; Figure 1 The database of the computer device 2 shown, wherein the computer device 2 and the MR device 1 are connected in the same local area network; the cloud database is exemplified as the database of the server in the surgical navigation system.

[0083] Before implementing step S110, the processing device calls the reference body 3D virtual model and the observation body 3D virtual model from the database of any of the aforementioned embodiments. In one embodiment, the processing device calls the reference body 3D virtual model and the observation body 3D virtual model from the database by identifying the two-dimensional codes respectively configured on the reference body and the observation body. Specifically, the processing device obtains the sensor data of the two-dimensional codes on the reference body and the observation body, and then the MR device calls the reference body 3D virtual model and the observation body 3D virtual model corresponding to the two-dimensional code from the database by identifying the sensor data. In another embodiment, the processing device can also obtain the reference body 3D virtual model and the observation body 3D virtual model through the displayed model selection interface. For example, the processing device displays the model selection interface in the second anchoring field of view and calls the reference body 3D virtual model and the observation body 3D virtual model from the database based on the user's trigger operation on the displayed model selection interface.

[0084] Furthermore, after the processing device acquires the reference body 3D virtual model and the observation body 3D virtual model, the reference body 3D virtual model and the observation body 3D virtual model are also registered with the corresponding reference body and the observation body. The registration refers to determining the posture of the reference body 3D virtual model and the observation body 3D virtual model displayed in the MR device based on the posture of the reference body and the observation body in the actual physical space (for example, the posture in the MR device coordinate system) to determine the first anchoring field of view, so that the user can see the reference body 3D virtual model with the same posture as the reference body at the position of the reference body through the MR device, and see the observation body 3D virtual model with the same posture as the observation body at the position of the observation body.

[0085] In one embodiment, the processing device determines the posture of the QR code on the reference body by acquiring the sensor data of the QR code on the reference body, and determines the posture of the QR code model in the 3D virtual model of the reference body displayed in the MR device based on the posture of the QR code on the reference body to achieve alignment between the reference body and the 3D virtual model of the reference body; similarly, the processing device aligns the observation body and the 3D virtual model of the observation body through the sensor data of the QR code on the observation body. In another embodiment, in the embodiment where the reference body is a part to be operated on, a plurality of registration point marks that can be identified by a scanning device are configured on the reference body (for example, when scanning a patient, a registration point mark is attached to the reference body), and the user touches the registration probe configured with a registration QR code with the registration point marks respectively. When the registration probe touches each registration point mark, the processing device obtains the sensing data of the registration QR code to determine the position and posture of each registration point mark on the reference body, and then the position and posture of each registration point mark in the 3D virtual model of the reference body displayed in the MR device can be determined based on the position and posture of each registration point mark, so as to realize the registration of the reference body and the 3D virtual model of the reference body. Wherein, the registration QR code and the QR code used to call the 3D virtual model of the reference body can be the same or different. In the embodiment where the registration QR code and the QR code used to call the 3D virtual model of the reference body are different, when the registration probe touches each registration point mark, the QR code used to call the 3D virtual model of the reference body is blocked.

[0086] It should be noted that, in other embodiments, the processing device may also send the acquired sensor data to Figure 1 The computer device 2 shown is used for sending the postures of the reference volume 3D virtual model and the observation volume 3D virtual model displayed in the MR device to the processing device after the computer device 2 performs registration.

[0087] After completing the registration, the processing device displays the reference body 3D virtual model registered with a reference body and the observation body 3D virtual model registered with an observation body in the first anchoring field of view. Among them, the first anchoring field of view is the same or similar to that described above and will not be repeated here. In this way, the user can see the reference body 3D virtual model and the observation body 3D virtual model in the first anchoring field of view through the display device, and can also see the real scene through the display device, such as the real reference body can be seen in the display area of ​​the reference body 3D virtual model, and the real observation body can be seen in the display area of ​​the observation body virtual model. It should be noted that the present application can also display the 3D reference body 3D virtual model and the observation body 3D virtual model before registration in the first anchoring field of view through the MR device. At this time, the first anchoring field of view can be predetermined or preliminarily determined based on the QR code of the corresponding 3D virtual model.

[0088] In step S120, a guide window interface is displayed in the second anchoring field of view, wherein the guide window interface displays a local two-dimensional view of the reference body 3D virtual model, a first geometric figure representing the projection distance between a first target position on the observation body 3D virtual model and a second target position on the reference body 3D virtual model, and a second geometric figure representing the operating direction angle between the first target position and the second target position.

[0089] Wherein, the projection distance and the angle of the operation direction are determined by using the postures of the observation body and the reference body obtained based on the sensor data. Specifically, the postures of the observation body and the reference body are first obtained based on the sensor data, and then the projection distance and the angle of the operation direction are determined using the postures of the observation body and the reference body. In one embodiment, the postures of the observation body and the reference body are the postures of the two in the MR device coordinate system. Wherein, the MR device coordinate system can be established with any point in the MR device as the origin. For example, the center point of the MR device is the origin.

[0090] In the embodiment where the reference body includes a medical device implanted in a patient, the determined positions of the two-dimensional code configured on the observation body and the reference body can be used as the positions of the observation body and the reference body. In the embodiment where the reference body includes a part to be operated on, the position of the reference body is the position after registration, and the position of the observation body is the determined position of the two-dimensional code configured on the observation body.

[0091] Further, after the postures of the observation body and the reference body are determined, the projection distance and the angle of the operation direction can be calculated. In one embodiment, the processing device determines the projection distance and the angle of the operation direction using the postures of the observation body and the reference body. In another embodiment, the processing device sends the postures of the observation body and the reference body to the computing device for the computing device to determine the projection distance and the angle of the operation direction. The present application does not limit the execution device for determining the posture, projection distance, and the angle of the operation direction.

[0092] Since the postures of the observation body and the reference body are the postures of the observation body 3D virtual model registered with the observation body and the reference body 3D virtual model registered with the reference body, after determining the postures of the observation body and the reference body, the posture of the first target position on the observation body 3D virtual model (e.g., the position / posture of the first target position) and the posture of the second target position on the reference body 3D virtual model (e.g., the position / posture of the second target position) can be calculated based on the positions of the first target position and the second target position in the corresponding 3D virtual model, and then the projection distance and the operation direction angle can be calculated based on the posture of the first target position and the posture of the second target position.

[0093] In order to determine the first target position and the second target position. In one embodiment, the processing device is further used to display a selection window interface in the second anchoring field of view before displaying the guide window interface. The second anchoring field of view is the same or similar to that described above and will not be described in detail here.

[0094] In one embodiment, the processing device displays the selection window interface according to the detected trigger gesture of the user. The trigger gesture may be any of the recognizable gestures described above, for example, examples of the trigger gesture include an open palm with the palm facing up, an open palm with the back of the hand facing up, and a fist. After the processing device detects the trigger gesture, the area where the trigger gesture is located is used as the second anchor field of view to display the selection window interface in the second anchor field of view. In another embodiment, the processing device may also display the selection window interface according to the detected user voice. Examples of user voice include: "Display selection window interface" voice, "Select target location" voice, etc.

[0095] Further, after displaying the selection window interface, the processing device or computer device can determine the first target position and the second target position based on the user's operation on the selection window interface. In one embodiment, the selection window interface includes a plurality of targets to be selected, a first target position input bar, and a second target position input bar. Among them, the user's operation on the selection window interface is to select a target to be selected. After the user selects the target to be selected, the processing device or computer device can obtain the target to be selected and determine the first target position and the second target position. Among them, the user can select the target to be selected by clicking in the selection window interface, or by voice. It should be noted that the present application does not limit the method of selecting the target to be selected and the input order of the first target position and the second target position. The first target to be selected selected by the user on the selection window interface can be the target to be selected corresponding to the first target position or the target to be selected corresponding to the second target position.

[0096] In some embodiments, the target to be selected may be a target position directly, or a target area corresponding to the target position. For example, the first target position is the center point of the transverse hole in the main nail of the intramedullary nail, and the second target position is the center point of the front end of the guide sleeve. Then, the target to be selected may be the center point of the transverse hole in the main nail of the intramedullary nail and the center point of the front end of the guide sleeve directly, or the transverse hole and the front end of the guide sleeve. In an embodiment in which the target to be selected is a target area, a correspondence relationship of the target position is pre-stored in the processing device or computer device, and the correspondence relationship of the target position indicates the correspondence relationship between the target position (the first target position and the second target position) and the target area, and then after the target area is acquired, the target position (the first target position and the second target position) can be determined according to the correspondence relationship.

[0097] See also Figure 3 and Figure 4 , respectively showing schematic diagrams of the selection window interface when no target to be selected is selected and after the target to be selected is selected in one embodiment of the present application, such as Figure 3 As shown, the selection window interface includes target A, target B, target C, target D, target E, target F, target G, target H, and target I, and the selection window interface also includes a first target position input field and a second target position input field. Further, the selection window interface also includes a cancel area for the user to cancel the target that has been entered. The user Figure 3 After selecting target A in the selection window interface shown in the figure, Figure 4 As shown, the processing device displays the target A to be selected in the first target position input field. Figure 4 In the selection window interface shown, a target to be selected can continue to be selected as the target to be selected corresponding to the second target position.

[0098] After the processing device or computer equipment determines the first target position on the 3D virtual model of the observation body and the second target position on the 3D virtual model of the reference body based on the selected target, the computer equipment or processing device can determine the posture of the first target position and the second target position according to the detected postures of the observation body and the reference body, and then obtain the projection distance and the operation direction angle.

[0099] In some embodiments, the projection distance may be the distance between the projection point of the first target position on the projection plane and the second target position, or the distance between the projection point of the first target position on the projection plane and the projection point of the second target position on the projection plane. The projection plane is a projection plane used to generate the local two-dimensional view, and the projection plane may be a plane passing through the second target position, or a plane located at the front side (the side away from the second operating direction) or the rear side (the side close to the second operating direction) of the second target position. The second operating direction will be described in detail later.

[0100] Specifically, in the embodiment where the projection plane passes through the second target position, the projection distance is the distance between the projection point of the first target position on the projection plane and the second target position; in the embodiment where the projection plane does not pass through the second target position, the projection distance is the distance between the projection point of the first target position on the projection plane and the projection point of the second target position on the projection plane. Figure 5 and Figure 6 , respectively, are schematic diagrams showing projection distances in different embodiments of the present application, such as Figure 5 As shown, the projection plane J passes through the second target position M1 on the reference body 3D virtual model K1, and the corresponding projection distance L is the distance between the projection point m2 of the first target position M2 on the observation body 3D virtual model K2 on the projection plane J and the second target position M1. Figure 6 As shown, the projection plane J is located on the rear side of the second target position M1 on the reference body 3D virtual model K1, and the projection distance L is the distance between the projection point m2 of the first target position M2 on the projection plane J and the projection point m1 of the second target position M1 on the projection plane J.

[0101] In some embodiments, the operation direction angle is the angle between the first operation direction corresponding to the first target position and the second operation direction corresponding to the second target position. The first operation direction is the extension direction of the front end of the 3D virtual model of the observation body, and the second operation direction is the surgical direction that meets the surgical requirements, that is, the direction perpendicular to the projection plane. When the observation body contacts the reference body according to the surgical direction that meets the surgical requirements, the surgical requirements for the observation body angle can be met.

[0102] See also Figure 7 , which is a schematic diagram of the operating direction angle in one embodiment of the present application. As shown in the figure, the second operating direction N1 is a direction perpendicular to the projection plane J of the reference body 3D virtual model K1, and the angle θ between the first operating direction N2 and the second operating direction N1 is the operating direction angle.

[0103] It should be noted that merely moving forward or backward from the current position in a direction perpendicular to the projection surface does not result in a change in the projection distance and the angle between the operation direction.

[0104] After determining the projection distance and the angle of the operation direction, the processing device can display the guide window interface in the second anchoring field of view. For example, while displaying the guide window interface in the second anchoring field of view, a 3D virtual model of a reference body registered with a reference body and a 3D virtual model of an observation body registered with an observation body are displayed in the first anchoring field of view. Figure 8 , showing a schematic diagram of a guide window interface and a reference volume 3D virtual model and an observation volume 3D virtual model displayed in an embodiment of the present application. As shown in the figure, the user can see a reference volume 3D virtual model K1 aligned with a reference volume and an observation volume 3D virtual model K2 aligned with an observation volume in the first anchoring field of view P1 through the MR device, and can see the guide window interface in the second anchoring field of view P2. It should be noted that the present application Figures 8 to 11 Although the real scene is not shown in the illustrated embodiment, in actual use, the user can see the real scene through the MR device. Figures 8 to 11 The virtual object shown can also see the real scene in the actual physical space (such as the reference volume and the observation volume in the actual physical space).

[0105] In one embodiment, the processing device displays the guide window interface in the second anchoring field of view that displays the selection window interface described above. In other words, the selection window interface is updated to the guide window interface in the second anchoring field of view. Furthermore, the area where the guide window interface is displayed can be changed according to the user's movement gesture. The movement gesture can be any of the recognizable gestures described above. For example, the movement gesture is pinching, and the user changes the display area where the guide window interface is displayed by pinching the guide window interface.

[0106] Among them, the guide window interface displays a local two-dimensional view of the reference body 3D virtual model, a first geometric figure representing the projection distance between the first target position on the observation body 3D virtual model and the second target position on the reference body 3D virtual model, and a second geometric figure representing the operation direction angle between the first target position and the second target position.

[0107] In one embodiment, the local two-dimensional view is a local enlarged view of the two-dimensional projection of the reference body 3D virtual model corresponding to a projection surface. For example, the two-dimensional projection includes an image obtained by projecting the reference body 3D virtual model on the front side of the projection surface onto the projection surface; further, the two-dimensional projection also includes a cross-sectional image of the reference body 3D virtual model at the projection surface. Figure 8As shown, the local two-dimensional view is an enlarged view of the two-dimensional projection of the front end of the 3D virtual model of the main intramedullary nail. In a specific embodiment, the local two-dimensional view is a local enlarged view of the two-dimensional projection corresponding to a projection surface of the 3D virtual model of the reference body including the second target position. In other embodiments, the content displayed in the local two-dimensional view of the 3D virtual model of the reference body can be determined according to surgical requirements.

[0108] In some embodiments, the first geometric figure may be a regular figure such as a ray, a circle, or a regular polygon, and similarly, the second geometric figure may also be a regular figure such as a ray, a circle, or a regular polygon. In the following embodiments, the first geometric figure is a circle and the second geometric figure is a ray whose starting point falls on the circumference of the circle.

[0109] In one embodiment, the size of the first geometric figure is used to characterize the size of the projection distance. The size of the second geometric figure is used to characterize the size of the operating direction angle. In an embodiment where the first geometric figure is a circle and the second geometric figure is a ray whose starting point falls on the circumference of the circle, the size of the circle (such as the radius or diameter) is used to represent the size of the projection distance, that is, the larger the projection distance, the larger the size of the circle, for example, the size of the circle is directly proportional to the projection distance. The size of the ray (such as the length of the ray) is used to characterize the size of the operating direction angle, that is, the larger the operating direction angle, the larger the size of the ray, for example, the size of the ray is directly proportional to the operating direction angle. It should be noted that when the projection distance is close to 0, the first geometric figure is approximately a point; similarly, when the operating direction angle is close to 0, the second geometric figure is approximately a point.

[0110] Further, in one embodiment, the position of the first geometric figure in the local two-dimensional view of the reference body 3D virtual model represents the deviation direction of the projection point of the first target position relative to the second target position. The position of the second geometric figure in the local two-dimensional view of the reference body 3D virtual model is used to represent the deviation direction of the first operation direction relative to the second operation direction. Figure 8 The circle is offset upward relative to the local two-dimensional view, indicating that the projection point of the first target position is offset upward relative to the second target position. Similarly, the ray is offset upward relative to the local two-dimensional view, indicating that the first operating direction of the first target position is offset upward relative to the second operating direction of the second target position.

[0111] In an embodiment where the first geometric figure is a circle and the projection plane passes through the second target position, the circumference of the circle passes through the second target position and the projection point of the first target position on the partial cross-sectional view. In an embodiment where the first geometric figure is a circle and the projection plane does not pass through the second target position, the circumference of the circle passes through the projection point of the second target position and the projection point of the first target position. Figure 5 In the illustrated embodiment, the circumference of the circle representing the projection distance passes through the projection point m2 of the second target position M1 and the first target position M2. Figure 6 In the illustrated embodiment, the circumference of the circle representing the projection distance passes through the projection point m1 of the second target position M1 and the projection point m2 of the first target position M2. In an embodiment in which the second geometric figure is a ray whose starting point falls on the circumference of the circle and the projection surface passes through the second target position, the starting point of the ray passes through the second target position on the partial cross-sectional view. In an embodiment in which the second geometric figure is a ray whose starting point falls on the circumference of the circle and the projection surface does not pass through the second target position, the starting point of the ray passes through the projection point of the second target position on the partial cross-sectional view. Figure 5 In the illustrated embodiment, the starting point of the ray representing the operation direction angle passes through the second target position M1. Figure 6 In the illustrated embodiment, the starting point of the ray representing the operation direction angle passes through the projection point m1 of the second target position M1.

[0112] It should be noted that, when the first target position is located on the projection plane, the first target position is the projection point of the first target position.

[0113] In one embodiment, when the reference body and / or the observation body moves, the first geometric figure and / or the second geometric figure in the guide window interface may also change accordingly. For example, the first geometric figure is a circle passing through the second target position, the second geometric figure is a ray whose starting point falls on the second target position, and the projection surface passes through the second target position. Fig. 9 Combined with Figure 8 , Fig. 9 Show this application Figure 8 The schematic diagram of the first geometric figure after the change in the embodiment shown, Figure 8 In the embodiment shown, the observation volume corresponding to the observation volume 3D virtual model K2 is represented by Figure 8 The corresponding position moves to the right away from the second target position (the center point of the transverse hole in the main nail of the intramedullary nail) in a direction parallel to the projection plane, and moves to Fig. 9The position corresponding to the 3D virtual model K2 of the observation body causes the projection distance between the first target position (the center point of the front end of the guide sleeve) and the second target position to change, thereby Fig. 9 The size of the first geometric figure (circle) in is greater than Figure 8 the size of the first geometric figure (the circle), and Fig. 9 The first geometric figure (circle) in Figure 8 The first geometric figure (circle) in the partial 2D view of the reference volume 3D virtual model K1 is offset to the right. Fig.10 Combined with Figure 8 , Fig.10 Show this application Figure 8 The schematic diagram of the second geometric figure after the change in the embodiment shown in the figure is as shown in the figure. Figure 8 In the embodiment shown, the observation volume corresponding to the observation volume 3D virtual model K2 is represented by Figure 8 The corresponding position is deflected to the right and moves away from the projection surface in the direction perpendicular to the projection surface to Fig.10 The position corresponding to the 3D virtual model K2 of the observation body causes the direction of the operation direction angle between the first target position and the second target position to change, thereby Fig.10 The ray in Figure 8 The ray in is deflected to the right.

[0114] In some embodiments, in the guide window interface, different colors of the first geometric figure are used to represent the projection distance within different distance ranges, and different colors of the second geometric figure are used to represent the operation direction angle within different angle ranges. In this way, the user can clearly understand the size of the projection distance and the operation direction angle by observing the colors of the first geometric figure and the second geometric figure, and then accurately control the movement of the observation body. In one example, the distance range can be divided into a long distance range, a medium distance range, and a short distance range, and the color of the first geometric figure corresponding to each distance range is different. For example, the color of the first geometric figure of the projection distance belonging to the long distance range is red, the color of the first geometric figure of the projection distance belonging to the medium distance range is yellow, and the color of the first geometric figure of the projection distance belonging to the short distance range is green. Similarly, the angle range can also be divided into a large angle range, a medium angle range, and a small angle range, and the color of the second geometric figure corresponding to each angle range is different. For example, the color of the second geometric figure of the operation direction angle belonging to the large angle range is red, the color of the second geometric figure of the operation direction angle belonging to the medium angle range is yellow, and the color of the second geometric figure of the operation direction angle belonging to the small angle range is green. It should be noted that the division of distance ranges and angle ranges and the examples of colors in this application are limited to exemplary descriptions. In other embodiments, other colors such as blue and purple can be used to represent the projection distance and the angle between the operation direction. The distance range and angle range can be further subdivided, or the number of divided distance ranges and the number of divided angle ranges can be different.

[0115] In one embodiment, when the processing device detects that the observation body passes through the second target position of the reference body, the local two-dimensional view of the 3D virtual model of the observation body is displayed on the guide window interface, wherein the local two-dimensional view of the 3D virtual model of the observation body is a local enlarged view of the two-dimensional projection of the 3D virtual model of the observation body corresponding to the projection surface. In this way, during the operation, the user can know whether the observation body passes through the second target position by highlighting the local two-dimensional view of the 3D virtual model of the observation body on the guide window interface, so as to avoid excessive extension into the reference body.

[0116] In one embodiment, the processing device also synchronously displays the first geometric figure and the second geometric figure on the reference body 3D virtual model of the first anchoring field of view. In one example, the size of the first geometric figure in the guide window interface is larger than the size of the first geometric figure on the reference body 3D virtual model, and the size of the second geometric figure in the guide window interface is larger than the size of the second geometric figure on the reference body 3D virtual model. Figure 8As shown, the sizes of the circles and rays displayed in the guide window interface are larger than those displayed on the reference body 3D virtual model K1. In this way, the user can observe the first geometric figure and the second geometric figure on the reference body 3D virtual model, which is convenient for hand-eye coordination during surgery, and can also observe the larger first geometric figure and the second geometric figure in the guide window interface when the first geometric figure and the second geometric figure in the reference body 3D virtual model are smaller, thereby improving surgical efficiency and accuracy.

[0117] Among them, the first geometric figure and the second geometric figure displayed on the reference body 3D virtual model change synchronously with the first geometric figure and the second geometric figure displayed in the guide window interface. For example, zooming in or out synchronously, color changes synchronously, etc. Further, taking the first geometric figure as a circle and the second geometric figure as a ray as an example, in an embodiment where the circumference of the circle in the guide window interface passes through the second target position on the local cross-sectional view and the projection point of the first target position, and the starting point of the ray in the guide window interface passes through the second target position on the local cross-sectional view, the circumference of the reduced circle in the second anchoring field of view passes through the second target position in the reference body 3D virtual model and the projection point of the first target position, and the reduced ray in the second anchoring field of view passes through the second target position in the reference body 3D virtual model.

[0118] It should be noted that, in some embodiments, Figure 8 Partial areas of the first geometric figure and the second geometric figure displayed on the reference volume 3D virtual model are blocked by the reference volume 3D virtual model.

[0119] When the first geometric figure and the second geometric figure are both very small, for example, when they are both approximately one point, it means that the posture of the observation body meets the surgical requirements. However, due to the processing error of the observation body (surgical instrument), when the displayed first geometric figure and the second geometric figure are both approximately one point, there is actually a deviation between the first target position on the observation body and the second target position on the reference body, thereby reducing the surgical accuracy.

[0120] To this end, the guide window interface also includes a control area for adjusting the second target position, so that the processing device adjusts the offset of the second target position based on the user's operation in the control area. The offset is used to adjust the coordinates of the second target position in the up, down, left, and right directions in calculating the angle between the projection distance and the operation direction, and the plane formed by the up, down, left, and right directions is parallel to the projection surface. Figure 8, the upward offset can be adjusted with the "Up + 1" button, the downward offset can be adjusted with the "Down + 1" button, the left offset can be adjusted with the "Left + 1" button, and the right offset can be adjusted with the "Right + 1" button. For example, by clicking the "Up + 1" button in the control area, the corresponding upward offset is 1mm, and the coordinates of the second target position can be increased by 1mm; further, the offset can be reset to 0 by clicking the reset button. Figure 8 The buttons in the illustrated embodiment are only examples, and in other embodiments, other text or icon buttons may be used to adjust the offset. It should be noted that the offset described in this embodiment is used to calculate the projection distance and the angle of the operation direction using the coordinates of the second target position after the offset, but the coordinates of the second target position in the actual 3D virtual model and the local two-dimensional view remain unchanged.

[0121] In one embodiment, see Fig.11 , showing a schematic diagram of a guide window interface and a reference body 3D virtual model and an observation body 3D virtual model displayed in another embodiment of the present application. As shown in the figure, the guide window interface also displays specific values ​​of the projection distance and the operation direction angle.

[0122] In one embodiment, please continue to refer to Fig.11 , the guide window interface also displays a vertical distance value. The vertical distance is used to represent the distance between the first target position and the second target position in a direction perpendicular to the projection plane. In this way, the user can clearly know whether to move the observation object forward through the displayed vertical distance value to avoid excessive extension of the observation object.

[0123] In one embodiment, the processing device 12 includes one or more processors. The processing device 12 is operable to perform data read and write operations with the storage device 11. The processing device 12 includes one or more general-purpose microprocessors (such as CPUs or GPUs), one or more special-purpose processors (ASICs), one or more digital signal processors, one or more field programmable logic arrays (FPGAs), or any combination thereof. In one example, the special-purpose processor includes a graphics processor.

[0124] In some embodiments, the MR device further includes an interface device (not shown), and the interface device includes at least one interface unit. For example, the interface device includes but is not limited to: a serial interface or a parallel interface such as an HDMI interface or a USB interface. In one embodiment, the interface device further includes a network communication unit, which is a device for data transmission using a wired or wireless network, examples of which include but are not limited to: an integrated circuit including a network card, a local area network module such as a WiFi module or a Bluetooth module, a wide area network module such as a mobile network, etc. For example, the network communication unit is used to output the posture of the observation body and the reference body detected by the MR device for the computer device 2 receiving the posture to calculate the projection distance and the operation direction angle between the first target position on the observation body 3D virtual model and the second target position on the reference body 3D virtual model, receive the reference body 3D virtual model, and receive the observation body 3D virtual model.

[0125] In one embodiment, see Fig.12 , which is a schematic diagram of the structure of a computer device in one embodiment of the present application. As shown in the figure, the computer device 2 includes a storage device 20, a processing device 21 connected to the storage device 20, and an interface device 22.

[0126] In some embodiments, the storage device 20 is used to store at least one program, and the storage device 20 includes but is not limited to: read-only memory, random access memory, and non-volatile memory. For example, the storage device 20 includes a flash memory device or other non-volatile solid-state storage device. In some embodiments, the storage device 20 may also include a memory away from one or more processing devices 21, such as a network attached memory accessed via an RF circuit or an external port and a communication network, wherein the communication network may be the Internet, one or more intranets, a local area network, a wide area network, a storage area network, etc., or a suitable combination thereof. The memory controller may control access to the memory by other components of the device such as the CPU and peripheral interfaces.

[0127] In one embodiment, the storage device 20 is further used to store at least one reference volume 3D virtual model and an observation volume 3D virtual model adapted to the reference volume 3D virtual model, so as to be called by the MR device 1 .

[0128] In some embodiments, the interface device 22 is used to communicate with the MR device to receive an observation body obtained by the MR device and the position of a reference body adapted to the observation body. The interface device 22 includes at least one interface unit, each of which is used to output a visual interface, receive human-computer interaction events generated according to user operations, etc. For example, the interface device 22 includes but is not limited to: a serial interface such as an HDMI interface or a USB interface, or a parallel interface, etc. In one embodiment, the interface device 22 also includes a network communication unit, which is a device for data transmission using a wired or wireless network, examples of which include but are not limited to: an integrated circuit including a network card, a local area network module such as a WiFi module or a Bluetooth module, a wide area network module such as a mobile network, etc.

[0129] In some embodiments, the interface device 22 is further used to receive display data synchronously transmitted by the MR device, wherein the display data includes the real scene and virtual objects seen by the user through the MR device.

[0130] In some embodiments, the interface device is further used to send the reference volume 3D virtual model and the observation volume 3D virtual model adapted to the reference volume 3D virtual model to the MR device.

[0131] In some embodiments, the processing device 21 is connected to the storage device, and the processing device 21 is used to call the at least one program from the storage device 20 and execute it to determine the projection distance and the operation direction angle between the first target position on the 3D virtual model of the observation body and the second target position on the 3D virtual model of the reference body according to the acquired postures of the observation body and the reference body. The method of determining the projection distance and the operation direction angle is the same or similar to that described above, and will not be repeated here.

[0132] In some embodiments, the processing device 21 includes one or more processors. The processing device 21 can be operated to perform data read and write operations with the storage device 20. The processing device 21 includes one or more general-purpose microprocessors, one or more special-purpose processors (ASICs), one or more digital signal processors, one or more field programmable logic arrays (FPGAs), or any combination thereof. In one example, the special-purpose processor includes a graphics processor.

[0133] In one embodiment, if Fig.12 As shown, the computer device 2 further includes a display device 23. The display device is exemplified by a display or a touch screen of the computer device.

[0134] In an embodiment where the computer device includes the display device, the processing device is further configured to display the display data synchronously transmitted by the MR device and received by the interface device on the display device. In this way, a user who is not wearing the MR device during the operation can also view the virtual object and the real scene through the computer device.

[0135] In summary, in some embodiments provided in the present application, an MR device and a surgical navigation system are disclosed, which display a reference body 3D virtual model aligned with a reference body and an observation body 3D virtual model aligned with an observation body in a first anchoring field of view, and display a local two-dimensional view of the reference body 3D virtual model, a first geometric figure representing the projection distance, and a second geometric figure representing the angle of the operation direction in a guide window interface displayed in a second anchoring field of view, and detect the actual operation process so that the first geometric figure and the second geometric figure can change according to the real-time operation, so as to provide the user with intuitive guidance information / intuitive visual navigation, so that the user can quickly contact the surgical instrument with the real surgical site in a posture that meets the requirements of preoperative preparation or intraoperative surgery, so as to improve the efficiency and accuracy of the operation; further, by synchronously displaying the first geometric figure and the second geometric figure in the reference body 3D virtual model, the user can observe the first geometric figure and the second geometric figure on the reference body 3D virtual model, which is convenient for hand-eye coordination during the operation, and can also observe the larger first geometric figure and the second geometric figure in the guide window interface when the first geometric figure and the second geometric figure in the reference body 3D virtual model are small, thereby improving the efficiency and accuracy of the operation.

[0136] The above embodiments are merely illustrative of the principles and effects of the present application and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. An MR device, characterized in that: include: A sensing device for acquiring sensing data of an observation body and a reference body matched with the observation body; A storage device for storing at least one program; A display device, used to superimpose the first anchoring field of view and the second anchoring field of view on a real scene for presentation; A processing device connected to the storage device, the display device, and the sensor device, for implementing, when calling and executing the at least one program from the storage device: Displaying a 3D virtual model of a reference volume registered with a reference volume and a 3D virtual model of an observation volume registered with an observation volume in the first anchor field of view; and Displaying a guide window interface in the second anchoring field of view, wherein the guide window interface displays a local two-dimensional view of the reference volume 3D virtual model, a first geometric figure representing a projection distance between a first target position on the observation volume 3D virtual model and a second target position on the reference volume 3D virtual model, and a second geometric figure representing an operation direction angle between the first target position and the second target position; The projection distance and the operation direction angle are determined by using the positions and postures of the observation body and the reference body obtained based on the sensor data.

2. The MR device according to claim 1, characterized in that The processing device is also used to call the reference volume 3D virtual model and the observation volume 3D virtual model.

3. The MR device according to claim 2, characterized in that The processing device retrieves the reference body 3D virtual model and the observation body 3D virtual model from a database by identifying a two-dimensional code set on the reference body and / or the observation body.

4. The MR device according to claim 3, characterized in that The database is a local database, a local area network database, or a cloud database.

5. The MR device according to claim 1, characterized in that The reference body includes a medical device implanted in a surgical site or a patient, and the observation body includes a surgical instrument.

6. The MR device according to claim 1, characterized in that The processing device is further configured to display a selection window interface in the second anchoring field of view before displaying the guide window interface, so as to determine the first target position and the second target position based on the user's operation on the selection window interface.

7. The MR device according to claim 6, characterized in that The processing device displays the selection window interface according to the user's trigger gesture.

8. The MR device according to claim 1, characterized in that The guide window interface also includes a control area for adjusting the second target position, so that the processing device adjusts the offset of the second target position based on the user's operation in the control area.

9. The MR device according to claim 1, characterized in that The local two-dimensional view is a local enlarged view of the two-dimensional projection of the reference body 3D virtual model corresponding to a projection surface.

10. The MR device according to claim 9, characterized in that The projection distance is the distance between the projection point of the first target position on the projection surface and the second target position, or the distance between the projection point of the first target position on the projection surface and the projection point of the second target position on the projection surface, and the operation direction angle is the angle between the first operation direction corresponding to the first target position and the second operation direction corresponding to the second target position.

11. The MR device according to claim 1, characterized in that The size of the first geometric figure is used to characterize the size of the projection distance, and the size of the second geometric figure is used to characterize the size of the operating direction angle.

12. The MR device according to claim 1, characterized in that The position of the first geometric figure in the local two-dimensional view of the reference body 3D virtual model represents the deviation direction of the projection point of the first target position relative to the second target position, and the position of the second geometric figure in the local two-dimensional view of the reference body 3D virtual model is used to characterize the deviation direction of the operating direction of the second target position relative to the operating direction of the first target position.

13. The MR device according to claim 1, 11 or 12, characterized in that: The first geometric figure is a circle, and the second geometric figure is a ray whose starting point falls on the circumference of the circle.

14. The MR device according to claim 13, characterized in that The circumference of the circle passes through the second target position on the partial cross-sectional view and the projection point of the first target position or passes through the projection point of the second target position and the projection point of the first target position, and the starting point of the ray passes through the second target position on the partial cross-sectional view or the projection point of the second target position.

15. The MR device according to claim 1, characterized in that In the guide window interface, different colors of the first geometric figure are used to represent projection distances within different distance ranges, and in the guide window interface, different colors of the second geometric figure are used to represent operation direction angles within different angle ranges.

16. The MR device according to claim 1, characterized in that The processing device is also used to synchronously display the first geometric figure and the second geometric figure on the 3D virtual model of the reference volume in the first anchor field of view.

17. The MR device according to claim 1, characterized in that The processing device is further configured to display a local two-dimensional view of the 3D virtual model of the observation body on the guide window interface when it is detected that the observation body passes through a second target position of the reference body.

18. A surgical navigation system, characterized in that: include: An MR device as described in any one of claims 1 to 17, and a computer device communicatively connected to the MR device.

19. The surgical navigation system according to claim 18, characterized in that: The computer device comprises: A storage device, used to store at least one program; An interface device, used for communication connection with the MR device to receive a posture of an observation body and a reference body matched with the observation body sent by the MR device; A processing device is connected to the storage device and the interface device, and is used to determine the projection distance and the operation direction angle between the first target position on the 3D virtual model of the observation body and the second target position on the 3D virtual model of the reference body according to the posture of the observation body and the reference body when calling and executing the at least one program from the storage device.

20. The surgical navigation system according to claim 19, characterized in that: The storage device is also used to store at least one reference body 3D virtual model and an observation body 3D virtual model adapted to the reference body 3D virtual model, so as to be called by the MR device.

21. The surgical navigation system according to claim 19, characterized in that: The computer device further comprises a display device, and the processing device is used to display the display data synchronously transmitted by the MR device and received by the interface device on the display device.

22. The surgical navigation system according to claim 18, characterized in that: The surgical navigation system also includes a field of view sharing device that is communicatively connected to the computer device, and the field of view sharing device is used to synchronously display the reference volume 3D virtual model, the observation volume 3D virtual model, and the guide window interface.

Images