Mixed reality surgical navigation system and device

Abstract

This application relates to a mixed reality surgical navigation system and apparatus. The mixed reality surgical navigation system includes a processor configured to perform the following methods: displaying a first mixed reality model of a target body part and guiding a user to register based on the first mixed reality model, obtaining a registration result; anchoring the first mixed reality model to the target body part based on the registration result, guiding the user through a preparation process based on the first mixed reality model anchored to the target body part, and performing surgical navigation based on the first mixed reality model anchored to the target body part. Using this system can improve interaction efficiency.

Description

Technical Field

[0001] This application relates to the field of medical technology, and in particular to a mixed reality surgical navigation system and device. Background Technology

[0002] Surgical navigation technology can help doctors accurately locate complex anatomical structures, reduce damage to surrounding tissues, and thus improve the safety, effectiveness, and precision of surgery.

[0003] However, the relevant technologies mainly rely on two-dimensional images on the screen for navigation, which has problems such as unintuitive information expression and inconvenient user interaction, resulting in low interaction efficiency of current mixed reality surgical navigation methods. Summary of the Invention

[0004] Therefore, it is necessary to provide a mixed reality surgical navigation system and device that can improve interaction efficiency in response to the above-mentioned technical problems.

[0005] In a first aspect, this application provides a mixed reality surgical navigation system, which includes a processor for performing the following methods:

[0006] The system displays a first mixed reality model of the target area and guides the user to register based on this model, resulting in a registration outcome.

[0007] Based on the registration results, the first mixed reality model is anchored to the target area for display, guiding the user through the preparation process and providing surgical navigation based on the first mixed reality model anchored to the target area.

[0008] Secondly, this application also provides a mixed reality surgical navigation device, comprising:

[0009] The first guidance module is used to display the first mixed reality model of the target area and guide the user to register based on the first mixed reality model, thereby obtaining the registration result;

[0010] The second guidance module is used to anchor the first mixed reality model to the target area based on the registration results, so as to guide the user through the preparation process based on the first mixed reality model anchored to the target area, and to perform surgical navigation based on the first mixed reality model anchored to the target area.

[0011] Thirdly, this application also provides a mixed reality surgical navigation system, including a mixed reality device, which includes glasses and a processing device;

[0012] The processing device is used to display a first mixed reality model of a target body part through glasses, guide the user to register based on the first mixed reality model, obtain the registration result, and anchor the first mixed reality model to the target body part through glasses to display it, guide the user to complete the preparation process based on the first mixed reality model anchored to the target body part, and perform surgical navigation based on the first mixed reality model anchored to the target body part.

[0013] Fourthly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of any of the above methods.

[0014] Fifthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above methods.

[0015] Sixthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above methods.

[0016] The aforementioned mixed reality surgical navigation system and device, by displaying a first mixed reality model of the target area and guiding the user through registration based on this model, provides a unique advantage. Users can see not only the real physical space but also the virtual first mixed reality model, enabling efficient and intuitive registration. Furthermore, because the first mixed reality model can be anchored to the target area based on the registration result, users can gain a more intuitive understanding of the target area. This allows for efficient guidance during preparation and surgical navigation based on the anchored first mixed reality model. Throughout this process, users do not need to view a two-dimensional image on a screen or switch their gaze frequently, resulting in high interaction efficiency. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a diagram illustrating the application environment of a mixed reality surgical navigation method in one embodiment.

[0019] Figure 2 This is a flowchart illustrating a mixed reality surgical navigation method in one embodiment;

[0020] Figure 3 This is a flowchart illustrating the process of determining the registration result in one embodiment;

[0021] Figure 4 This is a schematic diagram illustrating the effect of guiding registration during implementation;

[0022] Figure 5 This is a schematic diagram illustrating the process of completing accuracy verification in one embodiment;

[0023] Figure 6 This is a schematic diagram illustrating the effect of guiding accuracy verification during implementation;

[0024] Figure 7 This is a schematic diagram illustrating the process of completing the incision marking in one embodiment;

[0025] Figure 8 This is a schematic diagram illustrating the effect of a guide cut marking during implementation;

[0026] Figure 9 This is a schematic diagram illustrating the process of adjusting the guide device in one embodiment;

[0027] Figure 10 This is a schematic diagram illustrating the effect of adjusting the guide device in one embodiment;

[0028] Figure 11 This is a schematic diagram illustrating the effect of guiding surgery in one embodiment;

[0029] Figure 12 This is a schematic diagram illustrating the effect of confirming the target surgical path in one embodiment;

[0030] Figure 13 This is a schematic diagram of a visual interface in one embodiment;

[0031] Figure 14 This is a schematic diagram of yet another visual interface in one embodiment;

[0032] Figure 15 This is a schematic diagram of yet another visual interface in one embodiment;

[0033] Figure 16 This is a schematic diagram of the overall workflow in one embodiment;

[0034] Figure 17 This is a structural block diagram of a mixed reality surgical navigation device in one embodiment;

[0035] Figure 18This is a schematic diagram of a mixed reality surgical navigation system in one embodiment;

[0036] Figure 19 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0038] Figure 1 This diagram illustrates the application environment of a mixed reality surgical navigation method in one embodiment. A user (e.g., a doctor) can wear a mixed reality device 100 and perform surgery on a target site based on the device. It should be noted that the target site can be a real patient area or a non-physical model, such as a phantom used for experiments or testing.

[0039] The mixed reality device 100 includes at least a glasses device 101 and a processing device ( Figure 1 (Not shown in the image) The processing device can communicate with the glasses device. The processing device sends the display content to be displayed to the glasses device, and the glasses device displays the display content. In this way, the user can see the real scene and the virtual display content through the glasses device.

[0040] The glasses device 101 includes, but is not limited to, virtual reality (VR) glasses, augmented reality (AR) glasses, etc. The processing device may include, but is not limited to, a central processing unit (CPU), a digital signal processor (DSP), a field-programmable gate array (FPGA), or other programmable logic devices.

[0041] In some embodiments, the processing device may also be implemented using a personal computer, laptop computer, server, or other similar device; this embodiment is not limited thereto. It is understood that... Figure 1 Taking the head as an example, the target area can also be other parts of the body.

[0042] In some embodiments, the mixed reality device 100 can also communicate with other servers to store or relay data that the mixed reality device 100 needs to use. The server can be a standalone server or a server cluster consisting of multiple servers.

[0043] Figure 1 The example only illustrates the application environment of one mixed reality surgical navigation method. In some application scenarios, the processing device can also project the content to be displayed directly into the real physical space, allowing users to directly see the real scene and the virtual display content through their eyes.

[0044] Figure 2 This is a flowchart illustrating a mixed reality surgical navigation method in one embodiment. In an exemplary embodiment, such as... Figure 2 As shown, a mixed reality surgical navigation method is provided. Taking the application of this method to the above-mentioned processing device as an example, it includes the following steps S201 to S202.

[0045] S201, Display the first mixed reality model of the target area, and guide the user to register based on the first mixed reality model to obtain the registration result.

[0046] In this embodiment, Mixed Reality (MRP) technology is a technique that merges the real world and the virtual world. In some embodiments, MRP technology is, for example, 3D holography. A MRP model is a virtual model superimposed on a real physical space (also known as real space). In one embodiment, the MRP model can also be called a 3D holographic model. The first MRP model is a MRP model of the target area, that is, a virtual model of the target area superimposed on real space.

[0047] Optionally, the size of the first mixed reality model is in a preset ratio to the size of the target part. For example, when the preset ratio is 1:1, it means that the size of the first mixed reality model and the target part are exactly the same.

[0048] Optionally, the first mixed reality model may include anatomical information of the target body part. For example, if the target body part is the head, the first mixed reality model may be a mixed reality model that can see the internal structure of the head.

[0049] The processing device can generate a first mixed reality model based on medical images of the target area. The medical images can be two-dimensional or three-dimensional, and include, but are not limited to, computed tomography (CT) images or magnetic resonance (MR) images; this embodiment does not impose any limitations.

[0050] Furthermore, the processing device is able to display a first mixed reality model of the target area. Figure 1For example, the processing device can send the first mixed reality model to the glasses device, which then displays the first mixed reality model. Of course, in some implementations, the processing device can also project the first mixed reality model near the target area; the following example primarily uses a mixed reality device. Other display processes are similar and will not be described further.

[0051] Therefore, the image seen by the user through the glasses device includes objects in the real physical space as well as a virtual first mixed reality model. In this way, the processing device can guide the user to register based on the first mixed reality model and obtain the registration result.

[0052] The registration result is used to represent the relationship between the actual physical space and the medical imaging space of the preoperative images. For example, the registration result can include the target transformation relationship between the real space coordinate system and the image coordinate system. The real space coordinate system is the coordinate system corresponding to the actual physical space, which can be determined based on the reference target corresponding to the target site during the registration process. The image coordinate system is the image coordinate system corresponding to the medical imaging space of the patient's preoperative images, which can be determined based on the medical images of the target site.

[0053] It should be noted that the processing device can display the first mixed reality model in any location, and the first mixed reality model can also follow the user's eye movement; this embodiment does not impose any restrictions.

[0054] S202, based on the registration results, the first mixed reality model is anchored to the target area for display, and the user is guided through the preparation process based on the first mixed reality model anchored to the target area, and surgical navigation is performed based on the first mixed reality model anchored to the target area.

[0055] In this embodiment, after obtaining the registration result, the processing device can anchor the first mixed reality model to the target location based on the registration result, that is, display the first mixed reality model anchored to the target location. For example, after the processing device anchors the first mixed reality model to the target location, the first mixed reality model seen by the user can overlay the target location in the real physical space.

[0056] It should be noted that in this embodiment, the processing device can anchor the first mixed reality model to the target part for display based on the registration result obtained in S201. In some application scenarios, the processing device can also anchor the first mixed reality model to the target part for display based on the registration result obtained in other ways. This embodiment is not limited to this.

[0057] Optionally, the first mixed reality model can have a certain degree of transparency, so that users can see both the first mixed reality model and the target part in the real physical space.

[0058] Then, the processing device can guide the user through the preparation process based on the first mixed reality model anchored to the target area. This preparation process may include at least one of the following: accuracy verification, incision marking, aseptic preparation, and guide device adjustment. Optionally, the processing device can output different prompts based on the first mixed reality model for different preparation processes to guide the user through the preparation process.

[0059] Furthermore, after completing the preparation process, users can use surgical instruments to perform the corresponding surgery. These surgical instruments may include, but are not limited to, puncture needles, biopsy needles, and other instruments used for puncture procedures.

[0060] In this way, users can see both the target area and surgical instruments in the real space, as well as the first mixed reality model that has been anchored to the target area. Users can perform surgery by observing the first mixed reality model, and the surgical navigation is guided by the first mixed reality model that has been anchored to the target area.

[0061] It should be noted that in some embodiments, when the first mixed reality model is anchored to the target area for display, the processing device can also adjust the display position of the first mixed reality model in response to the user-triggered movement operation, or hide the first mixed reality model (i.e., not display the first mixed reality model) in response to the user-triggered hiding operation. In this way, even if the first mixed reality model is automatically anchored to the target area after registration, the user can move or hide the first mixed reality model according to actual needs.

[0062] In the above embodiments, since a first mixed reality model of the target area can be displayed, and the user is guided to register based on the first mixed reality model to obtain the registration result, the user can see not only the real physical space but also the virtual first mixed reality model. Thus, guided by the first mixed reality model, registration can be completed efficiently and intuitively, resulting in a registration result. Furthermore, since the first mixed reality model can be anchored to the target area based on the registration result, the user can more intuitively understand the situation of the target area through the anchored first mixed reality model. This allows for efficient guidance of the user through the preparation process and surgical navigation based on the anchored first mixed reality model. During this process, the user does not need to view a two-dimensional image on the screen or switch their gaze back and forth, resulting in high interaction efficiency.

[0063] The registration process (also known as registration) is described below.

[0064] In one exemplary embodiment, optionally, the above-described mixed reality surgical navigation method further includes the following steps:

[0065] Display initial on-screen guidance information to guide users through the installation process before registration.

[0066] In this embodiment, the on-screen guidance information is information that guides the user in the visual dimension. It can be a two-dimensional graphic or a three-dimensional holographic image (i.e., a virtual image superimposed on a real physical space), and this embodiment is not limited to this. In one embodiment, optionally, the on-screen guidance information is three-dimensional guidance information. The first on-screen guidance information, the second on-screen guidance information, etc., mentioned below all belong to on-screen guidance information and will not be described again.

[0067] The guidance information on the first screen can be guidance information preset in the processing device, or it can be guidance information generated by the processing device based on the real physical environment. This embodiment is not limited to this.

[0068] Furthermore, the processing device can display first-screen guidance information to guide the user through the installation process before registration. For example, the first-screen guidance information may include guidance information for guiding the user to install the reference rack, or it may include guidance information for guiding the user to register by clicking on a dot; this embodiment is not limited to this.

[0069] In the above embodiments, since the first screen guidance information can be displayed, the user can be guided through the installation process before registration in a more intuitive and efficient manner based on the first screen guidance information, thereby improving the efficiency of registration.

[0070] In an exemplary embodiment, optionally, the above-mentioned "displaying first screen guidance information to guide the user through the installation process before registration based on the first screen guidance information" can be implemented in the following way:

[0071] The system displays the first sub-guide information to guide the user in installing the reference frame corresponding to the target part and the reference target on the reference frame; if the reference target is not detected, the system displays the second sub-guide information until the reference target is detected.

[0072] In this embodiment, the first screen guidance information includes first sub-guidance information and second sub-guidance information. The first sub-guidance information is used to guide the user in installing the reference frame and the reference target. Optionally, the first sub-guidance information may include at least one of the following: the installation location area, height, adjustment method of the reference frame, and instructions for installing the reference target on the reference frame.

[0073] Optionally, the processing device can display first sub-guided information in response to a user-triggered registration instruction. This registration instruction is triggered by the user through interaction and indicates that the user is about to begin registration. The processing device can also automatically determine the user's progress using visual sensors or similar methods, displaying the first sub-guided information when the user is about to begin registration.

[0074] Furthermore, the processing device is capable of detecting a reference target. Optionally, the processing device can identify whether the reference target is within the camera's field of view using a camera on the mixed reality device to determine whether a reference target has been detected. In some embodiments, the processing device may also detect the reference target using other sensors or other methods, and this embodiment is not limited thereto.

[0075] If no reference target is detected, the processing device will output a second sub-guidance message until a reference target is detected. The second sub-guidance message can be used to prompt the user that no reference target has been detected, or to prompt the user to readjust the reference frame.

[0076] In the above embodiments, since the first sub-guidance information can be displayed, the user can be efficiently guided to install the reference frame corresponding to the target part and the reference target on the reference frame based on the first sub-guidance information. Furthermore, since the second sub-guidance information can be displayed even if the reference target is not detected, until the reference target is detected, the user can be promptly reminded to make adjustments if the reference target is not installed properly, which helps to improve the efficiency of subsequent registration.

[0077] In one exemplary embodiment, optionally, the above-described mixed reality surgical navigation method further includes the following steps:

[0078] Upon detecting a reference target, a first mixed reality model is displayed, and the user is guided to register based on the first mixed reality model, thus obtaining the registration result.

[0079] In this embodiment, the processing device can display the first mixed reality model after detecting the reference target, so as to guide the user to register based on the first mixed reality model and obtain the registration result.

[0080] In one embodiment, if a reference target is detected, the processing device can output a detection success message. This embodiment does not limit the form of the detection success message; for example, if a reference target is detected, the processing device can display a three-dimensional holographic green highlight around the reference target.

[0081] In the above embodiments, since the first mixed reality model is displayed only after the reference target is detected, and the user is guided to register based on the first mixed reality model to obtain the registration result, the situation of unreasonable installation of the reference target can be avoided, which is conducive to improving the accuracy of registration.

[0082] In an exemplary embodiment, optionally, the above-mentioned "displaying a first mixed reality model of the target area, and guiding the user to register based on the first mixed reality model to obtain a registration result" may include the following steps:

[0083] The system displays a first mixed reality model and the data collection points on the first mixed reality model, and guides the user to register based on the first mixed reality model and the data collection points on the first mixed reality model, thus obtaining the registration result.

[0084] In this embodiment, the processing device can display a first mixed reality model and the acquisition points on the first mixed reality model. This embodiment does not limit the display format of the acquisition points on the first mixed reality model; they can be highlighted, geometrically marked, icons, or numerically marked, etc.

[0085] For example, the processing device can display a first mixed reality model upon detecting a reference target, and highlight the collection points that the user needs to collect on the first mixed reality model. As another example, the processing device can display the first mixed reality model in response to a user-triggered registration command, and highlight the collection points that the user needs to collect on the first mixed reality model.

[0086] In one embodiment, the processing device can display the first mixed reality model near the target location in a real physical space for easy comparison by the user.

[0087] Then, the processing device can guide the user to register based on the first mixed reality model and the acquisition points on the first mixed reality model, and obtain the registration result. For example, when the user sees the target part in the real physical space, the virtual first mixed reality model, and the acquisition points on the first mixed reality model, they can use a registration device to collect the corresponding acquisition points on the target part according to the position of the acquisition points on the first mixed reality model, in order to register. During the registration process, the processing device can obtain the registration result. The registration device includes, but is not limited to, probes.

[0088] In the above embodiments, since the first mixed reality model and the collection points on the first mixed reality model are displayed, the user can be intuitively guided to register based on the first mixed reality model and the collection points on the first mixed reality model, so as to obtain the registration result efficiently and accurately.

[0089] Figure 3 This is a schematic diagram of the process for determining the registration result in one embodiment. In an exemplary embodiment, such as... Figure 3 As shown, the above-mentioned "guiding users to register based on the first mixed reality model and the collection points on the first mixed reality model, and obtaining registration results" includes S301 to S303.

[0090] S301, in the case of displaying the first mixed reality model and the acquisition points on the first mixed reality model, in response to the user's acquisition operation based on the registered device, the real spatial coordinates of each acquisition point are determined.

[0091] In this embodiment, when the processing device displays the first mixed reality model and the collection points on the first mixed reality model, the user can see not only the target part in the real physical space, but also the first mixed reality model and the collection points on the first mixed reality model, which is beneficial for the user to initiate a collection operation through the registered device.

[0092] The data acquisition operation involves the user contacting each acquisition point with the end of the registered device. The processing device then responds to this acquisition operation and determines the true spatial coordinates of each acquisition point. These true spatial coordinates represent the coordinates of the acquisition point in the actual physical environment, such as the coordinates of the acquisition point in a real spatial coordinate system.

[0093] Optionally, the processing device can acquire the real spatial coordinates of each acquisition point through a sensor, which can be a position sensor set on the registration device or a camera on the mixed reality device.

[0094] For example, taking acquisition point A as an example, guided by the first mixed reality model and acquisition point A on the first mixed reality model, the user brings the end of the registration device into contact with acquisition point A corresponding to the target part in the real physical space. When the end of the registration device contacts acquisition point A, the processing device can obtain the real spatial coordinates of acquisition point A. Afterward, the user can acquire the next acquisition point until the processing device obtains the real spatial coordinates of all acquisition points.

[0095] S302, determine the registration error based on the actual spatial coordinates of each acquisition point and the medical images of the target site.

[0096] In this embodiment, the processing device is capable of acquiring medical images of the target site. Furthermore, after acquiring the true spatial coordinates of each acquisition point, the processing device can determine the registration error based on the true spatial coordinates of each acquisition point and the medical images of the target site. For example, the processing device can determine a candidate transformation relationship between the true spatial coordinate system and the image coordinate system based on the true spatial coordinates of each acquisition point and the medical images of the target site, and determine the predicted spatial coordinates of each acquisition point based on the candidate transformation relationship and the medical images of the target site, and determine the registration error based on the true spatial coordinates and predicted spatial coordinates of each acquisition point.

[0097] S303, in response to the user's confirmation of registration error, determines the registration result based on the actual spatial coordinates of each acquisition point and the medical image of the target site.

[0098] In this embodiment, after determining the registration error, the processing device can output the registration error, for example, by displaying it. If the user acknowledges the registration error, a confirmation operation is initiated. In this way, the processing device can respond to the user's confirmation operation and determine the registration result based on the actual spatial coordinates of each acquisition point and the medical image of the target site.

[0099] Optionally, the processing device can utilize a preset registration algorithm to register the real spatial coordinates of each acquisition point with the medical image of the target site, thereby obtaining the target transformation relationship. The preset registration algorithm includes, but is not limited to, deep learning algorithms and non-rigid registration algorithms.

[0100] In one embodiment, if the user does not accept the registration error, they can initiate a cancellation operation for the registration error, and the processing device can respond to the cancellation operation by returning to any previous step.

[0101] It is understandable that, taking the example of a user displaying a registration error, in some embodiments, the processing device may also provide voice prompts to indicate the registration error in response to the user's confirmation of the registration error.

[0102] In the above embodiments, since the real spatial coordinates of each acquisition point can be determined in response to the user's acquisition operation based on the registration device when displaying the first mixed reality model and the acquisition points on the first mixed reality model, and the registration error can be determined based on the real spatial coordinates of each acquisition point and the medical image of the target site, and the registration result can be determined based on the real spatial coordinates of each acquisition point and the medical image of the target site in response to the user's confirmation operation of the registration error, the registration result can be determined based on the real spatial coordinates of each acquisition point and the medical image of the target site. Therefore, the accuracy and efficiency of registration are improved.

[0103] In one exemplary embodiment, the mixed reality surgical navigation method described above further includes the following steps:

[0104] During the process of determining the true spatial coordinates of each acquisition point, if no reference target or registered target of the registered device is detected, the first target guidance information is output until the reference target and registered target of the registered device are detected.

[0105] In this embodiment, during process S301, that is, during the process of determining the true spatial coordinates of each acquisition point, the processing device will detect the reference target and the registration target. The process of detecting the reference target and the process of detecting the registration target can be referred to the above embodiment, and will not be repeated here. The registration target of the registered device includes, but is not limited to, the probe target of the probe.

[0106] If no reference target or registered target of the registered device is detected, the processing device will output first target guidance information until both are detected. This first target guidance information can be used to alert the user that no reference target or registered target was detected during the acquisition process, or to guide the user to readjust the reference target or registered target. For example, during the determination of the true spatial coordinates of each acquisition point, if no reference target is detected, the processing device can display a three-dimensional holographic red highlight around the reference target and indicate the area requiring adjustment.

[0107] In the above embodiments, if no reference target or registered target of the registered device is detected during the process of determining the true spatial coordinates of each acquisition point, the first target guidance information is output until the reference target and registered target of the registered device are detected. Therefore, the situation of target loss can be reduced during the acquisition process, and the navigation efficiency can be improved.

[0108] In one exemplary embodiment, optionally, the processing device may output acquisition prompts. These prompts may be used to indicate at least one of the following: the acquisition progress of each acquisition point, the acquisition points already acquired on the first mixed reality model, and the acquisition points not yet acquired. This embodiment does not limit the form of the output acquisition prompts; they may be visual prompts from mixed reality, or other forms of prompts such as voice.

[0109] Figure 4 This is a schematic diagram illustrating the effect of guided registration during implementation. The following will combine... Figure 4 Describe the process of guiding users through the registration process. For example... Figure 4 As shown, the visualization interface corresponding to the processing device may include a first display area, which may be the display area of ​​the first mixed reality model.

[0110] In some embodiments, the visualization interface corresponding to the processing device may further include a second display area and a third display area. The second display area is a display area for two-dimensional images, such as displaying multi-planar reconstruction (MPR) two-dimensional images. In some embodiments, the second display area may also display basic image information, such as patient name, image slice thickness, number of slices, and image magnification. The third display area is a display area for conventional interface prompts and explanations.

[0111] In other words, the visual interface corresponding to the processing device can include at least one display area, and different display areas can display different types of content.

[0112] During the registration process, the first and third display areas are activated, meaning they are put into use. The first display area is used to display the initial guidance information, which includes guidance information for three stages: reference frame installation guidance (i.e., "patient reference frame adjustment guidance" in the figure), reference frame adjustment guidance (i.e., "patient reference frame installation guidance" in the figure), and 3D holographic operation guidance for the registration process (i.e., "dot registration 3D holographic guidance" in the figure). The third display area displays important adjustment reminders.

[0113] Upon registration, the processing device displays a 3D holographic image in the first display area, highlighting the reference frame installation guide (e.g., showing the installation location and height of the reference frame), while other aspects are grayed out. Once the user has installed the reference frame, the guidance for that aspect disappears. The processing device then displays a notification in the third display area indicating that the reference frame installation guide is complete, accompanied by a voice announcement. Subsequently, the user installs the target on the reference frame. The camera on the mixed reality device automatically detects whether the reference target is within the field of view. If it is, the processing device displays a 3D holographic green highlight around the reference target. If it is not within the field of view, the processing device displays a 3D holographic image in the first display area guiding the user to adjust the reference target (e.g., showing the correct location and height of the reference target). After adjustment, the processing device displays a 3D holographic green highlight around the reference target.

[0114] The third step then proceeds. Taking the registration process with the head as the target area as an example, the processing device projects a first mixed reality model onto the vicinity of a real head and highlights the points the user needs to collect on the model. The user then uses a dot-mapping probe to perform the collection operation on the real patient's head, guided by the dot-mapping probe. The first mixed reality model displays completed collection points using other markers; for example, uncollected points are marked in red, and collected points are marked in green, while a collection progress bar is also displayed.

[0115] During the acquisition process, both the reference target and the probe target need to be in the same field of view of the camera. If the reference target or the probe target is lost, the processing device will display a three-dimensional holographic red highlight and indicate the area that needs to be adjusted.

[0116] Furthermore, after all data collection points are collected, the processing device displays the registration error in the third display area. After the user confirms that there are no errors, the user proceeds to the next preparation process, such as entering the accuracy verification.

[0117] The above examples mainly use dotted registration as the registration method. In some application scenarios, the registration method can also be other methods, such as non-contact optical marker registration or non-contact facial registration.

[0118] In non-contact optical marker registration, markers are typically affixed to the target site (e.g., the patient's face), and registration is performed based on the markers in preoperative images and those captured by intraoperative cameras to obtain the registration results.

[0119] In this registration method, the processing device can display marker information at the corresponding location on the first mixed reality model.

[0120] In non-contact facial registration, doctors can use a handheld structured light camera with a target to perform point cloud scanning on the target area. The scanned results are then registered with the preoperative images to obtain the registration result. This structured light camera can be different from the camera used in mixed reality devices.

[0121] In this registration method, the processing device can display a recommended area to be scanned on the first mixed reality model on display. The area to be scanned is a non-deformable bony area, such as the area from the patient's forehead to the tip of the nose, to guide the user to move the structured light camera to the appropriate position to perform point cloud scanning on the target area.

[0122] In some application scenarios, the first mixed reality model may not need to be displayed during the registration process. The processing device only needs to provide registration guidance information to guide the user to register in both the medical imaging space and the real physical space, thereby obtaining the registration result.

[0123] The preparation process is described below. In an exemplary embodiment, optionally, the "guiding the user through the preparation process based on the first mixed reality model anchored to the target location" in S202 above includes at least one of the following processes:

[0124] (1) Guide the user to complete the accuracy verification based on the first mixed reality model anchored to the target location. Accuracy verification is the process of verifying the accuracy of the registration. For example, the processing device can display the first mixed reality model and the verification points on the first mixed reality model, and guide the user to perform accuracy verification based on the first mixed reality model anchored to the target location and the verification points on the first mixed reality model.

[0125] (2) The user is guided to complete the incision marking based on the first mixed reality model anchored to the target site. Incision marking is the process by which the user designs the incision on the target site based on the target surgical path.

[0126] (3) Guide the user to complete aseptic preparation based on the first mixed reality model anchored to the target site. Aseptic preparation is the process by which the user sterilizes the real physical space of the target site.

[0127] (4) The user is guided to adjust the guide device based on the first mixed reality model anchored to the target location. The guide device adjustment is the process of installing and adjusting the guide device. The guide device includes, but is not limited to, the guide.

[0128] In the above embodiments, since the user can be guided to complete at least one of the following tasks based on the first mixed reality model anchored to the target area: accuracy verification, incision marking, aseptic preparation, and guide device adjustment, the preparation process can be completed flexibly and efficiently.

[0129] In some application scenarios, the first mixed reality model may not need to be displayed during the preparation process. The processing device only needs to provide preparation guidance information to guide the user through the preparation process after registration based on the registration guidance information.

[0130] The following describes the accuracy verification process in some embodiments.

[0131] Figure 5 This is a schematic diagram illustrating the process of completing accuracy verification in one embodiment. In an exemplary embodiment, such as... Figure 5 As shown, the above-mentioned "guiding the user to complete the accuracy verification based on the first mixed reality model anchored to the target location" includes S501 to S503.

[0132] S501, when the verification point is displayed on the first mixed reality model, in response to the user's verification operation on the current verification point based on the registered device, the real spatial coordinates of the current verification point are determined.

[0133] In this embodiment, the display format of the verification points on the first mixed reality model is not limited.

[0134] Optionally, the processing device can display verification points on the first mixed reality model in response to a verification command triggered by the user. The verification command is an instruction triggered by the user through interaction, indicating that the user is about to enter accuracy verification. The processing device can also automatically determine the user's progress using visual sensors or other means, and display verification points on the first mixed reality model when the user is about to enter accuracy verification.

[0135] When verification points are displayed on the first mixed reality model, users not only see the target body part in the real physical space, but also the first mixed reality model anchored to the target body part and the verification points on the first mixed reality model. This facilitates users initiating verification operations through the registered device. The verification points can be anatomical landmarks on the target body part, such as the inner and outer canthi, or the tip of the nose. The verification points can be the same as or different from the acquisition points.

[0136] The verification operation is an action performed by the user using the end of the registration device to contact the current verification point. Subsequently, the processing device can respond to this verification operation and determine the true spatial coordinates of the current verification point. The process of determining the true spatial coordinates of the current verification point is similar to S301. For example, guided by the first mixed reality model and verification point A on the first mixed reality model, the user causes the end of the registration device to contact the verification point A corresponding to the target location in the real physical space. With the end of the registration device contacting verification point A, the processing device obtains the true spatial coordinates of verification point A.

[0137] S502, determine the verification error of the current verification point based on the actual spatial coordinates of the current verification point and the medical image of the target area.

[0138] Continuing with the example in S501 above, the processing device can determine the verification error of verification point A based on the actual spatial coordinates of verification point A and the medical image of the target area. The principle for determining the verification error of the current verification point is similar to the principle for determining the registration error, and will not be repeated here.

[0139] S503, in response to the user's confirmation operation on the verification error of the current verification point, for the next verification point, return to the step of determining the true spatial coordinates of the current verification point in response to the user's verification operation on the current verification point based on the registered instrument, until the confirmation operation on the verification error of all verification points is completed, so as to complete the accuracy verification.

[0140] The processing device displays the verification error of the current verification point. If the user acknowledges the verification error, a confirmation operation for the verification error is initiated. In this way, the processing device can respond to the user's confirmation operation for the verification error and return to step S501 to determine the true spatial coordinates of the next verification point, until the verification error confirmation operation for all verification points is completed, which means the accuracy verification is completed.

[0141] Continuing with the example of S502 above, after the user initiates a confirmation operation for the verification error of verification point A, they can continue to initiate a verification operation for the next verification point B based on the registered device, so that the processing device can obtain the true spatial coordinates of verification point B. Then, the processing device determines the verification error of verification point B, so that the user can initiate a confirmation operation for the verification error of verification point B, and so on, until the user confirms the verification error of the last verification point.

[0142] In the above embodiments, when the verification points are displayed on the first mixed reality model, in response to the user's verification operation on the current verification point based on the registered device, the true spatial coordinates of the current verification point are determined; based on the true spatial coordinates of the current verification point and the medical image of the target site, the verification error of the current verification point is determined; in response to the user's confirmation operation on the verification error of the current verification point, the steps of determining the true spatial coordinates of the current verification point in response to the user's verification operation on the current verification point based on the registered device are repeated for the next verification point until the confirmation operation on the verification error of all verification points is completed to complete the accuracy verification. Therefore, the first mixed reality model can intuitively and efficiently guide the user to complete the accuracy verification, thereby improving the accuracy of the determined target transformation relationship and benefiting the accuracy of subsequent navigation.

[0143] In one exemplary embodiment, the above-described mixed reality surgical navigation method may optionally include the following steps:

[0144] If the end of the registered device is detected to be in contact with the current calibration point, a first cross-sectional guide image is displayed; the first cross-sectional guide image is a cross-sectional image of the target part at the first contact position; the first contact position is the contact position between the end of the registered device and the target part.

[0145] In this embodiment, the processing device can detect whether the end of the registration device is in contact with the current verification point by using a capacitor disposed at the end of the registration device. For example, the processing device can confirm that the end of the registration device is stably in contact with the current verification point if the capacitance value is greater than a preset capacitance value and is maintained for a preset duration.

[0146] Furthermore, when the processing device detects that the tip of the registered device is in contact with the current verification point, it can display a first-section guide image. The first-section guide image is a cross-sectional image of the target site at the first contact position; the first contact position is the contact point between the tip of the registered device and the target site. For example, the processing device can display a two-dimensional image cross-section of the layer where the probe and patient contact position are located.

[0147] In the above embodiments, when the end of the registered instrument is detected to be in contact with the current calibration point, a first cross-sectional guide image is displayed; the first cross-sectional guide image is a cross-sectional image of the target part at the first contact position; the first contact position is the contact position between the end of the registered instrument and the target part. Therefore, the user can intuitively understand the status of the current calibration point through the first cross-sectional guide image, which is beneficial for the user to perform accuracy calibration.

[0148] In one exemplary embodiment, the above-described mixed reality surgical navigation method may optionally include the following steps:

[0149] Display verification information during the accuracy verification process; the verification information includes at least one of the following: verification status, verification position of the current verification point, and verification error.

[0150] In this embodiment, the verification status refers to the verification status of the registered device, such as being verified or not being verified. The verification location refers to the location of the current verification point, such as the tip of the nose, the inner canthus, or the outer canthus. The verification error is the verification error determined in S502.

[0151] Optionally, the processing device can display verification information based on the location of the registered device. For example, the processing device can display verification information at the tail end of the registered device.

[0152] In some embodiments, the processing device may also display a fourth mixed reality model of the registered medical device. The fourth mixed reality model may have the same or similar appearance as the registered medical device, or may use other shapes, such as a cylinder. The processing device may obtain the real-time position of the registered medical device through a device target, and render the fourth mixed reality model based on the real-time position of the registered medical device for display. In one embodiment, the processing device may also anchor the fourth mixed reality model to the registered medical device for display based on its real-time position.

[0153] In the above embodiments, since the verification information of the current verification point can be displayed during the accuracy verification process, and the verification information includes at least one of the following: verification status, verification position of the current verification point, and verification error, users can understand the accuracy verification status in a timely manner, thus improving interaction efficiency.

[0154] In an exemplary embodiment, optionally, during the accuracy verification process, if the processing device does not detect the reference target or the registered target, it outputs second target guidance information until the reference target and the registered target are detected. In other words, during processes S501 to S503, the processing device can still continue to detect the reference target and the registered target. The process of detecting the reference target and the registered target can refer to the above embodiment, and will not be repeated here.

[0155] Similarly, the second target guidance information can be used to notify the user that no reference target or registered target was detected during the acquisition process, or to guide the user to readjust the reference target or registered target. Optionally, the second target guidance information can be three-dimensional holographic information.

[0156] In one exemplary embodiment, optionally, the processing device may output a verification prompt, wherein the verification prompt may be used to indicate at least one of the following: the verification progress of each verification point, the verification points verified on the first mixed reality model, and the verification points not yet verified.

[0157] Figure 6 This is a schematic diagram illustrating the effect of guided precision verification in one embodiment. The following will combine... Figure 6 This section describes the process of guiding users through accuracy verification. For example... Figure 6 As shown, during the accuracy verification process, the processing device anchors the first mixed reality model to the target area based on the registration results. At this time, the user's focus is on the first and second display areas. In the first display area, the processing device anchors the first mixed reality model to the real patient's head and displays recommended verification points using holographic high-brightness indicators. Verification points marked in red are those to be verified, while those marked in green are those that have been verified.

[0158] Next, the user holds the probe and touches the patient's calibration point with the tip of the probe. As the probe's position changes, the processing device displays a two-dimensional image cross-section of the contact layer between the probe and the patient in the second display area. Furthermore, the processing device displays a menu bar at the end of the probe, showing the calibration status, the current calibration position, and the calibration error.

[0159] The processing device displays a verification progress bar to ensure that the probe tip remains stably on the current verification point during the verification process. After the progress bar completes, the device displays the verification error for that point, allowing the user to determine if the error supports the surgery. If no problem is found, the process moves to the next verification point. After completing all verification points, the process proceeds to the next stage, such as incision marking.

[0160] Similarly, during the verification process of the calibration points, it is necessary to ensure that the reference target and the probe target are in the same field of view of the camera. If the reference target or the probe target is lost, the processing device will display a red highlight in the three-dimensional hologram and indicate the area that needs to be adjusted.

[0161] The following describes the incision marking process in some embodiments.

[0162] Figure 7This is a schematic diagram illustrating the process of completing the incision marking in one embodiment. In an exemplary embodiment, such as... Figure 7 As shown, the above-mentioned "guiding the user to complete the incision marking based on the first mixed reality model anchored to the target part" includes S701 to S703.

[0163] S701, based on the first mixed reality model, displays the first safe zone where the target surgical path is located.

[0164] In this embodiment, the target surgical path includes a puncture point and a target point. The puncture point can be understood as the starting point of the path, such as the entry point into the skull. The target point can be understood as the ending point of the path, and can represent any location of interest within the target area. The target surgical path can be a path sent in advance by other devices to the processing device, a path generated by the processing device, or a path set by the user based on the first mixed reality model; this embodiment is not limited to these limitations.

[0165] The first safety zone can be a pre-defined area. The first safety zone can be of any shape or size, and this embodiment is not limited thereto. For example, the processing device can display the first safety zone as an orange columnar area of ​​the target surgical path.

[0166] In this way, when the first safe zone where the target surgical path is located is displayed based on the first mixed reality model, the user can see the target body part and surgical instruments in the real physical space, the first mixed reality model anchored to the target body part, and the first safe zone where the target surgical path is located.

[0167] Optionally, the processing device can respond to a user-triggered incision marking command and display a first safe area containing the target surgical path based on a first mixed reality model. The incision marking command is triggered by the user through interaction and indicates that the user is about to enter the incision mark. The processing device can also automatically determine the user's progress using visual sensors or other means to display the first safe area when the user is about to enter the incision mark.

[0168] In one exemplary embodiment, optionally, during the incision marking process, the processing device may also display at least one of the following: a second mixed reality model corresponding to the surgical instruments, the target surgical path, and a region of interest in the first mixed reality model.

[0169] The region of interest in the first mixed reality model can be a lesion within the model. For example, when displaying the first mixed reality model, the processing device can mark the lesion in red, which is beneficial for the user to mark incisions later.

[0170] S702, when the surgical instrument is detected to have entered the first safety range, displays the difference between the end of the surgical instrument and the puncture point in the target surgical path.

[0171] In this embodiment, the processing device can detect whether surgical instruments have entered a first safe area. Optionally, the processing device can determine whether surgical instruments have entered the first safe area using a camera on a mixed reality device.

[0172] Furthermore, upon detecting that the surgical instrument has entered the first safe zone, the processing device can display the difference between the tip of the surgical instrument and the puncture point in the target surgical path. This difference can be an angular difference and / or a positional difference. This embodiment does not limit the form of display.

[0173] S703, when the difference between the end of the surgical instrument and the puncture point is less than a first preset difference, outputs a marking prompt message, which is used to guide the user to complete the incision marking.

[0174] Furthermore, the processing device can detect the difference between the tip of the surgical instrument and the puncture point. Similarly, the processing device can determine the difference between the tip of the surgical instrument and the puncture point using a camera on a mixed reality device. Then, if the difference between the tip of the surgical instrument and the puncture point is less than a first preset difference, it indicates that the tip of the surgical instrument is sufficiently close to the puncture point, and a marking prompt message can be output. The marking prompt message can be in the form of voice, 3D holographic image, 2D image, etc., and this embodiment is not limited to these.

[0175] For example, if the difference between the tip of the surgical instrument and the puncture point is less than a first preset difference, the processing device can give a voice prompt indicating that it is approaching and has reached the cranial entry point. In this way, the user can use a marker to mark the lesion and skull points and design the skin incision under the guidance of the marking prompt information.

[0176] In the above embodiments, since the first safe zone of the target surgical path is displayed based on the first mixed reality model, and the difference between the tip of the surgical instrument and the puncture point in the target surgical path is displayed when the surgical instrument is detected to have entered the first safe zone, the user can be guided to efficiently bring the tip of the surgical instrument closer to the puncture point in the target surgical path. Furthermore, when the difference between the tip of the surgical instrument and the puncture point is detected to be less than a first preset difference, a marking prompt is output, which can guide the user to complete the incision marking in a timely and accurate manner.

[0177] In one exemplary embodiment, the above-described mixed reality surgical navigation method may optionally include the following steps:

[0178] When the tip of the surgical instrument contacts the target site, a second sectional guide image is displayed; the second sectional guide image is a sectional image of the target site at the second contact position; the second contact position is the contact position between the tip of the surgical instrument and the target site.

[0179] In this embodiment, the processing device can detect whether the end of the surgical instrument is in contact with the target area by using a capacitor disposed at the end of the surgical instrument. In some embodiments, the processing device can also determine whether the end of the surgical instrument is in contact with the target area by a visual sensor such as a camera. This embodiment is not limited thereto.

[0180] Furthermore, when the processing device detects that the tip of the surgical instrument has contacted the target site, it can display a second-section guided image. This second-section guided image is a cross-sectional image of the target site at the second contact position; the second contact position is the point of contact between the tip of the surgical instrument and the target site. For example, the processing device can display a two-dimensional image cross-section of the layer where the probe and patient contact position are located.

[0181] In the above embodiments, a second sectional guide image is displayed when the tip of the surgical instrument contacts the target area. The second sectional guide image is a sectional image of the target area at the second contact position; the second contact position is the contact position between the tip of the surgical instrument and the target area. Therefore, the user can intuitively understand the incision situation through the second sectional guide image, which is beneficial for the user to mark the incision.

[0182] Figure 8 This is a schematic diagram illustrating the effect of guiding the incision marking during implementation. The following will combine... Figure 8 This section describes the process of guiding users to mark cuts. For example... Figure 8 As shown, the processing device can display a second mixed reality model and mark the lesions extracted before surgery in red, the target surgical path in green, and the first safe zone corresponding to the target surgical path in orange bar-shaped areas within the first mixed reality model. Taking a probe as an example, the user can hold the probe and find the direction and angle of the probe aligning with the target surgical path through interface prompts. After the probe enters the first safe zone, the processing device displays the distance and angle between the probe tip and the entry point in real time. When the probe tip and the entry point are aligned, the processing device provides a voice prompt indicating that the entry point is approaching or reached, allowing the user to use a marker to mark the lesions and skull points, design the skin incision, and proceed to the next step, such as aseptic preparation.

[0183] When entering the sterile preparation phase, users need to disinfect and drape the instruments as needed, and replace the relevant sterile targets and install sterile covers. At this time, the processing device can also provide three-dimensional holographic guidance or voice interaction guidance based on the first mixed reality model. For example, the processing device displays the sterile instruments and consumables that need to be replaced and the instruments that need to be fitted with sterile covers in the first and third display areas and broadcasts them by voice, and provides voice prompts after the sterile preparation is completed.

[0184] The following describes the adjustment of the guide device in some embodiments.

[0185] Figure 9 This is a schematic diagram illustrating the process of adjusting the guide device in one embodiment. In an exemplary embodiment, such as... Figure 9 As shown, the above-mentioned "guiding the user to complete the adjustment of the guidance device based on the first mixed reality model" includes S901 to S903.

[0186] S901, based on the first mixed reality model, displays second-screen guidance information; the second-screen guidance information is used to guide the user to install the guidance device and the guidance target of the guidance device.

[0187] In this embodiment, the second screen guidance information may include at least one of the following: the installation location area, height, adjustment method of the guide device, and guidance on installing a guide target on the guide device.

[0188] Optionally, the second screen guidance information can be guidance information preset in the processing device in advance, or guidance information generated by the processing device based on the first mixed reality model. This embodiment is not limited to this.

[0189] Furthermore, when adjusting the guide device, the processing equipment will not only display a first mixed reality model anchored to the target location, but also a second screen displaying guidance information. In this way, the user can see not only the target location in the real physical space and the first mixed reality model anchored to the target location, but also the second screen displaying guidance information. Consequently, guided by the first mixed reality model and the second screen displaying guidance information, the user can install the guide device and its target.

[0190] Optionally, the processing device can respond to a user-triggered adjustment command for the guidance device and display guidance information on a second screen. This adjustment command is triggered by the user through interaction and indicates that the user is about to enter the guidance device adjustment area. The processing device can also automatically determine the user's progress using visual sensors or similar methods, and display guidance information on a second screen when the user is about to enter the guidance device adjustment area.

[0191] S902, upon detecting the guide target, displays third-screen guidance information to guide the user in adjusting the guide device based on the third-screen guidance information.

[0192] In this embodiment, the processing device detects the guide target. Optionally, the processing device can identify whether the guide target is within the field of view of the camera on the mixed reality device to determine whether the guide target has been detected.

[0193] Furthermore, if a guide target is detected, the processing device will display third-screen guidance information, allowing the user to adjust the guide device under the guidance of this information. This third-screen guidance information may be, for example, an adjustment guidance video for the guide device.

[0194] Optionally, the user can adjust the guide device using an adjustment mechanism. The adjustment mechanism may include a coarse adjustment mechanism and / or a fine adjustment mechanism.

[0195] In the above embodiments, since the second screen guidance information can be displayed based on the first mixed reality model, the second screen guidance information can guide the user to install the guide device and the guide target of the guide device. Furthermore, when the guide target is detected, the third screen guidance information is then displayed, which allows the user to intuitively and efficiently complete the adjustment of the guide device based on the third screen guidance information.

[0196] In an exemplary embodiment, optionally, the third-screen guidance information includes third sub-guidance information, a second safety range, and joint adjustment information. The aforementioned "displaying the third-screen guidance information to guide the user to complete the adjustment of the guide device based on the third-screen guidance information" can be achieved in the following way:

[0197] The system displays the third sub-guidance information and the second safety range where the target surgical path is located, so as to guide the user to adjust the end of the guide device to the second safety range based on the third sub-guidance information and the second safety range; it displays joint adjustment information and guides the user to adjust the guide device based on the joint adjustment information, so that the difference between the path where the end of the guide device is located and the target surgical path is less than the second preset difference, so as to complete the adjustment of the guide device.

[0198] In this embodiment, when the processing device detects a guide target, it displays a second safety zone and a third sub-guide information indicating the location of the target surgical path. The second safety zone is similar to the first safety zone; the second safety zone containing the target surgical path may be the same as or different from the first safety zone.

[0199] The third sub-guidance information is, for example, a holographic highlight indicator for adjusting the guide device and an operation animation. In this way, the user can adjust the guide device to the second safety range under the guidance of the third sub-guidance information and the second safety range.

[0200] In one embodiment, the processing device can further detect whether the guide channel of the guiding device is within the second safety range. Further optionally, if the guide channel is within the second safety range, a success message can be output. If the guide channel is not within the second safety range, a failure message can be output. For example, after the guide channel of the guiding device is within the second safety range, the processing device can output a bright green holographic message.

[0201] Furthermore, the processing device will also display joint adjustment information. Optionally, the processing device can display the joint adjustment information when the end of the guide device is within a second safe range. This joint adjustment information guides the user in adjusting the guide device and may include adjustment information for the joints that need adjustment, such as the required angle and distance between the end of the guide device and the target surgical path.

[0202] In this way, the user can adjust the guide device based on the joint adjustment information, so that the difference between the path of the end of the guide device and the target surgical path is less than the second preset difference, thereby completing the adjustment of the guide device.

[0203] It should be noted that the end of the guide device can also be understood as the guide channel of the guide device, and the path of the end of the guide device is the path of the guide channel. For ease of description, it will be referred to as the actual path below. During the adjustment of the guide device, the user needs to adjust the difference between the actual path and the target surgical path to be less than the second preset difference.

[0204] The difference between the actual surgical path and the target surgical path can be determined based on the degree of overlap between the two paths, or based on the angle and distance between the two paths; this embodiment is not limited to this. The second preset difference can be set according to actual needs, for example, as a number close to 0. In one application scenario, the difference between the actual path and the target surgical path is less than the second preset difference, which may be due to the actual path and the target surgical path overlapping.

[0205] In the above embodiments, since the third sub-guidance information and the second safety range where the target surgical path is located can be displayed, the user can be guided to adjust the end of the guide device to the second safety range based on the third sub-guidance information and the second safety range. Furthermore, by displaying joint adjustment information, the user can be guided to adjust the guide device based on the joint adjustment information. For example, the user can adjust each joint sequentially, thereby making the difference between the path where the end of the guide device is located and the target surgical path less than a second preset difference, so as to efficiently and accurately complete the adjustment of the guide device under guidance.

[0206] In one exemplary embodiment, optionally, the third-screen guidance information includes a baseline graphic and a variation graphic. The aforementioned "displaying the third-screen guidance information to guide the user to complete the adjustment of the guidance device based on the third-screen guidance information" can be achieved in the following way:

[0207] Displays the reference graphic corresponding to the observation path and the variation graphic corresponding to the non-observation path; responds to the user's adjustment operation on the guide device, updates the shape of the variation graphic and / or the relative position between the variation graphic and the reference graphic to guide the user to complete the adjustment of the guide device; wherein, the reference graphic is obtained by projecting the observation path along its own extension direction, the variation graphic is obtained by projecting the non-observation path along the extension direction of the observation path, the observation path is either the target surgical path or the path where the end of the guide device is located, and the non-observation path is a path different from the observation path among the target surgical path and the path where the end of the guide device is located.

[0208] In this embodiment, one of the target surgical path and the actual path is designated as the observation path, and the other as the non-observation path. That is, the observation path is either the target surgical path of the surgical instrument or the actual path corresponding to the guiding device, while the non-observation path is the path that differs from the observation path between the target surgical path and the actual path.

[0209] For example, the target surgical path can be used as the observation path, and the non-observation path as the actual path. Alternatively, the actual path can be used as the observation path, and the non-observation path as the target surgical path. It's understood that the observation path is the path displayed as a reference in the visualization interface, while the non-observation path is the path displayed in the visualization interface with the observation path as the baseline.

[0210] When it is necessary to align the actual surgical path with the target surgical path, the user will adjust the guide device. This adjustment process can include coarse adjustment and / or fine adjustment. It is understood that the precision of the fine adjustment process is higher than that of the coarse adjustment process; that is, the relative pose difference between the two paths after fine adjustment is lower than the relative pose difference after coarse adjustment.

[0211] Optionally, the user can adjust the guide device by directly operating it, or by using its adjustment mechanism. Understandably, during adjustment, the guide device's position will change, thus altering the actual path.

[0212] Furthermore, during the adjustment of the guiding device, the processing equipment displays a reference graphic corresponding to the observation path and a variation graphic corresponding to the non-observation path. The reference graphic is obtained by projecting the observation path along its own extension direction, while the variation graphic is obtained by projecting the non-observation path along the extension direction of the observation path. It is understandable that, since the actual path may change, the variation graphic corresponding to the non-observation path will also change.

[0213] During the user's adjustment of the guide device, the processing device can also respond to the user's adjustment operation. Furthermore, in response to this adjustment operation, the processing device updates the shape of the changed graphic and / or the relative position between the changed graphic and the reference graphic. Specifically, the processing device can update the shape of the changed graphic in response to the user's angle adjustment operation of the guide device, and it can also update the relative position between the changed graphic and the reference graphic in response to the user's position adjustment operation of the guide device. In some embodiments, the processing device can also respond to the adjustment operation to update both the shape of the changed graphic and the relative position between the changed graphic and the reference graphic. For example, the adjustment mechanism may include a universal adjustment structure, through which the angle and position of the guide device can be adjusted simultaneously.

[0214] For example, taking the observation path as the target surgical path and the non-observation path as the actual path as an example, the reference graphic corresponding to the target surgical path can be a dot graphic. If the non-observation path is not parallel to and coincides with the observation path, the changing graphic can be a line graphic. Then, when the user adjusts the guide device, he first performs the angle adjustment operation, then the position adjustment operation, and observes the changing graphic. When the changing graphic is updated to a dot graphic, and the dot graphic corresponding to the changing graphic coincides with the dot graphic corresponding to the reference graphic, the difference between the actual path and the target surgical path is less than the second preset difference, and the user can complete the adjustment of the guide device.

[0215] If the position adjustment is performed first, followed by the angle adjustment, and the dotted graphic corresponding to the reference graphic is located on the line graphic corresponding to the changing graphic, and then the line graphic corresponding to the changing graphic gradually changes into a dotted graphic, and the dotted graphic corresponding to the changing graphic coincides with the dotted graphic corresponding to the reference graphic, then the difference between the actual path and the target surgical path will be less than the second preset difference, and the user can complete the adjustment of the guide device.

[0216] The above embodiments can display a reference graphic corresponding to the observation path and a variation graphic corresponding to the non-observation path. In response to the user's adjustment operation on the guide device, the shape of the variation graphic and / or the relative position between the variation graphic and the reference graphic are updated to guide the user in adjusting the guide device. Since the reference graphic is obtained by projecting the observation path along its own extension direction, and the variation graphic is obtained by projecting the non-observation path along the extension direction of the observation path, the observation path is either the target surgical path or the path where the end of the guide device is located, while the non-observation path is a path different from the observation path among the target surgical path and the path where the end of the guide device is located. Therefore, by observing the reference graphic and the variation graphic, the user can intuitively understand the relative situation between the target surgical path and the actual path, thereby enabling more precise and planned adjustment of the guide device and improving the adjustment efficiency of the guide device.

[0217] In one exemplary embodiment, the above-described mixed reality surgical navigation method may optionally include the following steps:

[0218] Based on the real-time position of the guide device, the third mixed reality model at the end of the guide device is anchored to the end of the guide device for display.

[0219] In this embodiment, the third mixed reality model is a mixed reality model of the guide channel. The third mixed reality model can be the same as or similar to the end of the guide device, or it can have other shapes.

[0220] Optionally, the processing device can determine the real-time position of the guiding device based on the real-time position of the guiding target, and render a third mixed reality model based on the real-time position of the guiding device. This third mixed reality model is then anchored to the end of the real guiding device for display. In this way, guided by the third mixed reality model, users can adjust the guiding device more intuitively and efficiently based on the model they see.

[0221] In the above embodiments, since the third mixed reality model at the end of the guide device is anchored to the end of the guide device for display according to the pose of the guide device, the user can more intuitively understand the current adjustment status through the third mixed reality model.

[0222] Figure 10 This is a schematic diagram illustrating the effect of adjusting the guide device in one embodiment. The following will be combined with... Figure 10 This describes the process of guiding users to adjust the guide device. For example... Figure 10 As shown, when adjusting the guide device, the user needs to install the guide according to the complexity and accuracy of the procedure, and adjust it according to the interface guidance.

[0223] First, the user secures the guide bracket to the patient's head frame, assembles the joints, and installs the guide target. At this point, the processing device anchors the first mixed reality model in the first display area onto the patient's actual head and marks the target surgical path in green. When the processing device recognizes the guide target, it displays a green safety zone around the target surgical path, shows a holographic highlight indicator for guide adjustment in the first display area, and displays an operation animation in the third display area, indicating that the guide should be adjusted to the green safety range near the path.

[0224] Once the guide is adjusted to the green safety range near the path, the processing device displays a pass mark on the corresponding joint of the guide (e.g., a bright holographic green highlight on the coarse adjustment mechanism of the guide). Subsequently, the processing device displays the differences from the target surgical path on other joints requiring adjustment, marked in red, indicating, for example, that the angle needs further adjustment by XX degrees (°) or the distance by XX millimeters (mm). Following the holographic highlight guidance, the user adjusts the corresponding joints to the correct position. Once the differences from the target surgical path are marked in green, all joints of the guide are locked, and the guide adjustment is complete.

[0225] The following describes the surgical procedure in some embodiments. During the procedure, the user needs to hold the surgical instrument and complete the puncture within the puncture channel of the guide device. It should be noted that this surgical procedure can be a surgical simulation or training process.

[0226] In one exemplary embodiment, the above-described mixed reality surgical navigation method may optionally include the following steps:

[0227] Based on the real-time position and registration results of the surgical instruments, a two-dimensional guidance image is displayed to enable surgical navigation based on the two-dimensional guidance image and a first mixed reality model anchored to the target site; the two-dimensional guidance image is used to characterize the relative positional relationship between the surgical instruments and the target site.

[0228] In this embodiment, during the procedure, the relative positional relationship between the surgical instrument and the target area changes as the user performs the surgery. The processing device can display this relative positional relationship using a two-dimensional guiding image. Optionally, the processing device can also display the relative positional relationship between the tip of the surgical instrument and the target area using a two-dimensional guiding image. In other words, the two-dimensional guiding image can be used to characterize the relative positional relationship between the tip of the surgical instrument and the target area.

[0229] Therefore, the processing device can display a two-dimensional guided image based on the real-time position of the surgical instruments and the registration results. The real-time position of the surgical instruments can be determined based on the real-time position of the instrument's target. Since the surgical instruments and their tips are relatively fixed, determining the real-time position of the surgical instruments is equivalent to determining the real-time position of their tips.

[0230] It is understandable that, since both the two-dimensional guided image and the second mixed reality model can change based on the real-time position of the surgical instruments, the two-dimensional guided image and the second mixed reality model change in tandem.

[0231] Optionally, the two-dimensional guiding image may include a two-dimensional medical image of the target site and an instrument image corresponding to the surgical instrument on the two-dimensional medical image. Further optionally, the two-dimensional guiding image may be a two-dimensional guiding image based on at least one imaging direction. For example, the two-dimensional guiding image may be a guiding image based on the sagittal, coronal, and transverse planes of the target site.

[0232] In the above embodiments, a two-dimensional guidance image can be displayed based on the real-time position and registration results of the surgical instrument. Since the two-dimensional guidance image is used to characterize the relative positional relationship between the surgical instrument and the target site, the user can intuitively understand the relative positional relationship between the surgical instrument and the target site based on the two-dimensional guidance image and the first mixed reality model anchored to the target site, which is beneficial for surgical navigation.

[0233] In one exemplary embodiment, the above-described mixed reality surgical navigation method may optionally include the following steps:

[0234] Based on the real-time position of the surgical instruments, a second mixed reality model of the surgical instruments is anchored to the surgical instruments for display, so as to perform surgical navigation based on a first mixed reality model anchored to the target site and a second mixed reality model anchored to the surgical instruments.

[0235] In this embodiment, the processing device can also anchor a second mixed reality model of the surgical instruments to the surgical instruments for display based on their real-time position. Optionally, the second mixed reality model can also have a certain degree of transparency, so that the user can see both the second mixed reality model and the surgical instruments in the real physical space.

[0236] Furthermore, by viewing the first mixed reality model anchored to the target site and the second mixed reality model anchored to the surgical instrument, users can intuitively and efficiently understand the relative relationship between the end of the surgical instrument and the target site. Thus, guided by the first mixed reality model anchored to the target site and the second mixed reality model anchored to the surgical instrument, surgical navigation can be achieved intuitively and efficiently.

[0237] In one exemplary embodiment, optionally, the above-described mixed reality surgical navigation method further includes the following steps:

[0238] Upon detecting surgical instruments, determine the instrument information; determine the target navigation algorithm based on the instrument information; and determine the real-time position of the surgical instruments based on the target navigation algorithm.

[0239] In this embodiment, the processing device is capable of detecting surgical instruments. Optionally, the processing device can use a camera on a mixed reality device to identify whether the instrument target of the surgical instrument is within the camera's field of view to determine whether a surgical instrument has been detected.

[0240] If a surgical instrument is detected, the processing device determines the instrument information of the surgical instrument. The instrument information can characterize the attributes of the surgical instrument. For example, the instrument information may include, but is not limited to, the instrument type, specifications, model, or size of the surgical instrument.

[0241] Optionally, the processing device can determine the instrument information of the surgical instruments from the images of the surgical instruments. For example, the processing device can input the images of the surgical instruments into a trained first recognition model, and the trained first recognition model can output the instrument information of the surgical instruments.

[0242] Then, the processing device can determine the target navigation algorithm based on the instrument information. For example, different instrument information can correspond to different navigation algorithms. In this way, after the processing device determines the instrument information of the actual surgical instrument, it can determine the target navigation algorithm that the surgical instrument should use.

[0243] Furthermore, the processing device can navigate and track surgical instruments based on a target navigation algorithm, thereby determining the real-time position of the surgical instruments. The real-time position of the surgical instruments is also their position in the real coordinate system.

[0244] Furthermore, the processing device can display a second mixed reality model corresponding to the surgical instruments based on their real-time position. For example, the processing device can render the surgical instruments based on their real-time position to obtain a second mixed reality model, and then display the second mixed reality model.

[0245] In the above embodiments, since the instrument information of the surgical instrument is determined when the surgical instrument is detected, and the target navigation algorithm is determined based on the instrument information, the target navigation algorithm is a navigation algorithm suitable for the surgical instrument. Based on the target navigation algorithm, the real-time position of the surgical instrument can be determined efficiently and accurately.

[0246] In one exemplary embodiment, optionally, the above-mentioned "determining the instrument information of the surgical instrument when it is detected" can be implemented in the following manner:

[0247] If a surgical instrument is detected, the corresponding instrument menu is displayed; in response to the user's selection of the instrument menu, the instrument information is determined.

[0248] In this embodiment, if a surgical instrument is detected, the processing device will display the corresponding instrument menu. This embodiment does not limit the display position of the instrument menu; for example, after the processing device detects a surgical instrument, it can display the instrument menu next to the instrument target of the surgical instrument.

[0249] The device menu can include multiple candidate device options, and the user can select one to initiate a selection process. The processing device then responds to the user's selection and determines the device information. For example, the processing device can determine the device information based on the candidate device selected by the user.

[0250] In the above embodiments, since the instrument menu corresponding to the surgical instrument is displayed when the surgical instrument is detected, and the instrument information is determined in response to the user's selection of the instrument menu, the instrument information can be determined in combination with the user's selection, thereby improving the accuracy of the determination of the instrument information.

[0251] In one exemplary embodiment, the above-described mixed reality surgical navigation method may optionally include the following steps:

[0252] Based on instrument information and the real-time location of surgical instruments, a second mixed reality model of the surgical instruments is rendered.

[0253] In this embodiment, after determining the instrument information and real-time location of the surgical instruments, the processing device can render the surgical instruments based on the instrument information and the real-time location of the surgical instruments to obtain a second mixed reality model, so as to display the second mixed reality model.

[0254] The second mixed reality model can differ depending on the instrument information. For example, when the surgical instrument is a common navigation probe, the instrument corresponds to instrument information A, and the rendered second mixed reality model can be a cylinder. When the surgical instrument is a biopsy needle, the instrument corresponds to instrument information B, and the rendered second mixed reality model can be a cylinder with a window.

[0255] In the above embodiments, since the second mixed reality model and the instrument image are rendered based on the instrument information and the real-time position of the surgical instruments, the second mixed reality model and the corresponding instrument image can be accurately displayed according to the actual situation of the surgical instruments, and the user can be efficiently guided to navigate through intuitive screens.

[0256] In one exemplary embodiment, the above-described mixed reality surgical navigation method may optionally include the following steps:

[0257] The second mixed reality model is rendered based on the real-time position of the surgical instruments, the puncture point in the target surgical path, and the registration results.

[0258] In this embodiment, because the user operates the surgical instruments by hand during the surgery, the real-time position of the surgical instruments will constantly change. Therefore, the processing device will render a second mixed reality model based on the real-time position of the surgical instruments, the puncture point in the target surgical path, and the registration result. Optionally, rendering the second mixed reality model includes, but is not limited to, rendering the color and / or shape of the second mixed reality model.

[0259] Optionally, the processing device can determine the puncture progress of the surgical instrument based on its real-time position, puncture points in the target surgical path, and registration results, and render a second mixed reality model based on the puncture progress. For example, the portion of the second mixed reality model that has passed through the puncture point is rendered in green, and the portion that has not passed through the puncture point is rendered in red.

[0260] In the above embodiments, since the second mixed reality model is rendered based on the real-time position of the surgical instruments, the puncture point in the target surgical path, and the registration results, users can understand the puncture situation in a timely and accurate manner through the second mixed reality model, which is beneficial to improving surgical accuracy.

[0261] In one exemplary embodiment, the above-described mixed reality surgical navigation method may optionally include the following steps:

[0262] During surgical navigation, navigation prompts are output; the navigation prompts include at least one of the following: the distance between the tip of the surgical instrument and the puncture point in the target surgical path, the distance between the tip of the surgical instrument and the target point in the target surgical path, and the puncture progress.

[0263] In this embodiment, navigation prompts may include, but are not limited to, 3D holographic prompts, text prompts, voice location prompts, and 2D screen prompts; this embodiment is not limited to these.

[0264] Optionally, the processing device can use a camera on a mixed reality device to determine the distance between the tip of the surgical instrument and the puncture point in the target surgical path, and the distance between the tip of the surgical instrument and the target point in the target surgical path. The distance between the tip of the surgical instrument and the puncture point in the target surgical path can represent the puncture depth, and the distance between the tip of the surgical instrument and the target point in the target surgical path can represent the remaining puncture distance. Furthermore, the processing device can determine the puncture progress based on the puncture depth and the remaining puncture distance.

[0265] In the above embodiments, since navigation prompts can be output during surgical navigation, and the navigation prompts include at least one of the following: the distance between the surgical instrument and the puncture point in the target surgical path, the distance between the surgical instrument and the target point in the target surgical path, and the puncture progress, the user can understand the puncture progress in a timely and accurate manner through the navigation prompts.

[0266] In one embodiment, optionally, the processing device can detect whether the distance between the tip of the surgical instrument and the target point is less than a preset safe distance. Further optionally, the processing device can output a safe distance prompt if the distance between the tip of the surgical instrument and the target point is less than the preset safe distance.

[0267] In one embodiment, the processing device may also display preset safety distance information, such as a safety distance warning line, so that the user can refer to the safety distance warning line to see if the distance between the end of the surgical instrument and the target point is less than the preset safety distance.

[0268] Figure 11 This is a schematic diagram illustrating the effect of guiding surgery in one embodiment. The following will combine... Figure 11 This describes the process of guiding users through the surgery. For example... Figure 11 As shown, during surgery, the user disassembles sterile surgical instruments as needed. To ensure precise surgical execution, real-time tracking and navigation of the instruments is required. Since there are many surgical instruments, different navigation algorithms are needed for different instruments. Therefore, when the user brings an instrument into the camera's field of view, the processing device identifies the instrument target. A menu pops up next to the instrument target, allowing the user to select the instrument type and specifications. After selection, the processing device obtains the instrument information and determines the target navigation algorithm based on this information. This algorithm then guides the surgical instruments in real-time, determining their real-time position.

[0269] Meanwhile, based on the real-time position and information of the surgical instruments, the processing device displays a second mixed reality model at a 1:1 scale in the first display area, and updates the relative position of the end of the surgical instruments to the patient in real time in the second display area for the user's reference.

[0270] Furthermore, the user changes the guide adapter according to the selected surgical instrument, and performs the puncture under the real-time navigation guidance of the guide clamp and the surgical instrument. Please continue to refer to Figure 11 During the puncture, the processing device updates the positional relationship between the instrument and the patient in real time in the first and second display areas, and displays the puncture progress bar in the third display area. The puncture progress bar shows the puncture depth and the remaining puncture distance.

[0271] Furthermore, the processing device displays a safe distance warning line marked in orange. When the user approaches the safe distance warning line with the surgical instrument in hand, the processing device will announce "Approaching the target" to warn of potential risks. When the end of the surgical instrument reaches the target, the processing device will announce "Near the target" to warn of potential risks. When the target is reached, the lower end of the puncture progress bar will display 0. When the end of the surgical instrument passes through the target, the processing device will announce a warning, and the lower number will change to a negative number and be marked in red. With multiple warnings from both the display and the voice, the user's operation will be more precise and safer.

[0272] In one exemplary embodiment, the above-described mixed reality surgical navigation method may optionally include the following steps:

[0273] In response to the user's adjustment operation on the first mixed reality model, adjust the display effect and / or display position of the first mixed reality model.

[0274] In this embodiment, the user can also initiate adjustment operations to modify the first mixed reality model, thereby adjusting its display effect and / or display position. The display effect may include, but is not limited to, the transparency, grayscale, and material of the first mixed reality model. The adjustment operation can be an interaction method such as a controller, gesture, or eye tracking; this embodiment is not limited to these methods. For example, the user can drag the first mixed reality model to adjust its display position.

[0275] In one embodiment, since the first mixed reality model is anchored to the target location after registration, the user can adjust the display position of the first mixed reality model before determining the registration result. After determining the registration result, the processing device responds to the user's adjustment operation on the first mixed reality model by adjusting only the display effect of the first mixed reality model.

[0276] In the above embodiments, since the display effect and / or display position of the first mixed reality model can be adjusted in response to the user's adjustment operation on the first mixed reality model, the user can display the first mixed reality model according to their needs, thereby improving the interaction efficiency.

[0277] The process of determining the target surgical path is described below. In an exemplary embodiment, optionally, in response to the user's confirmation of the candidate surgical path, the target surgical path is determined based on the candidate surgical path.

[0278] In this embodiment, the candidate surgical path can be a path pre-stored in the processing device. The processing device can display multiple candidate surgical paths, and the user can select one path as the target surgical path from the candidate surgical paths, or adjust the candidate surgical paths to determine the target surgical path.

[0279] In the above embodiments, since the target surgical path can be determined based on the candidate surgical path in response to the user's confirmation operation of the candidate surgical path, the user can confirm the surgical plan to determine the target surgical path that meets the requirements.

[0280] In an exemplary embodiment, optionally, the above-mentioned "determining the target surgical path based on the candidate surgical path in response to the user's confirmation operation on the candidate surgical path" can be implemented in the following way:

[0281] When displaying a third-section guide image corresponding to at least one candidate surgical path and / or a first mixed reality model corresponding to the third-section guide image, in response to the user's confirmation operation of the candidate surgical path, the target surgical path is determined based on the candidate surgical path; the third-section guide image is perpendicular to the puncture direction of the surgical instrument.

[0282] In this embodiment, the third-section guiding image corresponding to the candidate surgical path is perpendicular to the puncture direction of the surgical instrument. The third-section guiding image is also the image corresponding to the needle path perspective, which represents the perspective perpendicular to the puncture direction of the surgical instrument.

[0283] Furthermore, the third-view guide image may include at least one layer of third-view guide image. The processing device may display the third-view guide image of a specified target layer, or it may display multiple third-view guide images in layer order. It is understood that if the third-view guide image changes, the corresponding first mixed reality model may also change accordingly.

[0284] In this way, when displaying a third-section guide image corresponding to at least one candidate surgical path and / or a first mixed reality model corresponding to the third-section guide image, the user can see the puncture along the candidate surgical path from the needle path perspective, thereby determining whether the candidate surgical path needs to be modified, and initiating a confirmation operation on the candidate surgical path to determine the target surgical path based on the candidate surgical path.

[0285] In the above embodiments, since at least one candidate surgical path is displayed with a third-section guide image and / or a first mixed reality model corresponding to the third-section guide image, in response to the user's confirmation operation of the candidate surgical path, the target surgical path is determined based on the candidate surgical path, and the third-section guide image is perpendicular to the puncture direction of the surgical instrument, the user can intuitively understand the situation of the candidate surgical path, which makes it easier to select the target surgical path from the candidate surgical path.

[0286] In one exemplary embodiment, the above-described mixed reality surgical navigation method may optionally include the following steps:

[0287] In response to the user's selection operation based on the first mixed reality model, the puncture point and target point in the first mixed reality model are determined; and a candidate surgical path is determined based on the puncture point and target point in the first mixed reality model.

[0288] In this embodiment, the user can find suitable puncture points and target points in the first mixed reality model to initiate a selection operation based on the first mixed reality model. In response to the user's selection operation based on the first mixed reality model, the processing device can determine the puncture points and target points in the first mixed reality model and determine candidate surgical paths based on the puncture points and target points. For example, the processing device may use line segments passing through the puncture points and target points as candidate surgical paths.

[0289] In the above embodiments, since the puncture point and target point in the first mixed reality model can be determined in response to the user's selection operation based on the first mixed reality model, and the candidate surgical path can be determined based on the puncture point and target point in the first mixed reality model, the candidate surgical path can be generated efficiently by combining the puncture point and target point selected by the user.

[0290] In one exemplary embodiment, the above-described mixed reality surgical navigation method may optionally include the following steps:

[0291] Identify the region of interest in the first mixed reality model; based on the region of interest, determine the puncture point and target point in the first mixed reality model; based on the puncture point and target point in the first mixed reality model, determine the candidate surgical path.

[0292] In this embodiment, the processing device can determine the region of interest (ROI) in the first mixed reality model. For example, the processing device can determine the ROI in the first mixed reality model according to a preset segmentation algorithm, or the processing device can determine the ROI in the first mixed reality model specified by the user. This embodiment does not impose any limitations.

[0293] Furthermore, after determining the region of interest, the processing device can automatically determine the puncture point and target point in the first mixed reality model based on the region of interest. For example, the processing device can input the region of interest into a trained second recognition model, which will then determine the puncture point and target point in the first mixed reality model.

[0294] Furthermore, after determining the puncture point and target point in the first mixed reality model, the processing device can determine candidate surgical paths based on the puncture point and target point. For example, the processing device can use line segments passing through the puncture point and target point as candidate surgical paths.

[0295] The number of candidate surgical paths can be multiple, and the processing device can display multiple candidate surgical paths in the form of a list for easy viewing or management by users.

[0296] In some embodiments, the processing device can perform path detection on multiple candidate surgical paths. If a risk is detected in a candidate surgical path, a risk warning message can be output so that the user can adjust the candidate surgical path based on the risk warning message. This embodiment does not limit the form of the risk warning message; for example, it can be in the form of voice or visual markers.

[0297] Path detection can include, but is not limited to, blood vessel detection or interference detection of important functional areas. The processing device can determine that a candidate surgical path poses a risk if it passes through pre-defined tissues (such as important blood vessels).

[0298] In the above embodiments, since the region of interest in the first mixed reality model can be determined, and the puncture point and target point in the first mixed reality model can be determined based on the region of interest, the candidate surgical path can be automatically and efficiently determined based on the puncture point and target point in the first mixed reality model.

[0299] Figure 12 This is a schematic diagram illustrating the effect of confirming the target surgical path in one embodiment. The following will combine... Figure 12 This describes the process of determining the target surgical approach. For example... Figure 12 As shown, users can determine the target surgical path on a mixed reality device. For example, users can browse and view the direction and angle of candidate surgical paths on the first mixed reality model, simulating the puncture process along the path through the needle tract view of the 3D model.

[0300] During the display of the third-section guide image corresponding to the candidate surgical path and the first mixed reality model corresponding to the third-section guide image, users can use gestures or a controller to interact and switch between different layers of the third-section guide image.

[0301] Users can also adjust the candidate surgical paths. The adjusted candidate surgical paths can be synchronized to a personal computer (PC) or other authorized mixed reality devices for viewing by users with different roles.

[0302] It should be noted that the above example of a user planning a surgery using a mixed reality device can also be used in some application scenarios. In some cases, the surgery plan can also be planned on a PC or other computer devices, and the surgical plan data will be automatically synchronized to the mixed reality device after completion.

[0303] In an exemplary embodiment, optionally, prior to S201, the above-described mixed reality surgical navigation method may further include the following steps:

[0304] In response to a user's selection action, target patient data is determined from multiple candidate patient data; a visualization interface is displayed based on the target patient data; the visualization interface includes at least one of medical images, a display model corresponding to the medical images, and a first mixed reality model.

[0305] In this embodiment, the processing device can store or import multiple candidate patient data in advance. Optionally, the processing device can manage the imported candidate patient data in a list, displaying basic patient information, including: name, gender, age, and imaging number. It also supports manual editing and adjustment of basic information by the user, as well as a patient query function.

[0306] Furthermore, the processing device can display multiple candidate patient data sets. Users can then select a target patient from these candidate data sets. The candidate patient data may include identifiers, medical images, or attribute information corresponding to at least one candidate patient. Attribute information includes, but is not limited to, gender.

[0307] Furthermore, the processing device can respond to the user's selection operation, determine the target patient data from multiple candidate patient data, and display a visualization interface based on the target patient data.

[0308] Here, the medical image refers to the medical image corresponding to the target patient data. The display model is a 3D model obtained by 3D rendering of the medical image. The first mixed reality model can be a mixed reality model of the target area generated based on the medical image and / or the display model. Therefore, the visualization interface can display at least one of the following: the medical image, the corresponding display model, and the first mixed reality model.

[0309] The following section will introduce the visual interface corresponding to the processing device. Figure 13 This is a schematic diagram of a visual interface in one embodiment, such as... Figure 13 As shown, with Figure 1For example, the visual interface corresponding to the processing device includes a first display area, a second display area, and a third display area. The first, second, and third display areas can be referenced above. Figure 4 The details of that will not be repeated here.

[0310] The layout of the visual interface can be adjusted interactively. For example, the second display area can be freely repositioned according to user preferences. Users can define the top, bottom, left, and right sides of the second display area around the third display area to meet the browsing and operation habits of different users.

[0311] Users can initiate interactive operations based on a visual interface to determine the target surgical path or complete other functions. Interactive operations can be achieved through eye tracking, gestures, voice, or controller interaction. For example, gesture or controller interaction serves as the basic interactive operation, while eye tracking or voice interaction serves as auxiliary interactive operations.

[0312] It should be noted that if eye-tracking interaction is required, the processing device will perform eye coordinate calibration and can output calibration prompts to guide the user to complete the eye coordinate calibration.

[0313] Optionally, during eye coordinate calibration, the processing device can determine the user's eye parameters. Then, during eye-tracking interaction, the processing device can determine the user's eye movements based on the eye movement data and eye parameters acquired by the eye tracker, and respond to the user's eye movements. These eye parameters include, but are not limited to, interpupillary distance.

[0314] In one exemplary embodiment, optionally, the visual interface further includes a menu control; the above-described mixed reality surgical navigation method may also include the following steps:

[0315] In response to a trigger operation on the menu control, the target function is determined from the candidate functions in the menu control; the operation corresponding to the target function is executed in the visual interface.

[0316] Figure 14 This is a schematic diagram of another visual interface in one embodiment. Please refer to [the diagram]. Figure 14 In this embodiment, the user can initiate a menu control trigger operation. For example, the user can trigger a pop-up menu control on the side of the second display area using a gesture or a controller, and select a target function from the candidate functions in the menu space. Then, the processing device can respond to the menu control trigger operation, determine the target function from the candidate functions in the menu control, and execute the operation corresponding to the target function in the visual interface.

[0317] Please refer to Figure 14 Menu controls may include, but are limited to, commonly used tool controls, image processing controls, and surgical simulation controls.

[0318] Commonly used tool controls can include transparency adjustment, scaling, panning, window width and level, brightness, view switching, and shortcut gesture definitions. By selecting the target function among these controls, users can adjust the transparency, window width and level, brightness, and view angle of the content displayed in the visual interface, and perform operations such as scaling and panning. Users can also input their habitual gestures through "shortcut gesture definitions," triggering different functions based on those gestures. For example, a computer device can acquire a user's input gesture, determine the corresponding function, and associate the gesture with the function. This allows the computer device to subsequently respond to the input gesture and execute the function associated with that gesture.

[0319] The image processing controls are primarily used for processing medical images and simulating surgical procedures. By selecting a target function within the image processing controls, functions such as multi-sequence image fusion and registration, blood vessel and tissue (lesion) segmentation, and skull extraction can be performed. Please continue to refer to [the relevant documentation / reference]. Figure 14 When users need to plan a surgical path, they can select "Path Simulation" under "Surgical Simulation" to determine candidate surgical paths.

[0320] In some embodiments, the display position of the menu control can be adjusted. For example, users can fix the menu control in other suitable locations in the space as needed, reducing the complexity of information in the same space. Meanwhile, the processing device can define shortcut tools based on user-initiated shortcut operations for flexible operation. If a user is using the device for the first time or is unfamiliar with the operation, they can also initiate a dialogue via voice. The processing device will acquire the user's voice, identify keywords in the voice, and determine the corresponding guidance information. This guidance information will be used for training and operation simulation for different modules.

[0321] In the above embodiments, since the target function is determined from the candidate functions in the menu control in response to the trigger operation of the menu control, and the operation corresponding to the target function is executed in the visual interface, the user can interact efficiently through the visual interface.

[0322] The above example uses the visual interface of a mixed reality device. Figure 15 This is a schematic diagram of another visualization interface in one embodiment. In some application scenarios, the processing device can also display the visualization interface on other devices such as PCs. The principle of the visualization interface displayed on PCs is similar to that in the above embodiments, and will not be repeated here.

[0323] For example, users can also perform routine processing of medical images via the PC, enabling functions such as image display, fusion registration, tissue segmentation and extraction, and surgical path simulation. Similarly, the first and second display areas can be freely adjusted according to user preferences, and each display area will show basic information as needed, such as patient name, image slice thickness, number of slices, and image magnification. Meanwhile, the image adjustment control area and image processing control area provide standard neurosurgical image processing and adjustment functions, which can be switched, adjusted, and added according to different surgical procedures. The PC's menu interface can also be interacted with via mouse or touchscreen.

[0324] In some embodiments, the processing device can perform surgical planning via a PC and / or a mixed reality device, with various interaction methods. Before S201-S202, the user can perform modal imaging examinations such as CT or MRI on the target area as needed to obtain medical images of the target area. Optionally, before scanning, markers may be fixed on the patient according to the needs of the navigation surgery, that is, the medical images include the markers.

[0325] Furthermore, the processing device can acquire medical images. Specifically, the processing device can acquire medical images via a PC workstation or mixed reality device, enabling subsequent surgical planning based on these images.

[0326] Optionally, the processing device can acquire medical images via removable storage disks, optical discs (CD) / digital versatile discs (DVD), picture archiving and communication systems (PACS) systems, the cloud, etc. The data format of the medical images may include, but is not limited to, digital imaging and communications in medicine (DICOM), STL (a file format for rapid prototyping technology), PLY (polygon file format), and other data formats.

[0327] Figure 16 This is a schematic diagram of the overall workflow in one embodiment, such as... Figure 16As shown, a complete procedure begins with preoperative examinations and assessments, such as a preoperative CT scan, to obtain medical images of the target area and perform preoperative assessments based on these images. Next, image import / processing allows the processing equipment to acquire the medical images used in subsequent surgical planning. Then, surgical planning is completed based on the medical images, determining the puncture point and target. Registration then begins, requiring the installation of a reference frame and reference target, and registration probe calibration to verify registration errors and achieve patient registration. Following this, accuracy verification, incision marking (skin incision marking), and aseptic preparation (surgical disinfection / draping) are performed sequentially. Finally, after incision and bone drilling, the guiding device is installed and adjusted, allowing the surgery to be performed under navigation, enabling navigation positioning, instrument and consumable insertion, and postoperative assessment.

[0328] Current navigation systems, such as neurosurgical navigation systems, typically suffer from the following problems in terms of interactive use: (1) Complex interface. Many navigation systems have complex user interface designs, containing a large number of buttons, menus, and options, which may confuse or distract surgeons during surgery. Complex interfaces may increase operation time, reduce surgical efficiency, and may lead to misoperation in emergency situations. (2) Information overload. Navigation systems in related technologies typically provide a large amount of real-time data and image information, such as CT and MRI scan results, real-time location data, etc. While this information is important, too much information may make it difficult for surgeons to quickly find the key data they need, causing them to spend more time filtering and understanding information during surgery, affecting the smoothness and accuracy of the operation. (3) Lack of intuitiveness. The interface design of navigation systems in related technologies is not intuitive enough. Important information may be hidden in multiple sub-menus, or the design of icons and buttons may not be clear enough. A lack of intuitiveness may make it difficult for surgeons to quickly find the required functions during a stressful surgical procedure, affecting the smooth progress of the operation. (4) Limited interaction methods. Navigation systems in related technologies mainly rely on touch screens or mouse and keyboard for operation, lacking diverse interaction methods such as gesture recognition and voice control. A single interaction method may limit the doctor's operational flexibility, especially when frequent switching of tools or perspectives is required during surgery. (5) Lack of personalized settings. Navigation systems in related technologies lack personalized settings options, making it impossible to customize the interface and functions according to the habits and needs of different doctors. The lack of personalized settings may cause inconvenience to doctors when using the system, affecting operational efficiency and user satisfaction. (6) Insufficient training and support. Navigation systems in related technologies lack sufficient user training and technical support, which leads to difficulties for doctors when using the system, increases the risk of operational errors, and affects surgical outcomes.

[0329] These issues impact the efficiency, accuracy, and user experience of surgeries. Addressing these problems requires improvements in multiple areas, including interface design, interaction methods, personalization settings, and training support, to enhance the user experience and operational efficiency of navigation systems. With the development of mixed reality technology, more and more applications are combining virtual information with the real environment to enhance user experience and improve work efficiency. Navigation and interactive workflows based on mixed reality technology can significantly improve user experience and operational efficiency, especially in complex neurosurgical environments, offering the following advantages:

[0330] (1) Intuitive 3D display. Mixed reality technology can directly overlay virtual navigation information onto the real environment, providing 3D navigation guidance. Users can understand navigation information more intuitively without frequently switching their gaze, reducing cognitive burden. (2) Enhanced sense of reality. MR technology seamlessly integrates virtual information with the real environment, providing an enhanced sense of reality. Users can see virtual navigation markers, paths, and instructions in the real environment, improving the accuracy and reliability of navigation. (3) Diverse interaction methods. MR devices can support multiple interaction methods, including gesture recognition, voice control, handheld control, eye tracking, etc. Users can choose the most suitable interaction method, making operation more flexible and convenient, especially in scenarios requiring delicate operations such as surgery. (4) Real-time dynamic updates. The mixed reality surgical navigation system can update the user's location and environmental information in real time, dynamically adjust navigation guidance, ensure the real-time and accuracy of navigation information, reduce errors, and improve operational efficiency. (5) Reduced information overload. Information can be displayed in a spatial distribution manner, placing different types of information in different spatial locations. Users can more easily find the information they need, reducing information overload and improving information processing efficiency. (6) Personalized settings. Personalized settings can be made according to user needs and habits, such as adjusting the display position, size and transparency of virtual information, to improve user comfort and satisfaction and enhance the system's adaptability. (7) Improve operational safety. Key navigation and warning information can be displayed in real time in the user's field of vision, improving operational safety and reducing risks caused by missing or misunderstanding information. (8) Training and simulation. MR technology can be used for training and simulation, providing a realistic virtual environment and interactive experience, enhancing training effectiveness and reducing errors and risks in actual operation.

[0331] It is evident that navigation interaction based on mixed reality technology can directly overlay virtual navigation information onto the real environment, enabling users to understand navigation guidance more intuitively, enhancing the navigation experience, and thus improving the efficiency and accuracy of surgery.

[0332] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0333] Based on the same inventive concept, this application also provides a mixed reality surgical navigation device for implementing the mixed reality surgical navigation method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more mixed reality surgical navigation device embodiments provided below can be found in the limitations of the mixed reality surgical navigation method described above, and will not be repeated here.

[0334] Figure 17 This is a structural block diagram of a mixed reality surgical navigation device in one embodiment. In an exemplary embodiment, such as... Figure 17 As shown, a mixed reality surgical navigation device 1700 is provided, including: a first guidance module 1701 and a second guidance module 1702, wherein:

[0335] The first guidance module 1701 is used to display a first mixed reality model of the target part and guide the user to register based on the first mixed reality model to obtain the registration result.

[0336] The second guidance module 1702 is used to anchor the first mixed reality model to the target area and display it based on the registration result, so as to guide the user through the preparation process based on the first mixed reality model anchored to the target area, and to perform surgical navigation based on the first mixed reality model anchored to the target area.

[0337] The principle of the mixed reality surgical navigation device 1700 can be referred to the mixed reality surgical navigation method described above, and will not be repeated here.

[0338] The modules in the aforementioned mixed reality surgical navigation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0339] In one exemplary embodiment, a mixed reality surgical navigation system is provided, the mixed reality surgical navigation system including a processor for performing the mixed reality surgical navigation method described above.

[0340] Figure 18 This is a schematic diagram of a mixed reality surgical navigation system in one embodiment. In an exemplary embodiment, a mixed reality surgical navigation system is also provided, such as... Figure 18 As shown, the mixed reality surgical navigation system 1800 includes a mixed reality device 1801. The mixed reality device 1801 can be worn by a user.

[0341] The mixed reality device 1801 includes a glasses device 1801a and a processing device 1801b, which is used to perform the mixed reality surgical navigation method described above.

[0342] Processing device 1801b is used to display a first mixed reality model of a target body part through glasses device 1801a, guide the user to register based on the first mixed reality model, obtain a registration result, and, based on the registration result, anchor the first mixed reality model to the target body part through glasses device 1801a for display, so as to guide the user to perform a preparation process based on the first mixed reality model anchored to the target body part, and to perform surgical navigation based on the first mixed reality model anchored to the target body part.

[0343] Optionally, the mixed reality device 1801 may also include a mixed reality device 1801c. The mixed reality device 1801c is used to track a target to implement the aforementioned mixed reality surgical navigation method. The principle of this navigation system can be referred to the aforementioned mixed reality surgical navigation method, and will not be repeated here.

[0344] In one embodiment, a mixed reality surgical navigation system is also provided, the mixed reality surgical navigation system comprising at least one of the following:

[0345] (1) Surgical planning module. The surgical planning module is used to display the medical images of the target object and provide planning guidance information to guide the user to formulate a surgical plan based on the medical images and determine the target surgical path.

[0346] Planning guidance information is guidance information provided during the surgical planning process. Optionally, planning guidance information may include, but is not limited to, at least one of the following: the first mixed reality model, the third-section guidance image, the first mixed reality model corresponding to the third-section guidance image, the candidate surgical path, the target surgical path, and medical images.

[0347] (2) Registration Module. The registration module provides registration guidance information to guide users to register the medical imaging space and the real physical space of the target object and obtain the registration result.

[0348] Registration guidance information is the guidance information provided during the registration process. Optionally, registration guidance information includes, but is not limited to, at least one of the following: the first screen guidance information, the first mixed reality model, the acquisition points on the first mixed reality model, the first target guidance information, and the fourth mixed reality model of the registration device.

[0349] (3) A preparation module, used to provide preparation guidance information to guide users to perform preparation operations after registration. The preparation operation may be at least one of the following: accuracy verification, incision marking, aseptic preparation, and guide device adjustment.

[0350] Preparation guidance information refers to the guidance information provided during the preparation operation. Optionally, preparation guidance information includes, but is not limited to, at least one of the following: the first mixed reality model, the verification point on the first mixed reality model, the first sectional guidance image, the verification information, the first safety range, the difference between the end of the surgical instrument and the puncture point in the target surgical path, the marking prompt information, the second sectional guidance image, the second screen guidance information, the third screen guidance information, the third mixed reality model of the end of the guiding device, and the fourth mixed reality model of the registration device.

[0351] (4) Surgical navigation module, used to provide navigation guidance information to guide the user to perform surgical operations on the target object, thereby realizing surgical navigation.

[0352] Navigation guidance information is guidance information provided during the surgical procedure. Optionally, navigation guidance information includes, but is not limited to, at least one of the following: the first mixed reality model, the two-dimensional guidance image, the second mixed reality model of the surgical instruments, and navigation prompts.

[0353] Furthermore, at least one of the following information—planning guidance, registration guidance, preparation guidance, and navigation guidance—is overlaid on the target object's real physical space for display.

[0354] Figure 19 This is an internal structure diagram of a computer device in one embodiment. In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as shown below. Figure 19As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores relevant data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network. When executed by the processor, the computer program implements a mixed reality surgical navigation method.

[0355] Those skilled in the art will understand that Figure 19 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0356] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0357] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0358] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0359] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0360] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0361] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A mixed reality surgical navigation system, characterized in that, The system includes a processor, the processor being configured to perform the following methods: Display a first mixed reality model of the target area, and guide the user to register based on the first mixed reality model to obtain the registration result; Based on the registration results, the first mixed reality model is anchored to the target area for display, so as to guide the user through the preparation process based on the first mixed reality model anchored to the target area, and to perform surgical navigation based on the first mixed reality model anchored to the target area.

2. The system according to claim 1, characterized in that, The processor is also used for: Based on the real-time position of the surgical instruments and the registration results, a two-dimensional guidance image is displayed to enable surgical navigation based on the two-dimensional guidance image and a first mixed reality model anchored to the target site; the two-dimensional guidance image is used to characterize the relative positional relationship between the surgical instruments and the target site.

3. The system according to claim 1, characterized in that, The processor is also used for: Based on the real-time position of the surgical instruments, a second mixed reality model of the surgical instruments is anchored to the surgical instruments for display, so as to perform surgical navigation based on the first mixed reality model anchored to the target site and the second mixed reality model anchored to the surgical instruments.

4. The system according to claim 3, characterized in that, The processor is also used for: The second mixed reality model is rendered based on the real-time position of the surgical instruments, the puncture point in the target surgical path, and the registration results.

5. The system according to any one of claims 1-4, characterized in that, The processor is also used for: During the surgical navigation process, navigation prompts are output; the navigation prompts include at least one of the following: the distance between the tip of the surgical instrument and the puncture point in the target surgical path, the distance between the tip of the surgical instrument and the target point in the target surgical path, and the puncture progress.

6. The system according to any one of claims 1-4, characterized in that, The processor is also used for: Display the first screen with guidance information to guide the user through the installation process before registration.

7. The system according to claim 6, characterized in that, The first screen guidance information includes first sub-guidance information and second sub-guidance information; the processor is further configured to: The first sub-guidance information is displayed to guide the user to install the reference frame corresponding to the target part and the reference target on the reference frame based on the first sub-guidance information; If the reference target is not detected, the second sub-guidance information is displayed until the reference target is detected.

8. The system according to claim 7, characterized in that, The processor is also used for: Upon detecting the reference target, the first mixed reality model is displayed, and the user is guided to register based on the first mixed reality model to obtain the registration result.

9. The system according to any one of claims 1-4, characterized in that, The processor is also used for: The system displays the first mixed reality model and the data collection points on the first mixed reality model, and guides the user to register based on the first mixed reality model and the data collection points on the first mixed reality model, thereby obtaining the registration result.

10. The system according to claim 9, characterized in that, The processor is also used for: When displaying the first mixed reality model and the acquisition points on the first mixed reality model, in response to the user's acquisition operation based on the registered device, the real spatial coordinates of each acquisition point are determined; The registration error is determined based on the actual spatial coordinates of each acquisition point and the medical images of the target site. In response to the user's confirmation of the registration error, the registration result is determined based on the actual spatial coordinates of each collection point and the medical image of the target site.

11. The system according to claim 10, characterized in that, The processor is also used for: During the process of determining the true spatial coordinates of each acquisition point, if no reference target or registered target of the registered device corresponding to the target part is detected, the first target guidance information is displayed until the reference target and the registered target are detected.

12. The system according to any one of claims 1-4, characterized in that, The processor is also used in at least one of the following processes: The user is guided to complete the accuracy verification based on the first mixed reality model anchored to the target location; The user is guided to complete the incision marking based on the first mixed reality model anchored to the target area; The user is guided through aseptic preparation based on the first mixed reality model anchored to the target area; The first mixed reality model, anchored to the target location, guides the user to adjust the guiding device.

13. The system according to claim 12, characterized in that, The processor is also used for: When a verification point is displayed on the first mixed reality model, the real spatial coordinates of the current verification point are determined in response to the user's verification operation on the current verification point based on the registered device. The verification error of the current verification point is determined based on the actual spatial coordinates of the current verification point and the medical image of the target area. In response to the user's confirmation of the verification error of the current verification point, for the next verification point, the process returns to the step of determining the true spatial coordinates of the current verification point in response to the user's verification operation based on the registered instrument, until the confirmation of the verification error of all verification points is completed, thereby completing the accuracy verification.

14. The system according to claim 13, characterized in that, The processor is also used for: If the end of the registration device is detected to be in contact with the current verification point, a first cross-sectional guide image is displayed; the first cross-sectional guide image is a cross-sectional image of the target part at the first contact position; the first contact position is the contact position between the end of the registration device and the target part.

15. The system according to claim 13, characterized in that, The processor is also used for: During the accuracy verification process, verification information is displayed; the verification information includes at least one of the following: verification status, verification position of the current verification point, and verification error.

16. The system according to claim 12, characterized in that, The processor is also used for: Based on the first mixed reality model, the first safe zone where the target surgical path is located is displayed; If the surgical instrument is detected to have entered the first safety zone, the difference between the tip of the surgical instrument and the puncture point in the target surgical path is displayed; If the difference between the tip of the surgical instrument and the puncture point is less than a first preset difference, a marking prompt message is output, which is used to guide the user to complete the incision marking.

17. The system according to claim 16, characterized in that, The processor is also used for: When the tip of the surgical instrument contacts the target site, a second cross-sectional guide image is displayed; the second cross-sectional guide image is a cross-sectional image of the target site at the second contact position; the second contact position is the contact position between the tip of the surgical instrument and the target site.

18. The system according to claim 12, characterized in that, The processor is also used for: Based on the first mixed reality model, a second screen of guidance information is displayed; the second screen of guidance information is used to guide the user to install the guidance device and the guidance target of the guidance device. Upon detecting the guide target, a third screen of guidance information is displayed to guide the user in adjusting the guide device based on the third screen of guidance information.

19. The system according to claim 18, characterized in that, The third screen guidance information includes third sub-guidance information, a second safety range, and joint adjustment information. The processor is further configured to: The third sub-guidance information and the second safety range where the target surgical path is located are displayed, so as to guide the user to adjust the end of the guide device to the second safety range based on the third sub-guidance information and the second safety range; Display joint adjustment information and guide the user to adjust the guide device based on the joint adjustment information, so that the difference between the path of the end of the guide device and the target surgical path is less than a second preset difference, so as to complete the adjustment of the guide device.

20. The system according to claim 18, characterized in that, The third-screen guidance information includes a baseline graphic and a variation graphic; the processor is further configured to: The display shows the baseline graphic corresponding to the observation path and the variation graphic corresponding to the non-observation path; wherein, the baseline graphic is obtained by projecting the observation path along its own extension direction, the variation graphic is obtained by projecting the non-observation path along the extension direction of the observation path, the observation path is either the target surgical path or the path where the end of the guide device is located, and the non-observation path is a path different from the observation path between the target surgical path and the path where the end of the guide device is located. In response to the user's adjustment operation on the guide device, the shape of the changed graphic and / or the relative position between the changed graphic and the reference graphic are updated to guide the user to complete the adjustment of the guide device.

21. The system according to claim 18, characterized in that, The processor is also used for: Based on the real-time position of the guide device, the third mixed reality model at the end of the guide device is anchored to the end of the guide device for display.

22. The system according to any one of claims 1-4, characterized in that, The processor is also used for: In response to the user's adjustment operation on the first mixed reality model, adjust the display effect and / or display position of the first mixed reality model.

23. A mixed reality surgical navigation device, characterized in that, The device includes: The first guidance module is used to display a first mixed reality model of the target area and guide the user to register based on the first mixed reality model to obtain the registration result; The second guidance module is used to anchor the first mixed reality model to the target body part based on the registration result, so as to guide the user through the preparation process based on the first mixed reality model anchored to the target body part, and to perform surgical navigation based on the first mixed reality model anchored to the target body part.

24. A mixed reality surgical navigation system, characterized in that, The system includes a mixed reality device, which includes glasses and a processing device. The processing device is configured to display a first mixed reality model of a target body part through the glasses device, guide the user to register based on the first mixed reality model, obtain a registration result, and, based on the registration result, anchor the first mixed reality model to the target body part through the glasses device for display, so as to guide the user to complete the preparation process based on the first mixed reality model anchored to the target body part, and to perform surgical navigation based on the first mixed reality model anchored to the target body part.

25. A mixed reality surgical navigation system, characterized in that, The system includes at least one of the following: The surgical planning module is used to display medical images of the target patient and provide planning guidance information to guide the user in developing a surgical plan based on the medical images and determining the target surgical path. The registration module is used to provide registration guidance information to guide users to register the medical imaging space and the real physical space of the target object and obtain the registration result; The preparation module provides preparation guidance information to guide users through the preparation process after registration. The surgical navigation module is used to provide navigation guidance information to guide the user to perform surgical operations on the target object; Among them, at least one of the planning guidance information, registration guidance information, preparation guidance information, and navigation guidance information is superimposed on the real physical space of the target object for display.

Images