Registration method, registration system, navigation information determination method, and navigation system

Abstract

The application relates to a registration method, a registration system, a navigation information determination method and a navigation system. The method comprises the following steps: determining position information of a target point of a target site in response to a collection signal sent by an interactive component; the collection signal is generated by the interactive component when a first medical instrument senses the target point; and registration is performed on the target site and a medical image based on the position information of the target point of the target site and the medical image, so as to obtain a registration matrix. The method can improve the efficiency of the registration process, and can assist medical staff in efficiently determining navigation information by using navigation technology.

Description

Technical Field

[0001] This application relates to the field of medical technology, and in particular to a registration method, a registration system, a navigation information determination method, and a navigation system. Background Technology

[0002] Navigation technology is increasingly being used in various fields, enabling efficient path planning and target localization. Typically, navigation relies on registration work performed before navigation begins. However, the application of registration and navigation in the medical field still requires further exploration.

[0003] Therefore, in the medical field, how to improve the efficiency of registration so as to use navigation technology to assist medical staff in efficiently determining navigation information is a key research topic for researchers in this field. Summary of the Invention

[0004] Therefore, it is necessary to provide a registration method, registration system, navigation information determination method, and navigation system that can improve the efficiency of registration and assist medical staff in efficiently determining navigation information using navigation technology, in order to address the above-mentioned technical problems.

[0005] In a first aspect, this application provides a registration method applied to a mixed reality device, the method comprising:

[0006] In response to the acquisition signal sent by the interactive component, the location information of the point of interest of the target area is determined; the acquisition signal is generated by the interactive component when the first medical device senses the point of interest.

[0007] Based on the location information of the point of interest of the target area and the medical image of the target area, the target area and the medical image are registered to obtain a registration matrix.

[0008] In one embodiment, the location information of the point of interest for determining the target region includes:

[0009] The system acquires the first coordinate data of the first marker corresponding to the target area in the camera coordinate system, and acquires the second coordinate data of the second marker corresponding to the first medical device in the camera coordinate system; the camera coordinate system is established based on the camera module of the mixed reality device.

[0010] Based on the first coordinate data, a first spatial transformation relationship between the camera coordinate system and the real space coordinate system is determined, and based on the second coordinate data, a third coordinate data of the point of interest of the target part in the camera coordinate system is determined; the real space coordinate system is established based on the first marker.

[0011] Based on the first spatial transformation relationship and the third coordinate data, the fourth coordinate data of the point of interest of the target part in the real spatial coordinate system is determined, and the fourth coordinate data is used as the location information of the point of interest of the target part.

[0012] Secondly, this application also provides a registration method for use in interactive components, the method comprising:

[0013] A data acquisition signal is generated by the interactive component when the first medical device senses a point of interest at the target site.

[0014] The acquisition signal is sent to the mixed reality device so that the mixed reality device responds to the acquisition signal, determines the location information of the point of interest of the target part, and registers the target part and the medical image based on the location information of the point of interest of the target part and the medical image of the target part to obtain a registration matrix.

[0015] In one embodiment, the generation of the acquisition signal includes:

[0016] In response to a first sensing signal sent by the first medical device, the acquisition function is switched from a disabled state to an enabled state; the first sensing signal is generated by the first medical device when it senses a point of interest at the target site.

[0017] When the acquisition function is enabled, the acquisition signal is generated in response to user operation.

[0018] Thirdly, this application also provides a registration system, which includes a mixed reality device, an interactive component, and a first medical device;

[0019] The first medical device is used to generate a first sensing signal and send it to the interactive component when a point of interest at a target site is sensed.

[0020] The interactive component is used to generate a collection signal and send it to the mixed reality device in response to a user operation upon receiving the first sensing signal.

[0021] The mixed reality device is used to determine the location information of the point of interest of the target part in response to the acquisition signal, and to register the target part and the medical image based on the location information of the point of interest of the target part and the medical image of the target part to obtain a registration matrix.

[0022] In one embodiment, the registration system further includes a first marker and a second marker, the first marker being disposed at the target site and the second marker being disposed at the first medical device;

[0023] The mixed reality device is used to respond to the acquisition signal, acquire the first coordinate data of the first marker and the second coordinate data of the second marker, and determine the location information of the point of interest of the target part based on the first coordinate data and the second coordinate data.

[0024] In one embodiment, the first medical device includes a first medical device body and a first sensor disposed on the first medical device body, the first sensor being used to generate a first sensing signal when a point of interest is sensed at the target site.

[0025] Fourthly, this application also provides a navigation information determination method applied to a mixed reality device, the method comprising:

[0026] Receives a second sensing signal sent by the interactive component;

[0027] The posture information of the second medical device is determined based on the second sensing signal; the second medical device is connected to the interactive component and has a puncture channel.

[0028] Based on the posture information of the second medical device, the puncture angle corresponding to the puncture channel is determined.

[0029] Fifthly, this application also provides a navigation information determination method applied to a mixed reality device, the method comprising:

[0030] The device receives a third sensing signal sent by the interactive component; the third sensing signal is generated by the second medical device when the second medical device senses the third medical device; the second medical device is connected to the interactive component and has a puncture channel for insertion of the third medical device.

[0031] The puncture depth of the third medical device is determined based on the third sensing signal.

[0032] Sixthly, this application also provides a navigation system, which includes a mixed reality device, an interactive component, a second medical device, and a third medical device; the second medical device is connected to the interactive component and has a puncture channel for insertion of the third medical device.

[0033] The interactive component is used to generate a second sensing signal and send it to the mixed reality device;

[0034] The second medical device is used to generate a third sensing signal and send it to the mixed reality device upon sensing the third medical device;

[0035] The mixed reality device is used to determine the puncture angle corresponding to the puncture channel based on the second sensing signal, and to determine the puncture depth of the third medical device based on the third sensing signal.

[0036] In one embodiment, the interactive component includes an interactive component body and a second sensor disposed on the interactive component body, and the second medical device includes a second medical device body and a third sensor disposed on the second medical device body;

[0037] The second sensor is used to generate the second sensing signal;

[0038] The third sensor is used to detect the third medical device by providing the third sensing signal.

[0039] The second sensor is a gravity sensor, and the third sensor is a travel sensor.

[0040] Seventhly, this application also provides a mixed reality device, the mixed reality device comprising:

[0041] The determination module is used to determine the location information of the point of interest of the target area in response to the acquisition signal sent by the interactive component; the acquisition signal is generated by the interactive component when the first medical device senses the point of interest;

[0042] The registration module is used to register the target region and the medical image based on the location information of the point of interest of the target region and the medical image of the target region, so as to obtain a registration matrix.

[0043] Eighthly, this application also provides an interactive component, which includes:

[0044] A generation module is used to generate an acquisition signal; this acquisition signal is generated by the interactive component when the first medical device senses a point of interest at the target site.

[0045] The transmitting module is used to send the acquisition signal to the mixed reality device so that the mixed reality device responds to the acquisition signal, determines the location information of the point of interest of the target part, and registers the target part and the medical image based on the location information of the point of interest of the target part and the medical image of the target part to obtain a registration matrix.

[0046] Ninthly, this application also provides a mixed reality device, the mixed reality device comprising:

[0047] The receiving module is used to receive the second sensing signal sent by the interactive component;

[0048] The first determining module is used to determine the posture information of the second medical device based on the second sensing signal; the second medical device is connected to the interactive component and has a puncture channel.

[0049] The second determining module is used to determine the puncture angle corresponding to the puncture channel based on the posture information of the second medical device.

[0050] In a tenth aspect, this application also provides a mixed reality device, the mixed reality device comprising:

[0051] A receiving module is used to receive a third sensing signal sent by an interactive component; the third sensing signal is generated by the second medical device when the second medical device senses the third medical device; the second medical device is connected to the interactive component and has a puncture channel for insertion of the third medical device.

[0052] The determination module is used to determine the puncture depth of the third medical device based on the third sensing signal.

[0053] Eleventhly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of any of the methods described above.

[0054] In a twelfth aspect, this application also provides a computer-readable storage medium. This computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of any of the methods described above.

[0055] In a thirteenth aspect, this application also provides a computer program product. This computer program product includes a computer program that, when executed by a processor, implements the steps of any of the methods described above.

[0056] The aforementioned registration method, registration system, navigation information determination method, and navigation system involve a mixed reality device responding to a collection signal sent by an interactive component to determine the location information of points of interest (POIs) at a target site. Based on the location information of the POIs and the medical image of the target site, the mixed reality device registers the target site and the medical image to obtain a registration matrix. Since the collection signal is generated by the interactive component when the first medical device senses a POI, the interactive component only generates the collection signal after the first medical device senses the POI at the target site, allowing the mixed reality device to respond to this signal and determine the location information of the POIs. In traditional technologies, registration requires pre-planning of the POIs to be collected, followed by the medical device successively sensing the planned POIs. Furthermore, the collection time at each POI must reach a target duration before collecting the next POI; for example, sensing each POI for 3 minutes. In this application, the registration process is completed using an interactive component and the first medical device. Medical personnel can pause or start registration at any time, and can flexibly control the number, distribution, and duration of POI sensing, thus improving the flexibility and efficiency of the registration process. Furthermore, the cooperation between the interactive components, the first medical device, and the mixed reality device during this process ensures the accuracy of the location information of the interest points of the final determined target area, thereby improving the accuracy of the registration matrix and thus improving the accuracy of the registration process. Attached Figure Description

[0057] Figure 1 This is an application environment diagram of the registration method and navigation information determination method in the embodiments of this application;

[0058] Figure 2 This is a flowchart illustrating the registration method in an embodiment of this application;

[0059] Figure 3 This is a flowchart illustrating a method for determining the location information of a point of interest in an embodiment of this application.

[0060] Figure 4 This is a registration diagram provided for an embodiment of this application;

[0061] Figure 5 This is a schematic diagram showing the connection between the interactive handle and the registration probe;

[0062] Figure 6 This is a flowchart illustrating the registration method in an embodiment of this application;

[0063] Figure 7 This is a schematic diagram of a process for generating a sampled signal in an embodiment of this application;

[0064] Figure 8This is a structural block diagram of the registration system in the embodiments of this application;

[0065] Figure 9 This is a schematic diagram showing a medical image in an embodiment of this application;

[0066] Figure 10 This is a flowchart illustrating a navigation information determination method according to an embodiment of this application;

[0067] Figure 11 This is a schematic diagram showing the connection between the interactive handle and the guide in this application;

[0068] Figure 12 A schematic diagram illustrating the process of determining the posture information of a second medical device;

[0069] Figure 13 This is a flowchart illustrating yet another navigation information determination method in this application.

[0070] Figure 14 This is a schematic diagram of the third sensor;

[0071] Figure 15 This is a structural block diagram of the navigation system in the embodiments of this application;

[0072] Figure 16 This is a flowchart illustrating the overall operation in this application;

[0073] Figure 17 This is a detailed flowchart illustrating the overall operation in this application;

[0074] Figure 18 This is a structural block diagram of a mixed reality device according to an embodiment of this application;

[0075] Figure 19 This is a structural block diagram of an interactive component in an embodiment of this application;

[0076] Figure 20 This is a structural block diagram of yet another mixed reality device in the embodiments of this application;

[0077] Figure 21 This is a structural block diagram of yet another mixed reality device in the embodiments of this application;

[0078] Figure 22 This is a diagram showing the internal structure of a computer device in an embodiment of this application. Detailed Implementation

[0079] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0080] Figure 1 This diagram illustrates the application environment of the registration method and navigation information determination method in this application. The registration method and navigation information determination method provided in this application can be applied to, for example... Figure 1 In the application environment shown, medical staff wear mixed reality devices and hold interactive components (not shown) to perform operations on target areas. The mixed reality devices communicate with the interactive components via wired or wireless means.

[0081] Operations on the target area can include two processes. One process is registration, where medical staff need to use interactive components to collect points of interest at the target area, thereby completing the registration of the target area and its corresponding medical image. The other process is navigation, where medical staff need to use interactive components to determine navigation information during the operation on the target area.

[0082] Mixed reality devices include a camera module and a display module. The camera module can be a visual sensor, such as at least one of binocular visual sensors, multi-view visual sensors, and laser visual sensors. The display module can be at least one of a display, an augmented reality (AR) device, a virtual reality (VR) device, and a holographic projection device.

[0083] The interactive component can be an interactive handle equipped with interactive elements, such as buttons and / or remote sensors. Medical staff can then interact with the mixed reality device via these buttons and / or remote sensors, performing actions such as data import, interface switching, and setting changes. Of course, in addition to buttons, the interactive component can also interact with medical staff through other interactive elements. For example, the interactive component could be a grip with voice control capabilities, allowing medical staff to control it via voice. This embodiment does not impose any limitations on these aspects.

[0084] In one embodiment, the mixed reality device can also communicate with other servers to store or relay the data required by the mixed reality device. The server can be a standalone server or a server cluster consisting of multiple servers.

[0085] The registration process will be described in detail below. Figure 2 This is a flowchart illustrating the registration method in an embodiment of this application. This method can be applied to... Figure 1 In one embodiment of the mixed reality device shown, such as Figure 2 As shown, it includes the following steps:

[0086] S201, in response to the acquisition signal sent by the interactive component, determine the location information of the point of interest of the target part; the acquisition signal is generated by the interactive component when the first medical device senses the point of interest.

[0087] In this embodiment, the interactive component may include a processor and a mechanical interface with integrated electrical functions. The interactive component is connected to the first medical device through this mechanical interface, which not only provides power and transmits signals to the interactive component but also serves to secure the interactive component and the first medical device. Of course, the interactive component can also be connected to the first medical device through other connection methods, and this embodiment is not limited thereto. The first medical device may include any one of a registration probe, a structured light emitter, a laser emitter, and a 3D scanner, and this embodiment is not limited thereto.

[0088] Furthermore, the interactive component and the first medical device can communicate. Taking the first medical device as an example, which includes a registration probe, the end of the registration probe can be integrated with a first sensor, which can be a capacitive sensor, a resistive sensor, a pressure sensor, etc.

[0089] Therefore, when wearing the mixed reality device, medical staff can hold the interactive component and use the first medical device connected to the interactive component to sense points of interest in the target area. Taking the determination of the location information of point of interest 1 as an example, assuming the target area is the head, when the medical staff holds the interactive component and uses the first medical device to sense point of interest 1 on the head, the interactive component obtains the corresponding acquisition signal 1, and then the mixed reality device obtains the acquisition signal 1.

[0090] It should be noted that the points of interest in the target area should be evenly distributed and reasonably numerous; when the first medical device includes a registration probe, the first medical device can sense the points of interest through contact sensing; when the first medical device includes a structured light emitter, a laser emitter, or a 3D scanner, the first medical device can sense the points of interest through non-contact sensing, and this embodiment does not impose any restrictions.

[0091] Furthermore, mixed reality devices can respond to the acquisition signals sent by interactive components to determine the location information of points of interest in the target area.

[0092] Taking an interactive handle as the interactive component and a 3D scanner as the first medical device as an example, medical staff can hold the interactive handle and use the 3D scanner connected to the handle to sense at least one point of interest (POI) on the target area. Then, after the 3D scanner detects at least one POI on the target area, it sends a signal back to the interactive handle. The mixed reality device then responds to this signal and uses its camera module to determine the position information of the POI in the real-world coordinate system.

[0093] S202, based on the location information of the point of interest of the target area and the medical image of the target area, the target area and the medical image are registered to obtain the registration matrix.

[0094] In this embodiment, the mixed reality device pre-stores medical images of the target area. These medical images can come from medical image acquisition devices such as magnetic resonance imaging (MRI) or computed tomography (CT) devices, or from other computer devices, or can be images that have been stored offline in the mixed reality device in advance.

[0095] Optionally, the medical images of the target site may include medical planning information, which may include pathway planning, puncture point marking, and other information, so that medical staff can better operate on the target site based on the medical images of the target site.

[0096] One method for mixed reality devices to acquire medical images carrying medical planning information is as follows: After a medical image acquisition device captures a medical image of the target area, it sends it to a computer. Medical personnel can then use the computer to perform medical planning on the target area based on this image. For example, they can set medical planning information such as the puncture path, puncture depth, and puncture angle on the image of the target area. The computer then obtains a medical image carrying this medical planning information. Finally, the computer sends this medical image carrying the medical planning information to the mixed reality device, which then acquires the medical image containing the medical planning information for the target area.

[0097] Furthermore, after the mixed reality device determines the location information of the points of interest in the target area, it can register the target area and the medical image based on the location information of the points of interest in the target area, thus obtaining a registration matrix. The registration matrix indicates the mapping relationship between the medical image and the target area.

[0098] Optionally, the mixed reality device can process the medical image based on the registration matrix to obtain a medical image in a real spatial coordinate system, and then display the medical image in that real spatial coordinate system. Displaying the medical image in the real spatial coordinate system can be achieved by projecting the medical image into the field of vision of medical personnel using holographic projection technology.

[0099] The methods for processing medical images based on the registration matrix can be as follows: the mixed reality device transforms the medical image based on the registration matrix to obtain the medical image in the real spatial coordinate system; or, the mixed reality device transforms the medical image based on the registration matrix and the transpose of the registration matrix to obtain the medical image in the real spatial coordinate system.

[0100] Alternatively, the mixed reality device can process the medical image based on the registration matrix and then directly overlay the medical image in the real spatial coordinate system onto the target area for display; the mixed reality device can also process the medical image based on the registration matrix and then perform post-processing on the medical image in the real spatial coordinate system. Post-processing may include transparency settings, scaling, image parameter adjustments, etc., and then display the post-processed medical image.

[0101] In this way, medical staff can better operate on the target area based on the medical images displayed in the real spatial coordinate system.

[0102] The registration method provided in this embodiment involves a mixed reality device responding to a collection signal sent by an interactive component to determine the location information of points of interest (POIs) at a target site. Based on the location information of the POIs and the medical image of the target site, the mixed reality device registers the target site and the medical image to obtain a registration matrix. Since the collection signal is generated by the interactive component when the first medical device senses a POI, the interactive component only generates the collection signal after the first medical device senses the POI at the target site, allowing the mixed reality device to respond to the collection signal and determine the location information of the POIs. In traditional techniques, registration requires pre-planning of the POIs to be collected, followed by the medical device successively sensing the planned POIs. Furthermore, the collection time at each POI must reach a target duration before collecting the next POI; for example, sensing each POI for 3 minutes. In this application, the registration process is completed using an interactive component and the first medical device. Medical personnel can pause or start registration at any time, and can flexibly control the number, distribution, and duration of POI sensing, thus improving the flexibility and efficiency of the registration process. Furthermore, the cooperation between the interactive components, the first medical device, and the mixed reality device during this process ensures the accuracy of the location information of the interest points of the final determined target area, thereby improving the accuracy of the registration matrix and thus improving the accuracy of the registration process.

[0103] Figure 3 This is a schematic diagram of a process for determining the location information of a point of interest in an embodiment of this application. (Refer to...) Figure 3This embodiment relates to an optional implementation of how to determine the location information of the point of interest (POI) of a target region. Based on the above embodiment, determining the location information of the POI of the target region in S201 includes the following steps:

[0104] S301, acquire the first coordinate data of the first marker corresponding to the target part in the camera coordinate system, and acquire the second coordinate data of the second marker corresponding to the first medical device in the camera coordinate system; the camera coordinate system is established based on the camera module of the mixed reality device.

[0105] In this implementation, Figure 4 This is a registration diagram provided for an embodiment of this application. Figure 5 This is a diagram illustrating the connection between the interactive handle and the registration probe. Please refer to it. Figure 4 and Figure 5 This embodiment uses an interactive handle as the interactive component and a registration probe as the first medical device for illustration.

[0106] In this context, "correspondence between the target area and the first marker" means that the target area and the first marker have a fixed positional relationship, and "correspondence between the first medical device and the second marker" means that the first medical device and the second marker have a fixed positional relationship. For example... Figure 4 As shown, the first marker has a fixed pose relationship with the target area, and the second marker has a fixed pose relationship with the registration probe. For example, the first marker and the target area remain relatively stationary, and the second marker and the registration probe remain relatively stationary. The first and second markers are used to assist in positioning, and they can be positioning balls, coded targets, QR codes, etc., which are not limited in this embodiment.

[0107] Furthermore, when medical staff wear mixed reality devices and use registration probes to sense points of interest at target sites, the camera module within the mixed reality device can acquire the first coordinate data of the first marker in the camera coordinate system and the second coordinate data of the second marker in the camera coordinate system. It is understood that the camera coordinate system is a coordinate system established based on the camera module of the mixed reality device.

[0108] S302, based on the first coordinate data, determine the first spatial transformation relationship between the camera coordinate system and the real space coordinate system, and based on the second coordinate data, determine the third coordinate data of the point of interest of the target part in the camera coordinate system; the real space coordinate system is established based on the first marker.

[0109] In this embodiment, since the first marker and the target part have a fixed pose relationship, the mixed reality device can establish a real spatial coordinate system based on the first marker. The real spatial coordinate system is also the spatial coordinate system of the target part and the environment in which the medical staff are located.

[0110] Furthermore, since the first coordinate data is the coordinate data of the first marker in the camera coordinate system, the mixed reality device, after acquiring the first coordinate data, can determine the first spatial transformation relationship between the camera coordinate system and the real space coordinate system based on the first coordinate data. Optionally, the first spatial transformation relationship may include the translation vector and rotation matrix between the camera coordinate system and the real space coordinate system.

[0111] Furthermore, the second coordinate data is the coordinate data of the second marker in the camera coordinate system, and the second marker has a fixed pose relationship with the first medical device. Therefore, the mixed reality device can determine the third coordinate data of the point of interest of the target area in the camera coordinate system based on the second coordinate data. That is, as the medical staff holds the interactive handle and uses the end of the registration probe to contact the point of interest of the target area, the end of the registration probe also moves accordingly. Then, the camera module in the mixed reality device can determine the third coordinate data of the point of interest of the target area in the camera coordinate system by utilizing the fixed pose relationship between the registration probe and the second marker during the movement of the registration probe. Taking the medical staff using the end of the registration probe to contact point of interest 1 as an example, when the medical staff uses the end of the registration probe to contact point of interest 1, the second marker also moves to the corresponding position with the registration probe. Then, the camera module in the mixed reality device can determine the second coordinate data 1 of the second marker in the camera coordinate system at this time, and then use the pose relationship between the second marker and the registration probe to calculate the third coordinate data 1 of point of interest 1 in the camera coordinate system.

[0112] S303, based on the first spatial transformation relationship and the third coordinate data, determine the fourth coordinate data of the interest point of the target part in the real spatial coordinate system, and use the fourth coordinate data as the location information of the interest point of the target part.

[0113] In this embodiment, after the mixed reality device determines the first spatial transformation relationship and the third coordinate data of the point of interest of the target part, it can determine the fourth coordinate data of the point of interest of the target part in the real space coordinate system based on the first spatial transformation relationship and the third coordinate data, and use the fourth coordinate data as the location information of the point of interest of the target part.

[0114] Continuing with the example of point of interest 1 above, after determining the third coordinate data 1 of point of interest 1 in the camera coordinate system, the third coordinate data 1 can be transformed to the real space coordinate system using the first spatial transformation relationship to obtain the fourth coordinate data 1. The fourth coordinate data 1 is the position information of point of interest 1 of the target part.

[0115] This embodiment acquires first coordinate data of a first marker with a fixed pose relationship to the target area in the camera coordinate system, and second coordinate data of a second marker with a fixed pose relationship to the first medical device in the camera coordinate system. Based on the first coordinate data, a first spatial transformation relationship between the camera coordinate system and the real-world coordinate system is determined. Based on the second coordinate data, third coordinate data of the point of interest (POI) of the target area in the camera coordinate system is determined. Finally, based on the first spatial transformation relationship and the third coordinate data, fourth coordinate data of the POI of the target area in the real-world coordinate system is determined. Since the camera coordinate system is established based on the camera module of the mixed reality device, and the real-world coordinate system is established based on the first marker, the mixed reality device can determine the first spatial transformation relationship between the camera coordinate system and the real-world coordinate system based on the first coordinate data. Therefore, it can determine the fourth coordinate data of the POI of the target area in the real-world coordinate system based on the first spatial transformation relationship and the third coordinate data. In this way, the mixed reality device obtains the location information of the POI of the target area, and can then perform registration between the target area and the medical image based on the location information of the POI and the medical image.

[0116] The above describes the execution process of the registration method in a mixed reality device. The following will describe the execution process of the registration method in an interactive component.

[0117] Figure 6 This is a flowchart illustrating the registration method in an embodiment of this application. This method can be applied to interactive components. In one embodiment, such as... Figure 6 As shown, it includes the following steps:

[0118] S601, Generate acquisition signal; the acquisition signal is generated by the interactive component when the first medical device senses a point of interest at the target site.

[0119] In this embodiment, taking the interactive handle and registration probe as an example, the interactive handle and registration probe are connected through a mechanical interface on the interactive handle. The end of the registration probe integrates a first sensor. Therefore, medical personnel can wear the mixed reality device and hold the interactive handle, using the registration probe connected to the interactive handle to sense points of interest at the target site. After the registration probe senses the points of interest at the target site, the first sensor integrated in the registration probe can send a first sensing signal to the interactive handle. Then, the interactive handle generates a acquisition signal when the registration probe senses the points of interest at the target site. Of course, the interactive component and the first medical device can also be in other forms; this embodiment does not impose limitations.

[0120] S602, send an acquisition signal to the mixed reality device so that the mixed reality device responds to the acquisition signal, determines the location information of the point of interest of the target part, and registers the target part and the medical image based on the location information of the point of interest of the target part and the medical image of the target part to obtain a registration matrix.

[0121] In this implementation, the interactive component and the mixed reality device are connected in communication. After the interactive component generates a collection signal, it sends the collection signal to the mixed reality device.

[0122] Furthermore, the mixed reality device can respond to the acquisition signals sent by the interactive component, determine the location information of the points of interest in the target area, and register the target area and the medical image based on the location information of the points of interest in the target area to obtain a registration matrix. The principle by which the mixed reality device obtains the registration matrix can be referred to in steps S201~S202 and S301~S303 above, and will not be repeated here.

[0123] The registration method provided in this embodiment involves an interactive component generating an acquisition signal when the first medical device senses a point of interest at the target site. This acquisition signal is then sent to a mixed reality device, causing the mixed reality device to respond to the acquisition signal, determine the location information of the point of interest at the target site, and register the target site and the medical image based on the location information of the point of interest and the medical image of the target site, thus obtaining a registration matrix. Therefore, this application utilizes the interaction of the interactive component, the first medical device, and the mixed reality device to obtain the registration matrix, thereby improving the accuracy of the registration process. Furthermore, by using the interactive component and the first medical device to complete the registration process, medical personnel can pause or start the registration at any time, improving the flexibility and efficiency of the registration process.

[0124] Figure 7 This is a schematic diagram of a process for generating a sampled signal in an embodiment of this application. (Refer to...) Figure 7 This embodiment relates to an optional implementation of how to generate the acquisition signal. Based on the above embodiment, step S601, generating the acquisition signal, includes the following steps:

[0125] S701, in response to the first sensing signal sent by the first medical device, the acquisition function is switched from the disabled state to the enabled state; the first sensing signal is generated by the first medical device when the first medical device senses the point of interest of the target site.

[0126] In this embodiment, a medical worker holds an interactive component and uses a first medical device connected to the interactive component to sense points of interest (POIs) on a target area. When the first medical device senses POIs, it sends a first sensing signal to the interactive component. The interactive component then responds to this first sensing signal by switching its data acquisition function from a disabled state to an enabled state. It is understood that when the interactive component's data acquisition function is disabled, it will not generate a data acquisition signal even if it receives the first sensing signal.

[0127] When the acquisition function is enabled, the S702 generates an acquisition signal in response to user operation.

[0128] In this embodiment, when the data acquisition function of the interactive component is enabled, the interactive component will only generate a data acquisition signal in response to a user operation. The user operation can be a medical staff member pressing the "Acquisition" button on the interactive component, or a voice command such as "Start".

[0129] For example, when a medical worker uses the tip of a registration probe to contact a point of interest (POI) on a target site, the first sensor integrated into the registration probe sends a first sensing signal to the interactive handle. At this point, the interactive handle's data acquisition function is enabled. Further, when the medical worker has the tip of the registration probe stably contacting the POI on the target site, and then presses the "Acquire" button on the interactive handle, the handle, with its acquisition function enabled, will generate an acquisition signal in response to the user's action. The interactive handle then sends the acquisition signal to the mixed reality device to complete the subsequent registration process.

[0130] In this embodiment, the interactive component responds to the first sensing signal sent by the first medical device, switching the acquisition function from a disabled state to an enabled state. When the acquisition function is enabled, it generates an acquisition signal in response to user operation. Since the first sensing signal is generated by the first medical device when it senses a point of interest at the target site, the acquisition function can only be enabled when the first medical device senses the target site. Furthermore, the interactive component only generates the acquisition signal when the acquisition function is enabled and user operation is required, thus ensuring the stability of the acquisition signal and improving the accuracy of registration.

[0131] Figure 8 This is a structural block diagram of the registration system in the embodiments of this application, such as... Figure 8 As shown, this application provides a registration system 800, including: a mixed reality device 801, an interactive component 802, and a first medical device 803.

[0132] The first medical device 803 is used to generate a first sensing signal and send it to the interactive component 802 when a point of interest is sensed at the target site.

[0133] The interactive component 802 is used to generate a collection signal and send it to the mixed reality device 801 in response to a user operation upon receiving a first sensing signal.

[0134] The mixed reality device 801 is used to determine the location information of the point of interest of the target part in response to the acquisition signal, and to register the target part and the medical image based on the location information of the point of interest of the target part and the medical image of the target part to obtain a registration matrix.

[0135] Optionally, the registration system 800 further includes a first marker 804 and a second marker 805. The first marker 804 is disposed at the target site, and the second marker 805 is disposed on the first medical device 803.

[0136] The mixed reality device 801 is used to respond to the acquisition signal, acquire the first coordinate data of the first marker 804 and the second coordinate data of the second marker 805, and determine the location information of the point of interest of the target part based on the first coordinate data and the second coordinate data.

[0137] Optionally, the first medical device 803 includes a first medical device body 806 and a first sensor 807 disposed on the first medical device body 806. The first sensor 807 is used to generate a first sensing signal when a point of interest is sensed at the target site.

[0138] Taking the interactive component 802 as an interactive handle and the first medical device 803 including a registration probe as examples, when a medical worker wears the mixed reality device 801 and holds the interactive handle, the medical worker can use the end of the registration probe to contact a point of interest on the target area. When the end of the registration probe contacts the point of interest on the target area, the first sensor 807 integrated at the end of the registration probe generates a first sensing signal, and the first sensor 807 sends the first sensing signal to the interactive handle through a mechanical interface. The first sensor on the registration probe can be a pressure sensor.

[0139] Furthermore, upon receiving the first sensing signal, the interactive controller switches the acquisition function from a disabled state to an enabled state. When the acquisition function is enabled, the interactive controller will only generate an acquisition signal in response to the user's operation and send the acquisition signal to the mixed reality device 801.

[0140] Furthermore, the mixed reality device 801 can receive the acquisition signal, thereby acquiring the first coordinate data of the first marker 804 and the second coordinate data of the second marker 805, and based on the first coordinate data and the second coordinate data, determine the location information of the point of interest of the target part, and then register it with the medical image of the target part based on the location information to obtain a registration matrix.

[0141] The interactive component 802 and the first medical device 803 can also be other devices. Taking a 3D scanner and an interactive handle as examples, when the 3D scanner identifies a point of interest at the target site, it generates a first sensing signal and sends the first sensing signal to the interactive component 802. The other processes are based on the same principles as the registration process of the interactive handle and the registration probe, and will not be described in detail here.

[0142] To reduce hardware costs during the registration process and improve the operational flexibility of medical staff, the first medical device optionally includes a registration probe with a first sensor integrated at its tip. This first sensor generates a first sensing signal when the tip of the registration probe contacts a point of interest at the target site. Therefore, medical staff can complete the registration process more flexibly and quickly when holding the interactive component and the registration probe.

[0143] The registration system provided in this embodiment, by utilizing the cooperation of interactive components, a first medical device, and a mixed reality device, can ensure the accuracy of the location information of the interest points of the final determined target area, thereby improving the accuracy of the registration matrix and thus improving the accuracy of the registration process.

[0144] Thus, after the mixed reality device completes the registration process, it can display medical images in a real spatial coordinate system. For example, the mixed reality device can use holographic projection technology to project the medical image into the field of vision of medical staff. Therefore, medical staff can perform subsequent operations on the target area based on the displayed medical image. Figure 9 This is a schematic diagram of a medical image display in an embodiment of this application. Taking the head as the target area and a CT image of the head carrying medical planning information as an example, since the head and the medical image are registered, one way to display it is that the size ratio of the medical image is the same as the actual head size. The mixed reality device overlays the medical image and the head position 1:1, and reduces the transparency of the medical image during the overlay, so that medical staff can observe both the actual target area and the medical image in their field of vision.

[0145] In other words, since the medical images in the real spatial coordinate system are already displayed in the field of vision of medical staff, they do not need to frequently switch their line of sight when operating on the target area, thereby improving the efficiency of medical staff.

[0146] The registration process has been described above. In the medical field, how to utilize navigation technology to navigate target areas to assist medical personnel in accurately performing procedures on those areas is also a key research area. Therefore, the navigation process will be described in detail below.

[0147] After the registration process is completed, the first medical device can be temporarily put out of use. Therefore, medical staff will remove the first medical device from the mechanical interface of the interactive component and then perform sterilization on the interactive component. The sterilization process can be at least one of ultraviolet treatment, high temperature and high pressure treatment, and sterile sleeve treatment.

[0148] Taking sterile sleeve processing as an example, after medical staff disassemble the interactive component and the first medical device, they place the interactive component into the sterile sleeve. The sterile sleeve has magnetic clasps that are used to attach and align with the interactive component. The sterile sleeve has corresponding protrusions for the interactive components such as remote sensing and buttons, so that the interactive component can move freely inside the sterile sleeve without slipping.

[0149] After the interactive components are sterilized, medical staff can use the sterile interactive components and the second medical device, or the interactive components, the second medical device and the third medical device, to operate on the target area. During the operation on the target area with the help of the second or third medical device, the mixed reality device can determine the navigation information of the second or third medical device and display the navigation information.

[0150] Figure 10 This is a flowchart illustrating a navigation information determination method according to an embodiment of this application. This method can be applied to... Figure 1 In one embodiment of the mixed reality device shown, such as Figure 10 As shown, it includes the following steps:

[0151] S1001, receives the second sensing signal sent by the interactive component.

[0152] In this embodiment, after sterilizing the interactive component, the medical staff connects the interactive component to the second medical device. Optionally, the medical staff can continue to connect the interactive component and the second medical device through a mechanical interface on the interactive component. The second medical device includes instruments with puncture channels, such as guides, to assist the medical staff in operating on the target site.

[0153] It should be noted that the interactive component can have only one general mechanical interface, or it can have multiple different types and specifications of mechanical interfaces to adapt to different medical devices.

[0154] Figure 11 This is a schematic diagram showing the connection between the interactive handle and the guide in this application, as shown below. Figure 11 As shown, taking the interactive component as an interactive handle and the second medical device as a guide as an example, medical staff can connect the interactive handle and the guide.

[0155] Furthermore, based on the above configuration process, medical personnel can wear a mixed reality device, hold an interactive handle, and use a second medical device connected to the interactive handle to perform operations on the target area based on the displayed medical images. Optionally, a second sensor can be integrated into the interactive component, such as a gravity sensor. When the medical personnel move or change posture on the target area using the second medical device, the second sensor of the interactive component can generate a second sensing signal and send the second sensing signal to the mixed reality device, thereby receiving the second sensing signal sent by the interactive component.

[0156] S1002, based on the second sensing signal, determine the posture information of the second medical device; the second medical device is connected to the interactive component, and the second medical device has a puncture channel.

[0157] In this embodiment, after receiving the second sensing signal, the mixed reality device determines the posture information of the second medical device based on the second sensing signal. Figure 12 This is a schematic diagram illustrating the process of determining the posture information of the second medical device. Taking an interactive handle as the interactive component and the second medical device as the guide as an example, as follows... Figure 12 As shown, when medical staff need to perform puncture on a target site, they first need to determine a suitable puncture location. Therefore, medical staff use the interactive handle to move the guide on the target site.

[0158] As the medical staff moves the guide, the interactive handle also moves accordingly. During this movement, the gravity sensor on the interactive handle sends a second sensing signal to the mixed reality device. This second sensing signal includes gravity data determined by the gravity sensor, which could be the current gravitational acceleration of the interactive handle.

[0159] like Figure 12 As shown, the coordinate system corresponding to the gravity sensor is coordinate system G, meaning that the direction of the gravity data from the interactive controller is always vertically downwards. The interactive controller has a preset coordinate system, for example, established with the center of mass of the interactive controller as the origin. Figure 12 The coordinate system shown is XYZ.

[0160] Furthermore, after receiving the second sensing signal from the interactive handle, the mixed reality device can determine the angle between the interactive handle's gravity data and the XYZ coordinate system based on the second sensing signal. Then, based on the angle between the interactive handle's gravity data and the XYZ coordinate system, it can determine the interactive handle's attitude information. Since the interactive handle and the guide have a fixed pose relationship, the attitude information of the guide, i.e., the attitude information of the guide's puncture channel, can also be calculated based on the interactive handle's attitude information.

[0161] S1003, based on the posture information of the second medical device, determine the puncture angle corresponding to the puncture channel.

[0162] In this embodiment, after determining the posture information of the second medical device, the mixed reality device can determine the puncture angle corresponding to the puncture channel based on the posture information of the second medical device. The puncture angle corresponding to the puncture channel indicates the relative angle between the puncture channel and the target site.

[0163] Continuing with the example above, once the mixed reality device determines the orientation information of the puncture channel of the guide, that is, the orientation information of the guide, it can determine the puncture angle corresponding to the puncture channel in the guide. Furthermore, the mixed reality device can display or announce the puncture angle corresponding to the puncture channel of the guide, so that medical staff can determine the optimal puncture angle based on the real-time changing puncture angle displayed.

[0164] In some embodiments, the mixed reality device can also utilize a third marker to determine the pose information of the second medical device, thereby determining the puncture position and angle of the puncture channel. The third marker can have a fixed pose relationship with the interactive component and / or the second medical device. For example, the third marker is relatively stationary with respect to the second medical device, that is, the third marker is placed on the second medical device. Furthermore, when a medical worker wears the mixed reality device and moves the second medical device over the target area, the mixed reality device can determine the coordinate data of the second medical device in the camera coordinate system based on the third marker. Then, the mixed reality device can determine the pose information of the second medical device by utilizing the first spatial transformation relationship between the camera coordinate system and the real space coordinate system. The third marker can also be a positioning ball, an coded target, or a QR code; this embodiment is not limited to these methods.

[0165] In this way, when using interactive components and third medical devices to operate on the target site, medical staff can accurately determine the puncture position and puncture angle to achieve navigation of the target site, thereby assisting medical staff to complete subsequent operations accurately and efficiently.

[0166] The navigation information determination method provided in this embodiment involves a mixed reality device receiving a second sensing signal sent by an interactive component and determining the posture information of a second medical device based on the second sensing signal. Based on the posture information of the second medical device, the puncture angle of the puncture channel is then determined. Since the second medical device is connected to the interactive component, the interactive component and the second medical device also have a fixed positional relationship. Furthermore, the second medical device has a puncture channel. Therefore, during the movement of the second medical device at the target site by medical personnel, the mixed reality device can determine the posture information of the second medical device based on the second sensing signal to determine the puncture angle of the puncture channel, improving the flexibility and accuracy of the operation.

[0167] Figure 13 This is a flowchart illustrating another navigation information determination method in this application embodiment. This method can be applied to... Figure 1 In one embodiment of the mixed reality device shown, such as Figure 13 As shown, it includes the following steps:

[0168] S1301, receiving a third sensing signal sent by the interactive component; the third sensing signal is generated by the second medical device when the second medical device senses the third medical device; the second medical device is connected to the interactive component and has a puncture channel for insertion of the third medical device.

[0169] In this embodiment, if the puncture angle and puncture location have been determined, medical personnel can use a third medical device to puncture the target site.

[0170] The third medical device can be a puncture needle, bone drill, or other instrument used for puncture. The second medical device integrates a third sensor; therefore, when the third medical device moves within the puncture channel of the second medical device, the second medical device can generate a third sensing signal from the third sensor upon sensing the third medical device. Furthermore, the second medical device sends this third sensing signal to the processor in the interaction component via a mechanical interface. After receiving the third sensing signal, the interaction component sends it to the mixed reality device. Thus, the mixed reality device can receive the third sensing signal sent by the interaction component.

[0171] Figure 14 This is a schematic diagram of the third sensor. Taking an interactive handle as the interactive component, a guide as the second medical device, and a puncture needle as the third medical device as an example, as follows... Figure 14As shown, a third sensor is integrated on the guide. This third sensor can be a travel sensor, which uses photoelectric principles to determine the third sensing signal. When the puncture needle is advanced along the puncture channel of the guide towards the target site, the third sensor generates a corresponding third sensing signal. Since the guide is connected to the interactive handle, the third sensing signal generated by the guide's third sensor can be sent to the processor in the interactive handle. The interactive handle can then receive this third sensing signal and further transmit it to the mixed reality device.

[0172] S1302, determine the puncture depth of the third medical device based on the third sensing signal.

[0173] In this embodiment, Figure 14 Based on this, the mixed reality device can determine the puncture depth of the third medical device according to the third sensing signal, which is used to indicate the puncture depth of the third medical device in the target site.

[0174] Furthermore, after the mixed reality device determines the puncture depth of the third medical device, it can display or announce the puncture depth in real time. This allows medical staff to determine whether the puncture has reached the appropriate position based on the puncture depth of the third medical device, thus assisting them in performing the corresponding operations more effectively.

[0175] Thus, during the operation of the target site using interactive components and third medical devices, medical staff can accurately determine the puncture depth to further achieve navigation of the target site, thereby assisting medical staff to complete the operation of the target site accurately and efficiently.

[0176] The navigation information determination method provided in this embodiment involves a mixed reality device receiving a third sensing signal sent by an interactive component and determining the puncture depth of a third medical device based on the third sensing signal. Since the third sensing signal is generated by the second medical device when it senses the third medical device, the mixed reality device can determine the puncture depth of the third medical device based on the third sensing signal, improving operational flexibility and accuracy.

[0177] Figure 15 This is a structural block diagram of the navigation system in an embodiment of this application, such as... Figure 15 As shown in the embodiments of this application, a navigation system 1500 is provided. The navigation system 1500 includes a mixed reality device 1501, an interaction component 1502, a second medical device 1503, and a third medical device 1504. The second medical device 1503 is connected to the interaction component 1502 and has a puncture channel for insertion of the third medical device 1504.

[0178] Interactive component 1502 is used to generate a second sensing signal and send it to mixed reality device 1501.

[0179] The second medical device 1503 is used to generate a third sensing signal and send it to the mixed reality device 1501 when the third medical device 1504 is sensed.

[0180] The mixed reality device 1501 is used to determine the puncture angle corresponding to the puncture channel based on the second sensing signal, and to determine the puncture depth of the third medical device 1504 based on the third sensing signal.

[0181] Optionally, the interactive component 1502 includes an interactive component body and a second sensor 1506 disposed on the interactive component body, and the second medical device 1503 includes a second medical device body 1507 and a third sensor 1508 disposed on the second medical device body 1507.

[0182] The second sensor 1506 is used to generate the second sensing signal.

[0183] The third sensor 1508 is used to generate a third sensing signal when a third medical device is sensed.

[0184] Among them, the second sensor 1506 is a gravity sensor, and the third sensor 1508 is a travel sensor.

[0185] Optionally, the interaction component 1502 may also include a processor 1505.

[0186] Taking an interactive component 1502 as an interactive handle, a second medical device as a guide, a third medical device as a puncture needle, a second sensor 1506 as a gravity sensor, and a third sensor 1508 as a travel sensor as an example, during the process of medical personnel using the interactive handle and guide to find a suitable puncture position, the gravity sensor on the interactive handle generates a second sensing signal and sends this second sensing signal to the mixed reality device 1501. The mixed reality device 1501 then determines the puncture angle of the puncture channel based on the second sensing signal.

[0187] After determining the puncture site, medical personnel will insert the puncture needle into the puncture channel of the guide for puncture. During the puncture process, the travel sensor in the guide generates a third sensing signal and sends the third sensing signal to the interactive handle. The interactive handle then sends the third sensing signal to the mixed reality device 1501, which in turn determines the puncture depth of the third medical device 1504 based on the third sensing signal.

[0188] During navigation, the mixed reality device can display navigation information in real time or based on user needs. This navigation information includes the puncture angle of the puncture channel of the second medical device and / or the puncture depth of the third medical device. It should be noted that the navigation information can be displayed as a two-dimensional planar auxiliary indicator on the mixed reality device, such as an instrument; it can also be a three-dimensional holographic display, or other methods; this embodiment does not impose any limitations.

[0189] To provide a more detailed and clear explanation of the overall operation in this application, the following is combined with... Figure 16 and Figure 17 illustrate. Figure 16 This is a flowchart illustrating the overall operation in this application. Figure 17 This is a detailed flowchart illustrating the overall operation in this application. Figure 16 The medical devices in this context include a first medical device, a second medical device, and a third medical device. The following explanation will use an interactive handle as the interactive component, a registration probe as the first medical device, a guide as the second medical device, and a puncture needle as the third medical device as an example.

[0190] Please combine Figure 16 and Figure 17 When medical staff need to perform procedures on a target site, they first need to conduct medical planning to obtain medical images that include medical planning information about the target site. For example, medical planning information includes puncture path, puncture depth, puncture angle, and other information.

[0191] After obtaining medical images containing medical planning information for the target area, medical staff wear mixed reality devices, such as... Figure 1 As shown. Furthermore, medical personnel can configure interactive handles and registration probes, referring to... Figure 5 Medical staff connect the interactive handle and the registration probe through a mechanical interface and establish communication connections between the mixed reality device and the interactive handle, and between the interactive handle and the registration probe.

[0192] Furthermore, medical personnel use the interactive handle and the registration probe to complete the registration process. During registration, the medical personnel can use the tip of the registration probe to contact the point of interest at the target site. When the tip of the registration probe contacts the point of interest at the target site, a first sensor integrated into the tip of the registration probe generates a first sensing signal, and this first sensor sends the first sensing signal to the interactive handle through a mechanical interface.

[0193] Then, upon receiving the first sensing signal, the interactive controller switches its acquisition function from disabled to enabled. With the acquisition function enabled, the user must press the "Acquire" button on the controller for it to generate an acquisition signal, which is then sent to the mixed reality device. The mixed reality device then responds to this acquisition signal, determines the location information of the points of interest (POIs) at the target area. Once the mixed reality device has acquired a sufficient number of POI location information, it can perform registration with the medical image of the target area based on this location information, obtaining a registration matrix.

[0194] Furthermore, mixed reality devices can process medical images based on a registration matrix and display the medical images in a real-world spatial coordinate system. In other words, after registration is completed, the medical image and the target area are fused into the same real-world spatial coordinate system, and the mixed reality device displays the medical image in the real-world spatial coordinate system.

[0195] Furthermore, medical personnel can wear mixed reality devices to manipulate the target site based on displayed medical images and target areas, using interactive handles and guides, as well as interactive handles, guides, and puncture needles, and in conjunction with navigation information determined by the mixed reality device. The navigation information includes the puncture angle of the guide's puncture channel and / or the puncture depth of the puncture needle.

[0196] During the operation of the target site using the interactive handle and guide, that is, during the process of medical staff using the interactive handle and guide to find a suitable puncture position, the second sensor on the interactive handle generates a second sensing signal and sends the second sensing signal to the mixed reality device, and then the mixed reality device determines the puncture angle of the puncture channel based on the second sensing signal.

[0197] After medical personnel determine the puncture site, during the operation using the interactive components, guide, and puncture needle at the target site—specifically, when the medical personnel insert the puncture needle into the guide—a third sensor integrated on the guide generates a third sensing signal as the needle moves through the puncture channel. This third sensor then sends the third sensing signal to the processor in the interactive handle, allowing the handle to receive and transmit the signal to the mixed reality device. The mixed reality device then determines the puncture depth based on the third sensing signal.

[0198] When the mixed reality device determines that the puncture depth is greater than zero based on the third sensing signal, it displays the puncture depth. When the mixed reality device determines that the puncture depth is not greater than zero based on the third sensing signal, it may not display the puncture depth, or it may display that the puncture depth is zero.

[0199] Furthermore, when the puncture needle in the guide has reached the target depth, the mixed reality device can notify the medical staff that the puncture has reached the target depth. The target depth can be determined based on medical planning information on the target medical image, or it can be a value preset in the mixed reality device; this embodiment does not impose any limitations.

[0200] In summary, this application utilizes the cooperation between the interactive handle, the registration probe, and the mixed reality device to first achieve the registration process, and then utilizes the cooperation between the interactive handle, the guide, the puncture needle, and the mixed reality device to achieve the navigation process. This allows medical personnel to complete the operation on the target site and perform the corresponding examination procedures after the operation is completed.

[0201] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0202] Based on the same inventive concept, embodiments of this application also provide a mixed reality device and an interactive component for implementing the registration method described above, as well as a mixed reality device for implementing the navigation information determination method. The solutions provided by the mixed reality device and interactive component are similar to the solutions described in the above methods. Therefore, the specific limitations in one or more embodiments of the mixed reality device and interactive component provided below can be found in the limitations of the registration method and navigation information determination method described above, and will not be repeated here.

[0203] Figure 18 This is a structural block diagram of a mixed reality device according to an embodiment of this application, such as... Figure 18 As shown, this application embodiment provides a mixed reality device 1800, including: a determination module 1801 and a registration module 1802, wherein:

[0204] The determination module 1801 is used to determine the location information of the point of interest of the target part in response to the acquisition signal sent by the interactive component; the acquisition signal is generated by the interactive component when the first medical device senses the point of interest.

[0205] The registration module 1802 is used to register the target area and the medical image based on the location information of the point of interest of the target area and the medical image of the target area to obtain a registration matrix.

[0206] The mixed reality device provided in this embodiment, in response to a collection signal sent by an interactive component, determines the location information of points of interest (POIs) at a target site. Based on the location information of the POIs and the medical image of the target site, it performs registration between the target site and the medical image to obtain a registration matrix. Since the collection signal is generated by the interactive component when the first medical device senses a POI, the interactive component only generates the collection signal after the first medical device senses the POI at the target site, allowing the mixed reality device to respond to the collection signal and determine the location information of the POIs. In conventional technologies, registration requires advance planning of the POIs to be collected, followed by the medical device sensing the planned POIs sequentially. Furthermore, the collection time at each POI must reach a target duration before collecting the next POI; for example, each POI requires 3 minutes of sensing. In this application, the registration process is completed using an interactive component and the first medical device. Medical personnel can pause or start registration at any time, and can flexibly control the number and distribution of POIs, as well as the time spent sensing POIs, thus improving the flexibility and efficiency of the registration process. Furthermore, the cooperation between the interactive components, the first medical device, and the mixed reality device during this process ensures the accuracy of the location information of the interest points of the final determined target area, thereby improving the accuracy of the registration matrix and thus improving the accuracy of the registration process.

[0207] Optionally, the determining module 1801 includes:

[0208] The acquisition unit is used to acquire the first coordinate data of the first marker corresponding to the target part in the camera coordinate system, and to acquire the second coordinate data of the second marker corresponding to the first medical device in the camera coordinate system; the camera coordinate system is established based on the camera module of the mixed reality device.

[0209] The first determining unit is used to determine the first spatial transformation relationship between the camera coordinate system and the real space coordinate system based on the first coordinate data, and to determine the third coordinate data of the point of interest of the target part in the camera coordinate system based on the second coordinate data; the real space coordinate system is established based on the first marker.

[0210] The second determining unit is used to determine the fourth coordinate data of the point of interest of the target part in the real space coordinate system based on the first spatial transformation relationship and the third coordinate data, and to use the fourth coordinate data as the position information of the point of interest of the target part.

[0211] Figure 19This is a structural block diagram of an interactive component in an embodiment of this application, such as... Figure 19 As shown, this application embodiment provides a mixed reality device 1900, including: a generation module 1901 and a transmission module 1902, wherein:

[0212] The generation module 1901 is used to generate the acquisition signal; the acquisition signal is generated by the interactive component when the first medical device senses the point of interest at the target site.

[0213] The sending module 1902 is used to send acquisition signals to the mixed reality device so that the mixed reality device responds to the acquisition signals, determines the location information of the point of interest of the target part, and performs registration of the target part and the medical image based on the location information of the point of interest of the target part and the medical image of the target part to obtain a registration matrix.

[0214] The interactive component provided in this embodiment generates a acquisition signal when the first medical device senses a point of interest at the target site and sends the acquisition signal to the mixed reality device. The mixed reality device responds to the acquisition signal, determines the location information of the point of interest at the target site, and registers the target site and the medical image based on the location information of the point of interest and the medical image of the target site to obtain a registration matrix. Therefore, this application utilizes the interaction of the interactive component, the first medical device, and the mixed reality device to obtain the registration matrix, thereby improving the accuracy of the registration process. Furthermore, by using the interactive component and the first medical device to complete the registration process, medical personnel can pause or start the registration at any time, improving the flexibility and efficiency of the registration process.

[0215] Optionally, the generation module 1901 includes:

[0216] The switching unit is used to switch the acquisition function from a disabled state to an enabled state in response to a first sensing signal sent by the first medical device; the first sensing signal is generated by the first medical device when the first medical device senses a point of interest at the target site.

[0217] The generation unit is used to generate acquisition signals in response to user operations when the acquisition function is enabled.

[0218] Figure 20 This is a structural block diagram of another mixed reality device in the embodiments of this application, such as... Figure 20 As shown, this application embodiment provides a mixed reality device 2000, including: a receiving module 2001, a first determining module 2002, and a second determining module 2003, wherein:

[0219] Receiver module 2001 is used to receive the second sensing signal sent by the interactive component;

[0220] The first determining module 2002 is used to determine the posture information of the second medical device based on the second sensing signal; the second medical device is connected to the interactive component and has a puncture channel.

[0221] The second determining module 2003 is used to determine the puncture angle corresponding to the puncture channel based on the posture information of the second medical device.

[0222] The mixed reality device provided in this embodiment receives a second sensing signal sent by an interactive component and determines the posture information of a second medical device based on the second sensing signal. Then, based on the posture information of the second medical device, it determines the puncture angle of the puncture channel. Since the second medical device is connected to the interactive component, the interactive component and the second medical device also have a fixed positional relationship. Furthermore, the second medical device has a puncture channel. Therefore, during the operation of the second medical device on the target site by medical personnel, the mixed reality device can determine the posture information of the second medical device based on the second sensing signal to determine the puncture angle of the puncture channel, improving the flexibility and accuracy of the operation.

[0223] Figure 21 This is a structural block diagram of another mixed reality device in the embodiments of this application, such as... Figure 21 As shown, this application embodiment provides a mixed reality device 2100, including: a receiving module 2101 and a determining module 2102, wherein:

[0224] The receiving module 2101 is used to receive a third sensing signal sent by the interactive component; the third sensing signal is generated by the second medical device when the second medical device senses the third medical device; the second medical device is connected to the interactive component and has a puncture channel for insertion of the third medical device.

[0225] The determination module 2102 is used to determine the puncture depth of the third medical device based on the third sensing signal.

[0226] The mixed reality device provided in this embodiment receives a third sensing signal sent by an interactive component and determines the puncture depth of a third medical device based on the third sensing signal. Since the third sensing signal is generated by the second medical device when the second medical device senses the third medical device, the mixed reality device can determine the puncture depth of the third medical device based on the third sensing signal, improving the flexibility and accuracy of the operation.

[0227] The modules in the aforementioned mixed reality device and interactive components can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0228] Figure 22 This is an internal structure diagram of a computer device in an embodiment of this application. This application provides a computer device, which may be a server, and its internal structure diagram can be as follows. Figure 22 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores relevant data. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements a registration method and / or a navigation information determination method.

[0229] Those skilled in the art will understand that Figure 22 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0230] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0231] In response to a collection signal sent by the interactive component, the location information of the point of interest in the target area is determined; the collection signal is generated by the interactive component when the first medical device senses the point of interest.

[0232] Based on the location information of the points of interest of the target area and the medical image of the target area, the target area and the medical image are registered to obtain a registration matrix.

[0233] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0234] The system acquires first coordinate data of a first marker corresponding to the target area in the camera coordinate system, and acquires second coordinate data of a second marker corresponding to the first medical device in the camera coordinate system; the camera coordinate system is established based on the camera module of the mixed reality device; based on the first coordinate data, a first spatial transformation relationship between the camera coordinate system and the real space coordinate system is determined, and based on the second coordinate data, a third coordinate data of the point of interest of the target area in the camera coordinate system is determined; the real space coordinate system is established based on the first marker; based on the first spatial transformation relationship and the third coordinate data, a fourth coordinate data of the point of interest of the target area in the real space coordinate system is determined, and the fourth coordinate data is used as the position information of the point of interest of the target area.

[0235] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0236] A collection signal is generated; the collection signal is generated by the interactive component when the first medical device senses a point of interest in the target area; the collection signal is sent to the mixed reality device so that the mixed reality device responds to the collection signal, determines the location information of the point of interest in the target area, and registers the target area and the medical image based on the location information of the point of interest in the target area and the medical image of the target area to obtain a registration matrix.

[0237] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0238] In response to a first sensing signal sent by the first medical device, the acquisition function is switched from a disabled state to an enabled state; the first sensing signal is generated by the first medical device when it senses a point of interest at the target site; when the acquisition function is enabled, the acquisition signal is generated in response to user operation.

[0239] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0240] The system receives a second sensing signal sent by the interactive component; determines the posture information of the second medical device based on the second sensing signal; the second medical device is connected to the interactive component and has a puncture channel; and determines the puncture angle corresponding to the puncture channel based on the posture information of the second medical device.

[0241] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0242] The system receives a third sensing signal sent by the interactive component; the third sensing signal is generated by the second medical device when the second medical device senses the third medical device; the second medical device is connected to the interactive component and has a puncture channel for the third medical device to be inserted; the puncture depth of the third medical device is determined based on the third sensing signal.

[0243] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0244] In response to a collection signal sent by the interactive component, the location information of the point of interest in the target area is determined; the collection signal is generated by the interactive component when the first medical device senses the point of interest.

[0245] Based on the location information of the points of interest of the target area and the medical image of the target area, the target area and the medical image are registered to obtain a registration matrix.

[0246] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0247] The system acquires first coordinate data of a first marker corresponding to the target area in the camera coordinate system, and acquires second coordinate data of a second marker corresponding to the first medical device in the camera coordinate system; the camera coordinate system is established based on the camera module of the mixed reality device; based on the first coordinate data, a first spatial transformation relationship between the camera coordinate system and the real space coordinate system is determined, and based on the second coordinate data, a third coordinate data of the point of interest of the target area in the camera coordinate system is determined; the real space coordinate system is established based on the first marker; based on the first spatial transformation relationship and the third coordinate data, a fourth coordinate data of the point of interest of the target area in the real space coordinate system is determined, and the fourth coordinate data is used as the position information of the point of interest of the target area.

[0248] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0249] A collection signal is generated; the collection signal is generated by the interactive component when the first medical device senses a point of interest in the target area; the collection signal is sent to the mixed reality device so that the mixed reality device responds to the collection signal, determines the location information of the point of interest in the target area, and registers the target area and the medical image based on the location information of the point of interest in the target area and the medical image of the target area to obtain a registration matrix.

[0250] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0251] In response to a first sensing signal sent by the first medical device, the acquisition function is switched from a disabled state to an enabled state; the first sensing signal is generated by the first medical device when it senses a point of interest at the target site; when the acquisition function is enabled, the acquisition signal is generated in response to user operation.

[0252] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0253] The system receives a second sensing signal sent by the interactive component; determines the posture information of the second medical device based on the second sensing signal; the second medical device is connected to the interactive component and has a puncture channel; and determines the puncture angle corresponding to the puncture channel based on the posture information of the second medical device.

[0254] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0255] The system receives a third sensing signal sent by the interactive component; the third sensing signal is generated by the second medical device when the second medical device senses the third medical device; the second medical device is connected to the interactive component and has a puncture channel for insertion of the third medical device; the puncture depth of the third medical device is determined based on the third sensing signal.

[0256] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0257] In response to a collection signal sent by the interactive component, the location information of the point of interest in the target area is determined; the collection signal is generated by the interactive component when the first medical device senses the point of interest.

[0258] Based on the location information of the points of interest of the target area and the medical image of the target area, the target area and the medical image are registered to obtain a registration matrix.

[0259] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0260] The system acquires first coordinate data of a first marker corresponding to the target area in the camera coordinate system, and acquires second coordinate data of a second marker corresponding to the first medical device in the camera coordinate system; the camera coordinate system is established based on the camera module of the mixed reality device; based on the first coordinate data, a first spatial transformation relationship between the camera coordinate system and the real space coordinate system is determined, and based on the second coordinate data, a third coordinate data of the point of interest of the target area in the camera coordinate system is determined; the real space coordinate system is established based on the first marker; based on the first spatial transformation relationship and the third coordinate data, a fourth coordinate data of the point of interest of the target area in the real space coordinate system is determined, and the fourth coordinate data is used as the position information of the point of interest of the target area.

[0261] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0262] A collection signal is generated; the collection signal is generated by the interactive component when the first medical device senses a point of interest in the target area; the collection signal is sent to the mixed reality device so that the mixed reality device responds to the collection signal, determines the location information of the point of interest in the target area, and registers the target area and the medical image based on the location information of the point of interest in the target area and the medical image of the target area to obtain a registration matrix.

[0263] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0264] In response to a first sensing signal sent by the first medical device, the acquisition function is switched from a disabled state to an enabled state; the first sensing signal is generated by the first medical device when it senses a point of interest at the target site; when the acquisition function is enabled, the acquisition signal is generated in response to user operation.

[0265] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0266] The system receives a second sensing signal sent by the interactive component; determines the posture information of the second medical device based on the second sensing signal; the second medical device is connected to the interactive component and has a puncture channel; and determines the puncture angle corresponding to the puncture channel based on the posture information of the second medical device.

[0267] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0268] The system receives a third sensing signal sent by the interactive component; the third sensing signal is generated by the second medical device when the second medical device senses the third medical device; the second medical device is connected to the interactive component and has a puncture channel for insertion of the third medical device; the puncture depth of the third medical device is determined based on the third sensing signal.

[0269] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0270] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0271] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0272] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A registration method, characterized in that, The registration method, applied to mixed reality devices, includes: In response to a collection signal sent by the interactive component, the location information of the point of interest in the target area is determined; the collection signal is generated by the interactive component when the first medical device senses the point of interest. Based on the location information of the points of interest of the target area and the medical image of the target area, the target area and the medical image are registered to obtain a registration matrix; The location information of the point of interest for determining the target region includes: The system acquires the first coordinate data of the first marker corresponding to the target area in the camera coordinate system, and acquires the second coordinate data of the second marker corresponding to the first medical device in the camera coordinate system; the camera coordinate system is established based on the camera module of the mixed reality device. Based on the first coordinate data, a first spatial transformation relationship between the camera coordinate system and the real space coordinate system is determined, and based on the second coordinate data, a third coordinate data of the point of interest of the target part in the camera coordinate system is determined; the real space coordinate system is established based on the first marker. Based on the first spatial transformation relationship and the third coordinate data, the fourth coordinate data of the interest point of the target part in the real spatial coordinate system is determined, and the fourth coordinate data is used as the position information of the interest point of the target part.

2. A registration method, characterized in that, The registration method, applied to interactive components, includes: A sampling signal is generated by the interactive component when the first medical device senses a point of interest at the target site. The acquisition signal is sent to the mixed reality device so that the mixed reality device responds to the acquisition signal, determines the location information of the point of interest of the target part, and registers the target part and the medical image based on the location information of the point of interest of the target part and the medical image of the target part to obtain a registration matrix; The location information of the point of interest for determining the target region includes: The system acquires the first coordinate data of the first marker corresponding to the target area in the camera coordinate system, and acquires the second coordinate data of the second marker corresponding to the first medical device in the camera coordinate system; the camera coordinate system is established based on the camera module of the mixed reality device. Based on the first coordinate data, a first spatial transformation relationship between the camera coordinate system and the real space coordinate system is determined, and based on the second coordinate data, a third coordinate data of the point of interest of the target part in the camera coordinate system is determined; the real space coordinate system is established based on the first marker. Based on the first spatial transformation relationship and the third coordinate data, the fourth coordinate data of the interest point of the target part in the real spatial coordinate system is determined, and the fourth coordinate data is used as the position information of the interest point of the target part.

3. The registration method according to claim 2, characterized in that, Generate the acquired signal, including: In response to a first sensing signal sent by the first medical device, the acquisition function is switched from a disabled state to an enabled state; the first sensing signal is generated by the first medical device when it senses a point of interest at the target site. When the acquisition function is enabled, the acquisition signal is generated in response to user operation.

4. A registration system, characterized in that, This includes mixed reality devices, interactive components, and first medical devices; The first medical device is used to generate a first sensing signal and send it to the interactive component when a point of interest at the target site is sensed. The interactive component is used to generate a collection signal and send it to the mixed reality device in response to a user operation upon receiving the first sensing signal. The mixed reality device is used to respond to the acquisition signal, determine the location information of the point of interest of the target part, and register the target part and the medical image based on the location information of the point of interest of the target part and the medical image of the target part to obtain a registration matrix; The determination of the location information of the point of interest of the target region includes: The system acquires the first coordinate data of the first marker corresponding to the target area in the camera coordinate system, and acquires the second coordinate data of the second marker corresponding to the first medical device in the camera coordinate system; the camera coordinate system is established based on the camera module of the mixed reality device. Based on the first coordinate data, a first spatial transformation relationship between the camera coordinate system and the real space coordinate system is determined, and based on the second coordinate data, a third coordinate data of the point of interest of the target part in the camera coordinate system is determined; the real space coordinate system is established based on the first marker. Based on the first spatial transformation relationship and the third coordinate data, the fourth coordinate data of the interest point of the target part in the real spatial coordinate system is determined, and the fourth coordinate data is used as the position information of the interest point of the target part.

5. The registration system according to claim 4, characterized in that, The first medical device includes a first medical device body and a first sensor disposed on the first medical device body. The first sensor is used to generate the first sensing signal when a point of interest is sensed at the target site.

6. A method for determining navigation information, characterized in that, The navigation information determination method, applied to mixed reality devices, includes: Receives a second sensing signal sent by the interactive component; The posture information of the second medical device is determined based on the second sensing signal; the second medical device is connected to the interaction component and has a puncture channel; Based on the posture information of the second medical device, the puncture angle corresponding to the puncture channel is determined; Before determining the puncture angle corresponding to the puncture channel, the method further includes: In response to the acquisition signal sent by the interactive component, the location information of the point of interest in the target area is determined; based on the location information of the point of interest in the target area and the medical image of the target area, the target area and the medical image are registered to obtain a registration matrix; the acquisition signal is generated by the interactive component when the first medical device senses the point of interest; The location information of the point of interest for determining the target region includes: The system acquires the first coordinate data of the first marker corresponding to the target area in the camera coordinate system, and acquires the second coordinate data of the second marker corresponding to the first medical device in the camera coordinate system; the camera coordinate system is established based on the camera module of the mixed reality device. Based on the first coordinate data, a first spatial transformation relationship between the camera coordinate system and the real space coordinate system is determined, and based on the second coordinate data, a third coordinate data of the point of interest of the target part in the camera coordinate system is determined; the real space coordinate system is established based on the first marker. Based on the first spatial transformation relationship and the third coordinate data, the fourth coordinate data of the interest point of the target part in the real spatial coordinate system is determined, and the fourth coordinate data is used as the position information of the interest point of the target part.

7. A method for determining navigation information, characterized in that, The navigation information determination method, applied to mixed reality devices, includes: The device receives a third sensing signal sent by the interactive component; the third sensing signal is generated by the second medical device when the second medical device senses the third medical device; the second medical device is connected to the interactive component and has a puncture channel for insertion of the third medical device; The puncture depth of the third medical device is determined based on the third sensing signal. Before determining the puncture depth of the third medical device, the method further includes: In response to the acquisition signal sent by the interactive component, the location information of the point of interest in the target area is determined; based on the location information of the point of interest in the target area and the medical image of the target area, the target area and the medical image are registered to obtain a registration matrix; the acquisition signal is generated by the interactive component when the first medical device senses the point of interest; The location information of the point of interest for determining the target region includes: The system acquires the first coordinate data of the first marker corresponding to the target area in the camera coordinate system, and acquires the second coordinate data of the second marker corresponding to the first medical device in the camera coordinate system; the camera coordinate system is established based on the camera module of the mixed reality device. Based on the first coordinate data, a first spatial transformation relationship between the camera coordinate system and the real space coordinate system is determined, and based on the second coordinate data, a third coordinate data of the point of interest of the target part in the camera coordinate system is determined; the real space coordinate system is established based on the first marker. Based on the first spatial transformation relationship and the third coordinate data, the fourth coordinate data of the interest point of the target part in the real spatial coordinate system is determined, and the fourth coordinate data is used as the position information of the interest point of the target part.

8. A navigation system, characterized in that, It includes a mixed reality device, an interactive component, a second medical device, and a third medical device; the second medical device is connected to the interactive component and has a puncture channel for insertion of the third medical device. The interactive component is used to generate a second sensing signal and send it to the mixed reality device; The second medical device is used to generate a third sensing signal and send it to the mixed reality device upon sensing the third medical device; The mixed reality device is used to determine the puncture angle corresponding to the puncture channel based on the second sensing signal, and to determine the puncture depth of the third medical device based on the third sensing signal. The mixed reality device is further configured to determine the location information of the point of interest of the target part in response to the acquisition signal sent by the interactive component; and to register the target part and the medical image based on the location information of the point of interest of the target part and the medical image of the target part to obtain a registration matrix; the acquisition signal is generated by the interactive component when the first medical device senses the point of interest. The location information of the point of interest for determining the target region includes: The system acquires the first coordinate data of the first marker corresponding to the target area in the camera coordinate system, and acquires the second coordinate data of the second marker corresponding to the first medical device in the camera coordinate system; the camera coordinate system is established based on the camera module of the mixed reality device. Based on the first coordinate data, a first spatial transformation relationship between the camera coordinate system and the real space coordinate system is determined, and based on the second coordinate data, a third coordinate data of the point of interest of the target part in the camera coordinate system is determined; the real space coordinate system is established based on the first marker. Based on the first spatial transformation relationship and the third coordinate data, the fourth coordinate data of the interest point of the target part in the real spatial coordinate system is determined, and the fourth coordinate data is used as the position information of the interest point of the target part.

9. The navigation system according to claim 8, characterized in that, The interactive component includes an interactive component body and a second sensor disposed on the interactive component body; the second medical device includes a second medical device body and a third sensor disposed on the second medical device body. The second sensor is used to generate the second sensing signal; The third sensor is used to generate the third sensing signal when the third medical device is sensed. The second sensor is a gravity sensor, and the third sensor is a travel sensor.

10. A mixed reality device, characterized in that, The mixed reality device includes: The determination module is used to determine the location information of the point of interest of the target area in response to the acquisition signal sent by the interactive component; the acquisition signal is generated by the interactive component when the first medical device senses the point of interest; The registration module is used to register the target region and the medical image based on the location information of the point of interest of the target region and the medical image of the target region to obtain a registration matrix; The location information of the point of interest for determining the target region includes: The system acquires the first coordinate data of the first marker corresponding to the target area in the camera coordinate system, and acquires the second coordinate data of the second marker corresponding to the first medical device in the camera coordinate system; the camera coordinate system is established based on the camera module of the mixed reality device. Based on the first coordinate data, a first spatial transformation relationship between the camera coordinate system and the real space coordinate system is determined, and based on the second coordinate data, a third coordinate data of the point of interest of the target part in the camera coordinate system is determined; the real space coordinate system is established based on the first marker. Based on the first spatial transformation relationship and the third coordinate data, the fourth coordinate data of the interest point of the target part in the real spatial coordinate system is determined, and the fourth coordinate data is used as the position information of the interest point of the target part.

11. An interactive component, characterized in that, The interactive component includes: A generation module is used to generate acquisition signals; the acquisition signals are generated by the interactive component when the first medical device senses a point of interest at the target site. The transmitting module is used to transmit the acquisition signal to the mixed reality device, so that the mixed reality device responds to the acquisition signal, determines the location information of the point of interest of the target part, and registers the target part and the medical image based on the location information of the point of interest of the target part and the medical image of the target part to obtain a registration matrix; The location information of the point of interest for determining the target region includes: The system acquires the first coordinate data of the first marker corresponding to the target area in the camera coordinate system, and acquires the second coordinate data of the second marker corresponding to the first medical device in the camera coordinate system; the camera coordinate system is established based on the camera module of the mixed reality device. Based on the first coordinate data, a first spatial transformation relationship between the camera coordinate system and the real space coordinate system is determined, and based on the second coordinate data, a third coordinate data of the point of interest of the target part in the camera coordinate system is determined; the real space coordinate system is established based on the first marker. Based on the first spatial transformation relationship and the third coordinate data, the fourth coordinate data of the interest point of the target part in the real spatial coordinate system is determined, and the fourth coordinate data is used as the position information of the interest point of the target part.

12. A mixed reality device, characterized in that, The mixed reality device includes: The receiving module is used to receive the second sensing signal sent by the interactive component; The first determining module is used to determine the posture information of the second medical device based on the second sensing signal; the second medical device is connected to the interaction component, and the second medical device has a puncture channel; The second determining module is used to determine the puncture angle corresponding to the puncture channel based on the posture information of the second medical device. The mixed reality device is further configured to, in response to a collection signal sent by the interactive component, determine the location information of the point of interest of the target part; based on the location information of the point of interest of the target part and the medical image of the target part, register the target part and the medical image to obtain a registration matrix; the collection signal is generated by the interactive component when the first medical device senses the point of interest; The location information of the point of interest for determining the target region includes: The system acquires the first coordinate data of the first marker corresponding to the target area in the camera coordinate system, and acquires the second coordinate data of the second marker corresponding to the first medical device in the camera coordinate system; the camera coordinate system is established based on the camera module of the mixed reality device. Based on the first coordinate data, a first spatial transformation relationship between the camera coordinate system and the real space coordinate system is determined, and based on the second coordinate data, a third coordinate data of the point of interest of the target part in the camera coordinate system is determined; the real space coordinate system is established based on the first marker. Based on the first spatial transformation relationship and the third coordinate data, the fourth coordinate data of the interest point of the target part in the real spatial coordinate system is determined, and the fourth coordinate data is used as the position information of the interest point of the target part.

13. A mixed reality device, characterized in that, The mixed reality device includes: A receiving module is used to receive a third sensing signal sent by an interactive component; the third sensing signal is generated by the second medical device when the second medical device senses the third medical device; the second medical device is connected to the interactive component and has a puncture channel for insertion of the third medical device; The determining module is used to determine the puncture depth of the third medical device based on the third sensing signal; The mixed reality device is further configured to, in response to a collection signal sent by the interactive component, determine the location information of the point of interest of the target part; based on the location information of the point of interest of the target part and the medical image of the target part, register the target part and the medical image to obtain a registration matrix; the collection signal is generated by the interactive component when the first medical device senses the point of interest; The location information of the point of interest for determining the target region includes: The system acquires the first coordinate data of the first marker corresponding to the target area in the camera coordinate system, and acquires the second coordinate data of the second marker corresponding to the first medical device in the camera coordinate system; the camera coordinate system is established based on the camera module of the mixed reality device. Based on the first coordinate data, a first spatial transformation relationship between the camera coordinate system and the real space coordinate system is determined, and based on the second coordinate data, a third coordinate data of the point of interest of the target part in the camera coordinate system is determined; the real space coordinate system is established based on the first marker. Based on the first spatial transformation relationship and the third coordinate data, the fourth coordinate data of the interest point of the target part in the real spatial coordinate system is determined, and the fourth coordinate data is used as the position information of the interest point of the target part.

14. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1-3, 6 or 7.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-3, 6 or 7.

16. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1-3, 6 or 7.

Images