Puncture needle tracking system and method

Abstract

The embodiment of the invention discloses a puncture needle tracking system and method. An inertial measurement unit is arranged on the puncture needle, and the method comprises the steps that in the process that the puncture needle punctures a target object, a plurality of inertial measurement data sets collected by the inertial measurement unit and a plurality of image data sets, collected by image collection equipment, of a reference object are received; in response to each received inertial measurement data set, performing first pose updating on a puncture needle model corresponding to the puncture needle in a virtual world coordinate system based on the inertial measurement data set; and in response to each received image data set, performing second pose updating on the puncture needle model in the virtual world coordinate system based on the image data set. By updating the first pose and / or the second pose, the embodiment of the invention can realize accurate three-dimensional positioning and tracking of the puncture needle in the virtual world coordinate system.

Description

Technical Field

[0001] This specification relates to the field of medical devices, and in particular to a puncture needle tracking system and method. Background Technology

[0002] In the medical licensing examination, bone marrow aspiration, lumbar puncture, thoracentesis, and paracentesis are common puncture-related assessment items. The standardization of puncture techniques and procedures is crucial for passing both the exam and clinical practice. Currently, traditional teaching, training, examination, and clinical practice scenarios lack effective means for digitally sensing and recording the movement and changes in the state of the puncture needle during these procedures. Accurate digital sensing is of great significance for assessing procedure quality, identifying errors, providing precise feedback in teaching and training, and automating exam scoring.

[0003] Therefore, a puncture needle tracking system and method are needed to achieve accurate puncture needle pose tracking. Summary of the Invention

[0004] One embodiment of this specification provides a puncture needle tracking method, wherein the puncture needle is equipped with an inertial measurement unit. The method may include, during the process of puncturing a target object with the puncture needle, receiving multiple inertial measurement datasets acquired by the inertial measurement unit and multiple image datasets of a reference object acquired by an image acquisition device; in response to receiving each inertial measurement dataset, updating the first pose of the puncture needle model corresponding to the puncture needle in a virtual world coordinate system based on the inertial measurement dataset; and in response to receiving each image dataset, updating the second pose of the puncture needle model in the virtual world coordinate system based on the image dataset.

[0005] In some embodiments, the plurality of inertial measurement datasets are acquired at a first frequency, and the plurality of image datasets are acquired at a second frequency, wherein the first frequency is greater than the second frequency.

[0006] In some embodiments, the reference objects include markers disposed on the puncture needle and the user's hand. A second pose update of the puncture needle model in the virtual world coordinate system based on the image dataset includes: determining, based on the image dataset, whether marker tracking and hand tracking are effective; in response to determining that both marker tracking and hand tracking are effective, performing a second pose update based on the markers in the image dataset and updating a first transformation relationship between the hand coordinate system and the puncture needle coordinate system; or in response to determining that marker tracking is ineffective but hand tracking is effective, performing a second pose update based on the hand in the image dataset.

[0007] In some embodiments, the image acquisition device corresponds to a head coordinate system, and the second pose update based on the marker in the image dataset includes: determining a first pose of the marker in the head coordinate system based on the image dataset; determining a second pose of the marker in the virtual world coordinate system based on the first pose of the marker in the head coordinate system and a second transformation relationship between the head coordinate system and the virtual world coordinate system; and determining an updated pose of the puncture needle model in the virtual world coordinate system based on the second pose and a third transformation relationship between the marker coordinate system and the puncture needle coordinate system.

[0008] In some embodiments, the second pose update based on the hand in the image dataset includes: updating the position parameters of the puncture needle model in the virtual world coordinate system based on the hand in the image dataset; and updating the pose parameters of the puncture needle model in the virtual world coordinate system based on the inertial measurement dataset.

[0009] In some embodiments, updating the second pose based on the hand in the image dataset includes: determining a fourth transformation relationship between the hand coordinate system and the virtual world coordinate system based on the image dataset; obtaining the latest first transformation relationship between the hand coordinate system and the puncture needle coordinate system; and determining the updated pose of the puncture needle model in the virtual world coordinate system based on the fourth transformation relationship and the latest first transformation relationship.

[0010] In some embodiments, the puncture needle tracking method further includes updating a fifth transformation relationship between the Earth coordinate system and the virtual world coordinate system in response to determining that the marker tracking is valid. The step of updating the first pose of the puncture needle model corresponding to the puncture needle in the virtual world coordinate system based on the inertial measurement dataset includes: determining a third pose of the puncture needle in the Earth coordinate system based on the inertial measurement dataset; and determining an updated pose of the puncture needle model in the virtual world coordinate system based on the third pose and the latest fifth transformation relationship.

[0011] In some embodiments, the puncture needle tracking method further includes, in response to determining that both the marker tracking and the hand tracking are invalid, not performing a second pose update based on the image dataset.

[0012] In some embodiments, the step of updating the first pose of the puncture needle model corresponding to the puncture needle in the virtual world coordinate system based on the inertial measurement dataset in response to receiving each of the inertial measurement datasets includes: determining, in response to each of the received inertial measurement datasets, the number of times the first pose update has been performed since the last second pose update; determining whether the number of performances exceeds a threshold; updating the pose parameters of the puncture needle model in the virtual world coordinate system based on the inertial measurement dataset in response to determining that the number of performances exceeds the threshold; and updating the pose parameters and position parameters of the puncture needle model in the virtual world coordinate system based on the inertial measurement dataset in response to determining that the number of performances does not exceed the threshold.

[0013] One embodiment of this specification provides a computer-readable storage medium that stores computer instructions. When a computer reads the computer instructions from the storage medium, the computer executes the puncture needle tracking method described in this embodiment. Attached Figure Description

[0014] This specification will be further described by way of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. These embodiments are not limiting; in these embodiments, the same reference numerals denote the same structures, wherein: Figure 1 This is a schematic diagram of the structure of an exemplary puncture needle according to some embodiments of this specification; Figure 2A This is a schematic diagram of an exemplary action occurrence area according to some embodiments of this specification; Figure 2B This is a schematic diagram of exemplary detection points set in the area where the action occurs, according to some embodiments of this specification; Figure 3 This is a schematic diagram of the structure of an exemplary puncture needle according to some embodiments of this specification; Figure 4 This is a schematic diagram of the structure of another exemplary puncture needle according to some embodiments of this specification; Figure 5 This is a schematic diagram illustrating an application scenario of an exemplary puncture needle tracking system according to some embodiments of this specification; Figure 6 This is a block diagram of a puncture needle tracking device according to other embodiments of this specification; Figure 7 This is an exemplary flowchart of a puncture needle tracking method according to some embodiments of this specification; Figure 8 This is a schematic diagram illustrating the pose update of a puncture needle model based on the ESKF algorithm, according to some embodiments of this specification. Figure 9 This is an exemplary flowchart illustrating a second pose update based on an image dataset, according to some embodiments of this specification; Figure 10 This is an exemplary flowchart illustrating first pose update based on an inertial measurement dataset according to some embodiments of this specification. Detailed Implementation

[0015] To more clearly illustrate the technical solutions of the embodiments in this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely some examples or embodiments of this specification. For those skilled in the art, these drawings can be applied to other similar scenarios without creative effort. Unless obvious from the context or otherwise specified, the same reference numerals in the drawings represent the same structures or operations.

[0016] It should be understood that the terms “system,” “device,” “unit,” and / or “module” used herein are one way to distinguish different components, elements, parts, sections, or assemblies at different levels. However, if other terms can achieve the same purpose, they may be replaced by other expressions.

[0017] As indicated in this specification and claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" do not specifically refer to the singular and may also include the plural. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of expressly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0018] Flowcharts are used in this specification to illustrate the operations performed by the system according to embodiments of this specification. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, the steps can be processed in reverse order or simultaneously. Furthermore, other operations can be added to these processes, or one or more steps can be removed from them.

[0019] To better understand the tracking system and / or method described herein, the following description is based on the structure and / or information related to the puncture needle. It should be noted that the following description based on the structure and / or information related to the puncture needle is not intended to limit the scope of this specification. For those skilled in the art, the systems and methods disclosed in this specification can be applied to any other system and / or device requiring pose tracking.

[0020] Figure 1This is a schematic diagram of the structure of an exemplary puncture needle 100 according to some embodiments of this specification. Figure 1 As shown, the puncture needle 100 may include a needle body module 110, an external module 120, and a marker 130.

[0021] The needle module 110 may include a puncture needle body 111 and a sensor module 112. The sensor module 112 is disposed within the puncture needle body 111 and is used to detect the movement of the puncture needle 100 and the timing of that movement during the puncture of the target object by the puncture needle 100. In some embodiments, the sensor module 112 may include one or more sensors. These sensors may generate and output signals (e.g., electrical signals) based on material deformation, circuit discontinuity, magnetic field changes, etc., which can reflect changes in the state of the puncture needle 100, thereby reflecting the user's operation of the puncture needle 100 and the timing of that movement. Exemplary sensors may include mechanical on / off switches, contact / disengagement detection circuits, Hall elements, capacitance / resistance sensors, force-sensitive resistors, pressure sensors, magnetic sensors, photoelectric sensors, etc.

[0022] In some embodiments, the one or more sensors may be disposed in one or more action occurrence areas during the operation of the puncture needle 100, thereby detecting the user's operation of the puncture needle 100 and the timing of the action. The action occurrence area refers to a physical location on the puncture needle body 111 that can characterize the completion of an operation. Signal changes detected by sensors deployed in an action occurrence area can indicate whether the operation corresponding to that action occurrence area is in progress or has been completed.

[0023] Figure 2A This is a schematic diagram of an exemplary action occurrence area according to some embodiments of this specification; Figure 2B This is a schematic diagram illustrating exemplary detection points set in the area where the action occurs, according to some embodiments of this specification. Figure 2A and Figure 2B In the image, (a) shows a front view of the puncture needle, and (b) shows a side view of the puncture needle. Figure 2A and Figure 2BAs shown, the area where the action occurs may include the needle tip area 210, the needle cavity area 220, and the needle base area 230. The needle tip area 210 refers to the head area of ​​the puncture needle body 111. Detection points 211 can be set in the needle tip area 210, and sensors can be placed at the detection points to detect whether the syringe, pressure measuring tube, or other instruments are docked or disengaged from the puncture needle 100. For example, switch sensors with two contacts can be set at the two detection points 211 in the needle tip area 210. When the syringe is inserted into the needle tip area 210 of the puncture needle 100, it will compress or connect the two contacts to make them conductive, thereby generating an electrical signal related to the conduction, indicating that the syringe is docked with the puncture needle 100; when the syringe is removed from the puncture needle 100, the two contacts will disconnect, thereby generating an electrical signal related to the disconnection (or causing the electrical signal related to the conduction to disappear), indicating that the syringe is disengaged from the puncture needle 100. The control component (e.g., the main control module described below) can acquire the signal generated by the switch sensor to determine the connection or disconnection status between the syringe and the puncture needle 100.

[0024] The needle cavity region 220 refers to the cavity inside the puncture needle 100 used to accommodate the needle core or liquid, and its adjacent wall area. Sensors positioned at detection points 221 within the needle cavity region 220 can be used to detect the insertion / removal status of the needle core or the liquid retention status within the cavity. For example, taking the detection of the needle core insertion / removal status as an example, a switch sensor with two contacts can be installed on the inner wall of the needle cavity region 220 (e.g., near the needle base). When a metal needle core is inserted into the location of these two contacts, it simultaneously contacts these two contacts, causing them to conduct, thereby generating an electrical signal related to conduction, indicating that the needle core has not been removed. When the needle core is removed, the path between these two contacts is broken, and the electrical signal related to conduction disappears, indicating that the needle core has been removed. As another example, taking the detection of liquid retention status as an example, a liquid switch sensor with two contacts can be installed on the inner wall of the needle cavity region 220. When liquid is present in the cavity, a circuit is formed between the two contacts due to the liquid conduction, generating an electrical signal indicating the presence of liquid. When the liquid is removed from the cavity, the circuit is broken, and the electrical signal indicating the presence of liquid disappears. The control component (e.g., the main control module described below) can acquire the signal generated by the liquid switch sensor to determine whether liquid has been extracted and whether the extraction action has been completed.

[0025] The needle base area 230 refers to the area where the tail of the puncture needle 100 contacts the object being punctured. A sensor positioned at detection point 231 in the needle base area 230 can be used to detect whether the puncture needle 100 is in contact with the object or whether the puncture is complete. For example, the sensor positioned at detection point 231 in the needle base area 230 may include a pressure sensor. When the needle base contacts the object, the pressure sensor is triggered to generate an electrical signal, indicating that the puncture needle 100 is in contact with the object. As another example, when the pressure between the needle base and the object reaches a preset pressure value, the pressure sensor can be triggered to generate an electrical signal, indicating that the puncture needle 100 has been successfully punctured. Yet another example is that the sensor positioned at detection point 231 in the needle base area 230 may include a switch sensor. The switch sensor includes two contacts that are activated when in contact with the skin of the object being punctured, thereby detecting whether the puncture needle 100 is in contact with the object.

[0026] This manual incorporates sensors within the movement areas of the puncture needle (e.g., the needle tip, the inner cavity, and the base) to achieve precise monitoring of the puncture needle procedure without significantly altering its appearance or major mechanical structure. This maximizes the preservation of the original feel and clinical realism of the puncture needle, ensuring the effectiveness of simulation training and providing an objective and quantitative data basis for puncture needle operation evaluation.

[0027] An external module 120 is mounted on the puncture needle body 111 of the puncture needle 100 and is used to implement control, power supply, and measurement functions. For example, the external module 120 can be a standalone unit that is detachably connected to the puncture needle body 111. Alternatively, the external module 120 can be integrated with the puncture needle body 111. In some embodiments, the external module 120 may include a main control module, a battery module, and an inertial measurement unit.

[0028] The main control module can be used to receive, send, and / or process data. For example, the main control module can be used to receive and process signals from the inertial measurement unit. As another example, the main control module can be used to send processed data to an external computing device (e.g., a computer, tablet, smartphone, etc.) via wireless communication. As an example only, the main control module can have a wireless communication module integrated internally or connected externally. This wireless communication module can use remote communication technologies such as Bluetooth, Wi-Fi, Zigbee, and cellular communication to establish a wireless data link with the external device, enabling the software on the external device to receive data generated by the puncture needle 100 (e.g., data related to the puncture needle operation generated by the sensor module 112 in the needle body module, inertial measurement data generated by the inertial measurement unit, etc.). For example, the main control module can be implemented as a microcontroller, including but not limited to a single-chip microcomputer, a system-on-a-chip, or other embedded controllers.

[0029] The battery module can be electrically connected to various electrical components within the puncture needle 100 (e.g., the main control module, inertial measurement unit, sensor module 112, etc.) to provide power. In some embodiments, the battery module may include a rechargeable battery to enable cyclic use of the battery module. In some embodiments, the battery module may include a replaceable battery for recharging during extended operation or in field environments.

[0030] An inertial measurement unit (IMU) can be used to detect the spatial attitude and motion state of the puncture needle 100. In some embodiments, the IMU may include a gyroscope, an accelerometer, and a magnetometer. The gyroscope can be used to measure the angular velocity of the puncture needle body 111 rotating about its axes, the accelerometer can measure the linear acceleration of the puncture needle body 111 in each axis, and the magnetometer can be used to measure the ambient magnetic field to acquire magnetic field data. The magnetic field data can be used to calculate the orientation of the puncture needle body 111 in three-dimensional space. By fusing the output data of the gyroscope, accelerometer, and magnetometer, the attitude quaternion of the puncture needle body 111 can be determined. Thus, the inertial measurement data generated by the IMU can record and / or reconstruct the spatial attitude and / or motion state of the puncture needle body 111, thereby ensuring the standardization of puncture procedures, such as verifying whether the puncture angle is correct. It can also be used to analyze the operator's technique, assess the standardization of their movements, and provide feedback for simulation training.

[0031] This manual explains how integrating an inertial measurement unit and a main control module onto the puncture needle 100 endows the puncture needle body 110 with motion sensing and data processing capabilities, enabling standardized assessment and intelligent assistance in clinical procedures such as medical training. Furthermore, the main control module and battery in the external module can be integrated into a very small form factor, achieving independent power supply and data processing and transmission without altering the basic shape of the puncture needle, thus improving the user experience.

[0032] In some embodiments, the main control module, battery module, and inertial measurement unit can be encapsulated within the housing of the add-on module 120. This integrated and modular design allows the add-on module 120 to function as an independent, reusable functional unit, adaptable to various puncture needle structures, greatly improving the system's versatility and economy. In some embodiments, the main control module, battery module, and inertial measurement unit can also be integrated inside the puncture needle 100 as an inherent component of the puncture needle body 111. For example, the aforementioned modules or units can be miniaturized and embedded within the cavity of the puncture needle body 111, which helps maintain the integrity of the puncture needle and its operational feel.

[0033] The marker 130 is disposed on the needle module 110 (e.g., the puncture needle body 111) to provide visual positioning markers for the visual tracking system (e.g., camera, infrared sensor, etc.). The visual tracking system can calculate the position of the puncture needle 100 in the virtual world coordinate system based on image analysis, thereby achieving optical positioning of the puncture needle 100 during operation.

[0034] In some embodiments, the specific structure of the marker 130 can be adaptively adjusted according to the visual tracking system employed. For example, the marker 130 may include a cubic or polyhedral structure with reference markers such as ArUco affixed. As another example, the marker 130 may include a base with passive infrared reflective spheres or reflective patches. Yet another example, the marker 130 may include a mounting base with LED beads that actively emit infrared light.

[0035] In some embodiments, the optical positioning data provided by the marker 130 can be combined with the inertial measurement data acquired by the inertial measurement unit to achieve six-degree-of-freedom pose tracking of the puncture needle 100, significantly improving the robustness and positioning accuracy of the tracking system under different operating environments.

[0036] This manual modifies the puncture needle, integrating a sensor module for detecting the needle's movement and timing, an inertial sensor for detecting the needle's spatial attitude and motion, and markers for providing visual positioning during tracking. It also includes independent power supply, data processing, and transmission units, thus constructing a multimodal data acquisition and tracking system that enables synchronous perception of the needle's motion and pose during the puncture needle operation.

[0037] Figure 3 This is a schematic diagram of the structure of an exemplary puncture needle 300 according to some embodiments of this specification. Figure 3 As shown, the puncture needle 300 may include a puncture needle body 310, an external fastener 320, a marker 330, and a battery 340. The external fastener 320, the marker 330, and the battery 340 are respectively disposed on the puncture needle body 310 to provide different functions.

[0038] The puncture needle body 310 includes a top cover 311, a silicone tube 312, a fastener 313, a sleeve 314, and a needle core 315. The top cover 311 is located at the end of the fastener 313 away from the needle core 315 and cooperates with two fasteners 313 to form a cavity encapsulating the silicone tube 312 and the needle core 315. The sleeve 314 is sleeved around the fasteners 313 and has an opening at the end away from the top cover 311 (or referred to as the base of the sleeve 314) for the needle core 315 to enter and exit. The needle core 315 can be accommodated inside the sleeve 314 or extend out of the base of the sleeve 314 to perform the puncture operation.

[0039] External mounting bracket 320 can be used to support or secure external modules (e.g., Figure 1 The aforementioned add-on module 120). For example, such as... Figure 3 As shown, the external fastener 320 can be mechanically connected to a fastener 313 in the puncture needle body 310, and the external module can be detachably mounted on the external fastener 320.

[0040] A battery 340 is disposed on another fastener 313 of the puncture needle body 310 for electrically connecting to various electrical components within the puncture needle 300 to provide electrical power. In some embodiments, the battery 340 may include a magnetic battery, which is physically connected to the puncture needle body 310 by magnetic attraction and electrically connected to various electrical components within the puncture needle 300 through contacts on the puncture needle body 310.

[0041] The marker 330 may include a marker holder disposed on the puncture needle body 310 and at least one visual marker located on the marker holder, for providing visual positioning marking for a visual tracking system (e.g., a camera, infrared sensor, etc.). Figure 3 As shown, the marker 330 can be set at one end of the fastener 313 near the top cover 311.

[0042] In some embodiments, the puncture needle 300 may further include a sensor module (not shown) disposed on the puncture needle body 310. The sensor module may be disposed within the puncture needle body 310 and is used to detect the movement of the puncture needle 300 and the timing of the movement during the puncture of the target object. More information about the sensor module can be found in [link to relevant documentation]. Figure 1 Its description will not be repeated here.

[0043] This embodiment of the manual sets the battery and the external mounting bracket (or external module) as two independent modules. This allows for separate and rapid battery replacement or charging without disassembling or interfering with the puncture needle body or the external module, avoiding the risk of interruption of the puncture simulation operation due to battery depletion. It also allows the battery to be designed as a universal accessory, facilitating unified management, cyclic charging, and backup, significantly improving the usability of the puncture needle. Simultaneously, setting the battery independently of the external module reduces the size and weight of the external module, allowing for a more compact and stable fixation to the puncture needle body.

[0044] Figure 4 This is a schematic diagram of the structure of another exemplary puncture needle 400 shown according to some embodiments of this specification. Figure 4As shown, the puncture needle 400 may include a puncture needle body 410, an external fastener 420, and a battery 440. The external fastener 420 and the battery 440 are respectively disposed on the puncture needle body 410 to provide different functions. The puncture needle body 410 includes a top cover 411, a silicone tube 412, a fastener 413, a sleeve 414, and a needle core 415. In this embodiment, the puncture needle body 410, the external fastener 420, and the battery 440 can be respectively connected to… Figure 3 The puncture needle body 310, the external fastener 320, and the battery 340 are similar and will not be described in detail here.

[0045] and Figure 3 Unlike the puncture needle 300 shown, the puncture needle 400 in this embodiment may not include a marker. In some embodiments, when using a visual tracking system to track the position of the puncture needle 400 without a marker, the operator can hold the puncture needle 400 in a preset grip manner. This preset grip manner can reflect the relative positional relationship between the operator's hand and the puncture needle 400, thereby allowing the position of the puncture needle in three-dimensional space to be determined based on the tracking of the operator's hand.

[0046] By omitting markers, the embodiments in this specification not only simplify the puncture needle structure and reduce manufacturing costs, but also avoid tracking failures caused by marker occlusion. In some embodiments, while determining the position of the puncture needle in three-dimensional space based on the tracking of the operator's hand, the orientation of the puncture needle can also be determined based on inertial measurement data, constructing a continuous, occlusion-resistant tracking scheme that does not rely on markers. This simplifies the hardware structure while ensuring the accuracy and stability of the tracking process.

[0047] Figure 5 These are schematic diagrams illustrating application scenarios of exemplary puncture needle tracking systems according to some embodiments of this specification. For example... Figure 5 As shown, the puncture needle tracking system 500 may include a puncture needle 510, a head-mounted device 520, a processing device 530, a network 540, a storage device 550, and a terminal 560. The various components in the puncture needle tracking system 500 can be connected in various ways. For example, the puncture needle 510 and / or the head-mounted device 520 may be connected to the storage device 550 and / or the processing device 530 via the network 540, or they may be directly connected to the storage device 550 and / or the processing device 530. As another example, the storage device 550 may be directly connected to the processing device 530 or connected via the network 540. As yet another example, the terminal device 560 may be connected to the storage device 550 and / or the processing device 530 via the network 540, or it may be directly connected to the storage device 550 and / or the processing device 530.

[0048] The puncture needle tracking system 500 can be applied in scenarios such as simulation teaching and skills examinations. To simulate these scenarios, the movement and state changes of the puncture needle in the real world can be mapped to the virtual world. Users can perceive, calculate, and analyze real-world operational behaviors based on the puncture needle model corresponding to the puncture needle in the virtual world.

[0049] A puncture needle 510 is a device used to aspirate bodily fluids, inject drugs, insert catheters, or obtain biopsy samples by penetrating the skin and tissue. In some embodiments, the puncture needle 510 may be equipped with an inertial measurement unit (IMU) for detecting the spatial attitude and motion state of the puncture needle 510. For example, the IMU may be located in an external module (e.g., such as...). Figure 1 In the attached module 120 shown, the attached module can be mounted on the puncture needle 510 in an external manner. In some embodiments, the puncture needle 510 may also be provided with markers to provide visual positioning marks for the puncture needle tracking system 500. For example, markers (e.g., such as...) Figure 1 The marker 130 shown may be disposed on the surface or top of the puncture needle 510 to provide a visual positioning marker. In some embodiments, the puncture needle 510 may be or include Figures 1-4 For more information on the aforementioned puncture needle, please refer to [link / reference needed]. Figures 1-4 Its description will not be repeated here.

[0050] Head-mounted device 520 may include a computer terminal worn on the head, which projects digital content into the user's field of vision using near-eye display technology. For example, head-mounted device 520 can be used to simulate scenarios such as teaching and skills examinations, mapping the movement and state changes of the puncture needle 510 in the real world to the virtual world, thereby enabling the perception, calculation, and analysis of real-world operational behaviors based on the virtual world. Exemplary head-mounted device 520 may include virtual reality (VR) devices, augmented reality (AR) devices, mixed reality (MR) devices, etc. For example, head-mounted device 520 can be implemented as a helmet, glasses, head-mounted display, etc. In some embodiments, head-mounted device 520 may be equipped with an image acquisition device that can capture images of the surrounding environment in real time for identifying and tracking objects in the scene (e.g., the puncture needle 510, the user's hand, etc.). In some embodiments, head-mounted device 520 can also calculate and provide the camera's own pose information in real time based on the image stream acquired by the image acquisition device. Based on tracking data and the camera's own pose information, the 520 head-mounted device can project real-world scenes into the virtual world, creating a mixed reality space.

[0051] Processing device 530 can process data and / or information obtained from puncture needle 510, head-mounted device 520, storage device 550, and / or terminal 560. For example, processing device 530 can process an inertial measurement dataset obtained from puncture needle 510 and update the first pose of the puncture needle model corresponding to puncture needle 510 in the virtual world coordinate system based on the inertial measurement dataset. As another example, processing device 530 can process an image dataset of a reference object obtained from head-mounted device 520 (e.g., an image acquisition device) and update the second pose of the puncture needle model in the virtual world coordinate system based on the image dataset. In some embodiments, processing device 530 may include one or more sub-processing devices (e.g., a single-core processing device or a multi-core multi-chip processing device). By way of example only, the processing device 530 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), an application-specific instruction processor (ASIP), a graphics processing unit (GPU), a physical processor (PPU), a digital signal processor (DSP), a field-programmable gate array (FPGA), a programmable logic circuit (PLD), a controller, a microcontroller unit, a reduced instruction set computer (RISC), a microprocessor, or any combination thereof.

[0052] In some embodiments, the head-mounted device 520, the terminal 560, and other possible system components may include a processing device 530. For example, the processing device 530 or a functional module that enables the functions of the processing device 530 may be integrated into the head-mounted device 520, the terminal 560, and other possible system components.

[0053] Network 540 may include any suitable network that facilitates information and / or data exchange between the needle tracking system 500 and the needle tracking system 500. In some embodiments, one or more components of the needle tracking system 500 (e.g., needle 510, head-mounted device 520, processing device 530, storage device 550, or terminal 560) may connect to and / or communicate with other components of the needle tracking system 500 via network 540. For example, processing device 530 may acquire inertial measurement data from needle 510 via network 540. As another example, processing device 530 may acquire user feedback information from head-mounted device 520 or terminal 560 via network 540. In some embodiments, network 540 may be any form of wired or wireless network, or any combination thereof. In some embodiments, network 540 may be any one or more of wired or wireless networks. For example, network 540 may include cable networks, fiber optic networks, telecommunications networks, the Internet, local area networks (LANs), wide area networks (WANs), wireless local area networks (WLANs), metropolitan area networks (MANs), public switched telephone networks (PSTNs), Bluetooth networks, ZigBee networks, near field communication (NFC), internal device buses, internal device wiring, cable connections, etc., or any combination thereof. Network connections between components can employ one or more of these methods.

[0054] Storage device 550 may store data and / or instructions. In some embodiments, storage device 550 may store data obtained from puncture needle 510, head-mounted device 520, terminal 560, and / or processing device 530. For example, storage device 550 may store an inertial measurement dataset obtained from puncture needle 510. In some embodiments, storage device 550 may store data and / or instructions that processing device 530 may perform or be used to perform the exemplary methods described herein. In some embodiments, storage device 550 may be connected to network 540 to communicate with one or more components of puncture needle tracking system 500 (e.g., processing device 530, terminal 560, etc.). One or more components of puncture needle tracking system 500 may access the data or instructions stored in storage device 550 via network 540. In some embodiments, storage device 550 may be directly connected to or communicate with one or more components of puncture needle tracking system 500 (e.g., processing device 530, terminal 560, etc.). In some embodiments, storage device 550 may be part of processing device 530.

[0055] Terminal 560 may include mobile device 561, tablet computer 562, laptop computer 563, etc., or any combination thereof. In some embodiments, terminal 560 may remotely operate head-mounted device 520. In some embodiments, terminal 560 may operate head-mounted device 520 via wireless connection. In some embodiments, terminal 560 may receive information and / or instructions input by a user, and transmit the received information and / or instructions to head-mounted device 520 or processing device 530 via network 540.

[0056] It should be noted that the above description is provided for illustrative purposes only and is not intended to limit the scope of this specification. Various changes and modifications can be made by those skilled in the art based on the content of this specification. Features, structures, methods, and other features of the exemplary embodiments described herein can be combined in various ways to obtain other and / or alternative exemplary embodiments. However, these changes and modifications will not depart from the scope of this specification.

[0057] Figure 6 This is a block diagram of a puncture needle tracking device according to other embodiments of this specification. In some embodiments, Figure 6 The puncture needle tracking device 600 shown can be applied in software and / or hardware. Figure 5 The illustrated needle tracking system 500, for example, can be configured in software and / or hardware form to a head-mounted device 520, a processing device 530, and / or a terminal device 560 for needle tracking. Figure 6 As shown, the puncture needle tracking device 600 may include a receiving module 610 and an updating module 620.

[0058] The receiving module 610 can be used to receive multiple inertial measurement datasets acquired by the inertial measurement unit and multiple image datasets of a reference object acquired by the image acquisition device during the puncture of the target object. The reference object may include a marker set on the puncture needle and the user's hand.

[0059] The update module 620 can be used to update the first pose of the puncture needle model corresponding to the puncture needle in the virtual world coordinate system in response to receiving each inertial measurement dataset. For example, the update module 620 can determine the pose (or third pose) of the puncture needle in the Earth coordinate system based on the data in the inertial measurement dataset, and update the first pose of the puncture needle model in the virtual world coordinate system based on the third pose and the fifth transformation relationship between the Earth coordinate system and the virtual world coordinate system.

[0060] The update module 620 can be used to update the second pose of the puncture needle model in the virtual world coordinate system based on each received image dataset. For example, the update module 620 can determine whether marker tracking and hand tracking are valid based on the image dataset. If both marker tracking and hand tracking are valid, the update module 620 can perform a second pose update based on the markers in the image dataset and update the first transformation relationship between the hand coordinate system and the puncture needle coordinate system. Alternatively, if marker tracking is invalid but hand tracking is valid, the update module 620 can perform a second pose update based on the hand in the image dataset.

[0061] In some embodiments, in response to determining that marker tracking is valid, the update module 620 may also update the fifth transformation relationship between the Earth coordinate system and the virtual world coordinate system. During the first pose update, the update module 620 may determine the third pose of the puncture needle in the Earth coordinate system based on the inertial measurement dataset, and determine the updated pose of the puncture needle model in the virtual world coordinate system based on the third pose and the latest fifth transformation relationship.

[0062] In some embodiments, in response to the determination that both marker tracking and hand tracking are invalid, the update module 620 may not perform the second pose update based on the image dataset.

[0063] For more details about the above modules, please refer to other parts of this manual (e.g., Figures 3-10 (Parts and related descriptions), which will not be repeated here.

[0064] It should be noted that the above description of the puncture needle tracking device 600 is provided for illustrative purposes only and is not intended to limit the scope of this specification. It will be understood that those skilled in the art can, based on the description in this specification, arbitrarily combine the various modules, or construct subsystems and connect them to other modules, without departing from this principle. For example, Figure 6 The receiving module 610 and the updating module 620 described herein can be different modules in a single system, or a single module can implement the functions of two or more of the modules described above. Such variations are all within the scope of protection of this specification.

[0065] Figure 7 This is an exemplary flowchart of a needle tracking method according to some embodiments of this specification. In some embodiments, Figure 7 One or more operations in the process 700 shown can be performed Figure 5 This is performed in the puncture needle tracking system 500 shown. For example, Figure 7 The process 700 shown can be stored in storage device 550 as instructions and invoked and / or executed by processing device 530.

[0066] Process 700 can be executed during the process of a user puncturing a target object using a puncture needle (e.g., puncture needle 100, puncture needle 300, puncture needle 510, etc.), thereby mapping the movement and state changes of the puncture needle in the real world to the virtual world. The perception, calculation, and analysis of the real-world operational behavior of the puncture needle are achieved through a puncture needle model corresponding to the puncture needle in the virtual world. An inertial measurement unit is provided on the puncture needle.

[0067] In some embodiments, the target object may include biological and / or non-biological objects. For example, the object may include specific parts of the human body, such as the neck, chest, abdomen, etc., or combinations thereof. Another example is a human body model used in simulated teaching, skills testing, or other similar scenarios. Figure 7 As shown, process 700 may include the following steps.

[0068] Step 710 involves receiving multiple inertial measurement datasets acquired by the inertial measurement unit and multiple image datasets of a reference object acquired by the image acquisition device. In some embodiments, step 710 may be performed by the processing device 530 (e.g., the receiving module 610).

[0069] The multiple inertial measurement datasets can be acquired by an inertial measurement unit mounted on the puncture needle, and each inertial measurement dataset can include acceleration, angular velocity, and magnetic field data. For example, combined with... Figure 1 The inertial measurement unit on the puncture needle may include a gyroscope, an accelerometer, and a magnetometer. The gyroscope can be used to measure the angular velocity of the puncture needle rotating around its axes, the accelerometer can measure the linear acceleration of the puncture needle in each axis, and the magnetometer can be used to measure magnetic field data. By fusing the output data of the gyroscope, accelerometer, and magnetometer, the attitude quaternion of the puncture needle can be determined.

[0070] The multiple image datasets can be acquired by an image acquisition device (e.g., a camera), and each image dataset can include one or more images of the current scene. For example, the image acquisition device can be set on a head-mounted device worn by the user and acquire one or more images of the current scene during the puncture of the target object. Reference objects can include markers (marker 130, marker 330, etc.) set on the puncture needle and / or the user's hand.

[0071] The processing device 530 can receive the multiple inertial measurement datasets and the multiple image datasets in real time, and use them to update the pose of the puncture needle model in the virtual world coordinate system.

[0072] In some embodiments, multiple inertial measurement datasets can be acquired at a first frequency, and multiple image datasets can be acquired at a second frequency, where the first frequency is greater than the second frequency. For example, the first frequency can be 120 Hz, and the second frequency can be 30 Hz. By setting the first frequency to be greater than the second frequency, during subsequent puncture needle pose updates, the motion of the puncture needle can be interpolated and extrapolated using multiple inertial measurement datasets acquired at high frequencies within the acquisition interval of multiple image datasets. This overcomes tracking interruption problems caused by image acquisition delays, occlusion of reference objects, etc., improves the smoothness and real-time performance of pose updates, and ensures the continuous and stable tracking process of the puncture needle.

[0073] In some embodiments, before the user uses a puncture needle to puncture the target object, the initial pose of the puncture needle model in the virtual world coordinate system can be determined through an initialization process. For example, the puncture needle equipped with an inertial measurement unit can be left stationary in space for a period of time, and the inertial measurement data output by the inertial measurement unit can be recorded during this period. Based on the inertial measurement data, the processing device 530 can determine the rotational attitude of the inertial measurement unit. Further, images of the puncture needle and the markers set on it can be continuously acquired by an image acquisition device. When the pose information of the markers in the image is detected to have reached a stable state, the processing device 530 can determine the position and rotational attitude of the markers. Further, the processing device 530 can determine the rotational transformation relationship between the Earth coordinate system and the virtual world coordinate system based on the rotational attitude of the inertial measurement unit and the rotational attitude of the markers. Further, the processing device 530 can set the origin of the virtual world coordinate system to coincide with the origin of the Earth coordinate system, and then generate a coordinate transformation relationship (or a fifth transformation relationship) between the Earth coordinate system and the virtual world coordinate system based on the rotational transformation relationship. During the initialization process, when generating the coordinate transformation relationship between the Earth coordinate system and the virtual world coordinate system, the processing device 530 can also determine the initial position and initial orientation of the puncture needle in the virtual world coordinate system based on the position and rotational attitude of the marker, that is, determine the initial pose of the puncture needle model in the virtual world coordinate system. In some embodiments, the processing device 530 can also re-initialize in the above manner during the tracking process, resetting the coordinate transformation relationship between the Earth coordinate system and the virtual world coordinate system, thereby reducing or avoiding the cumulative deviation generated during the tracking process, and ensuring the accuracy and stability of the alignment between virtual and real space during long-term operation.

[0074] After determining the initial pose of the puncture needle model in the virtual world coordinate system, the pose of the puncture needle model can be updated based on the initial pose during the puncture of the target object, thereby achieving real-time tracking of the puncture needle. For example, in the current pose update process, the processing device 530 can update the pose of the puncture needle model determined in the previous pose update based on the inertial measurement dataset and / or image dataset; while when the pose update is performed for the first time, the pose determined in the previous pose update is the initial pose, and the processing device 530 can update the initial pose based on the inertial measurement dataset and / or image dataset.

[0075] Step 720: In response to receiving each of the inertial measurement datasets, update the first pose of the puncture needle model corresponding to the puncture needle in the virtual world coordinate system based on the inertial measurement datasets. In some embodiments, step 720 may be performed by processing device 530 (e.g., update module 620).

[0076] In some embodiments, in response to receiving each inertial measurement dataset, the processing device 530 can update the first pose of the puncture needle model corresponding to the puncture needle in the virtual world coordinate system based on the inertial measurement dataset. For example, the inertial measurement dataset may include acceleration, angular velocity, and magnetic field data. The processing device 530 can determine the pose (or third pose) of the puncture needle in the Earth coordinate system based on the data in the inertial measurement dataset, and update the first pose of the puncture needle model in the virtual world coordinate system based on the third pose and the fifth transformation relationship between the Earth coordinate system and the virtual world coordinate system. More information on the first pose update can be found in [link to relevant documentation]. Figure 10 The details and related descriptions will not be repeated here.

[0077] Step 730: In response to receiving each of the image datasets, a second pose update is performed on the puncture needle model in the virtual world coordinate system based on the image datasets. In some embodiments, step 730 may be performed by processing device 530 (e.g., update module 620).

[0078] In some embodiments, the reference objects include markers disposed on the puncture needle and the user's hand. To perform a second pose update on the puncture needle model in the virtual world coordinate system based on an image dataset, the processing device 530 can determine whether marker tracking and hand tracking are valid based on the image dataset. In response to determining that both marker tracking and hand tracking are valid, the processing device 530 can perform a second pose update based on the markers in the image dataset and update the transformation relationship (or first transformation relationship) between the hand coordinate system and the puncture needle coordinate system. In response to determining that marker tracking is invalid but hand tracking is valid, the processing device 530 can perform a second pose update based on the hand in the image dataset. For more information on performing a second pose update on the puncture needle model in the virtual world coordinate system based on an image dataset, please refer to [link to relevant documentation]. Figure 9 Its description will not be repeated here.

[0079] In some embodiments, steps 720 and 730 can be performed alternately based on the sequence of multiple received inertial measurement datasets and multiple image datasets of the reference object. For example, processing device 530 can execute step 720 each time an inertial measurement dataset is received; processing device 530 can execute step 730 each time an image dataset is received. In some embodiments, a first frequency for acquiring inertial measurement datasets is greater than a second frequency for acquiring image datasets. Accordingly, the frequency at which processing device 530 performs the first pose update described in step 720 can be greater than the frequency at which it performs the second pose update described in step 730. Thus, embodiments of this specification can interpolate and extrapolate the motion of the puncture needle using multiple inertial measurement datasets acquired at high frequencies within the acquisition interval of multiple image datasets, thereby overcoming tracking interruption problems caused by image acquisition delays, occlusion of the reference object, etc., improving the smoothness and real-time performance of pose updates, and ensuring the continuous and stable tracking process of the puncture needle.

[0080] In some embodiments, the pose update process of the puncture needle model can be implemented based on the Error State Kalman Filter (ESKF) algorithm. In the ESKF algorithm, the state of the puncture needle model can be divided into a nominal state and an error state. The nominal state reflects the predicted pose of the puncture needle model in the virtual world coordinate system, and the error state reflects the accuracy of the predicted pose. During pose update using the ESKF algorithm, the nominal state can be updated by performing prediction steps at high frequency based on the inertial measurement dataset, and the nominal state can be further updated (or corrected) by performing measurement steps at low frequency based on the image dataset, and the error state can also be updated. The execution of prediction steps to update the nominal state corresponds to the first pose update described above, and the execution of measurement steps to further update the nominal state corresponds to the second pose update described above. The latest nominal state determined after each prediction step or measurement step is used as the updated pose of the puncture needle model.

[0081] For example, in the prediction step, in response to receiving each inertial measurement dataset, the processing device 530 can acquire the previous state quantity maintained by the ESKF algorithm. The previous state quantity refers to the state quantity at the time of the last nominal state update (i.e., the pose of the puncture needle model at the time of the last pose update), which can be performed by either the prediction step or the measurement step. Further, the processing device 530 can predict the latest nominal state quantity based on the inertial measurement dataset and the previous state quantity; the latest nominal state quantity is the updated pose of the puncture needle model. In the measurement step, in response to receiving each image dataset, the processing device 530 can calculate the optimal estimate of the error state based on the image dataset and correct the result of the prediction step based on the optimal estimate, thereby updating the pose of the puncture needle model again. In some embodiments, after the measurement step uses the optimal estimate to correct the result of the prediction step, the error state can be reset to zero, and the covariance matrix of the system can be updated by calculating the Jacobian matrix of the reset function, so as to recalculate the optimal estimate of the error state when receiving the next image dataset.

[0082] Figure 8 This is a schematic diagram illustrating the pose update of a puncture needle model based on the ESKF algorithm, according to some embodiments of this specification. For example... Figure 8As shown, the pose update of the puncture needle model can be implemented using multithreading in the ESKF algorithm. The multithreading may include an inertial measurement data listening thread 810, an update thread 820, a marker tracking thread 830, and a hand tracking thread 840. The inertial measurement data listening thread 810 can acquire inertial measurement datasets at a first frequency (e.g., 120 Hz) and send them to the update thread 820. The update thread 820 can perform a prediction step based on each received inertial measurement dataset to update the first pose of the puncture needle model. The marker tracking thread 830 and the hand tracking thread 840 can perform marker tracking and hand tracking respectively at a second frequency (e.g., 30 Hz) and send the tracking results (e.g., marker pose, hand pose, etc.) to the update thread 820. The update thread 820 can perform a measurement step based on the received tracking results to update the second pose of the puncture needle model. During the execution of the measurement step, the second pose update can be performed based on at least one of the marker pose and the hand pose. For more information on second pose updates, please refer to [link / reference]. Figure 9 Its description will not be repeated here.

[0083] It should be noted that the above description of process 700 is for illustrative purposes only and does not limit the scope of this specification. Those skilled in the art can make various modifications and changes to process 700 under the guidance of this specification. However, these modifications and changes are still within the scope of this specification. In some embodiments, process 700 may also include one or more other steps. In some embodiments, one or more steps of process 700 may be omitted. For example, process 700 may include an initialization step for determining the initial pose of the puncture needle model in a virtual world coordinate system. As another example, steps 720 and 730 may not be performed in sequence. By way of example only, in response to receiving the image dataset first, step 730 may be performed before step 720. As another example, corresponding to the acquisition frequency of the inertial measurement dataset and the image dataset, step 730 may be performed after multiple executions of step 720.

[0084] Figure 9 This is an exemplary flowchart illustrating a second pose update based on an image dataset, according to some embodiments of this specification. In some embodiments, Figure 9 One or more operations in the process 900 shown can be performed Figure 5 This is performed in the puncture needle tracking system 500 shown. For example, Figure 9 The process 900 shown can be stored as instructions in storage device 550 and invoked and / or executed by processing device 530. In some embodiments, Figure 7Step 730 can be executed based on process 900. In some embodiments, process 900 can be executed by processing device 530 (e.g., update module 620). Figure 9 As shown, process 900 may include the following steps.

[0085] Step 910: Based on the image dataset, determine whether marker tracking and hand tracking are effective.

[0086] In some embodiments, the reference object may include a marker disposed on the puncture needle and the user's hand. Upon receiving the image dataset, the processing device 530 may track the image dataset to determine whether the image dataset contains a marker and / or a hand. For example, the processing device 530 may process the image dataset based on an image recognition algorithm to determine whether it contains a marker and / or a hand. In response to determining that the image dataset contains a marker, the processing device 530 may determine that marker tracking is valid; in response to determining that the image dataset contains a hand, the processing device 530 may determine that hand tracking is valid.

[0087] In some embodiments, marker tracking and hand tracking can be performed synchronously. For example, processing device 530 can synchronously track an image in an image dataset to determine whether the image contains a marker and a hand. Alternatively, the image dataset may include images corresponding to markers and images corresponding to hands; processing device 530 can track the images corresponding to markers and the images corresponding to hands separately to determine whether they contain markers and hands. In some embodiments, marker tracking and hand tracking can be performed asynchronously. For example, processing device 530 can perform marker tracking and hand tracking separately on an image in the image dataset at different time points. Alternatively, images corresponding to markers and images corresponding to hands can be acquired and received at different time points; processing device 530 can perform corresponding marker or hand tracking after receiving the image. In some embodiments, marker tracking and hand tracking can be performed at the same frequency. For example, the frequency of marker tracking and hand tracking can be equal to a second frequency used to acquire the image dataset.

[0088] Step 920: In response to determining that both the marker tracking and the hand tracking are effective, the second pose update is performed based on the markers in the image dataset, and the first transformation relationship between the hand coordinate system and the puncture needle coordinate system is updated.

[0089] In some embodiments, the processing device 530 may preferentially perform the second pose update based on markers. For example, when marker tracking is effective, the processing device 530 may perform the second pose update based on markers in the image dataset; while when marker tracking is ineffective but hand tracking is effective, the processing device 530 may perform the second pose update based on the hand in the image dataset.

[0090] In some embodiments, the image acquisition device for acquiring the image dataset is disposed on a head-mounted device worn on the user's head; therefore, the image acquisition device can correspond to a head coordinate system. To perform a second pose update based on markers in the image dataset, the processing device 530 can determine the first pose of the markers in the head coordinate system based on the image dataset. For example, the processing device 530 can identify markers in the image dataset and determine the first pose of the markers in the head coordinate system based on the identification result using a perspective-n-point (PnP) algorithm. Further, the processing device 530 can determine the second pose of the markers in the virtual world coordinate system based on the first pose and a second transformation relationship between the head coordinate system and the virtual world coordinate system.

[0091] The second transformation relationship between the head coordinate system and the virtual world coordinate system can be provided by the head-mounted device. For example, the head-mounted device can construct a virtual world coordinate system and establish its spatial origin during system initialization, and perform self-localization through environmental perception cameras and motion sensors installed on it to determine the pose of the head-mounted device in the virtual world coordinate system, thereby providing the second transformation relationship between the head coordinate system and the virtual world coordinate system.

[0092] Furthermore, the processing device 530 can determine the updated pose of the puncture needle model in the virtual world coordinate system based on the second pose and the third transformation relationship between the marker coordinate system and the puncture needle coordinate system. Since the marker is set at a fixed, known position on the puncture needle, the relative pose between the marker and the puncture needle can be determined, that is, the third transformation relationship between the marker coordinate system and the puncture needle coordinate system can be determined. Based on the third transformation relationship, the real-time pose of the puncture needle model in the virtual world coordinate system (i.e., the updated pose of the puncture needle model) can be directly derived given the known second pose of the marker in the virtual world coordinate system.

[0093] In some embodiments, when both marker tracking and hand tracking are effective, the processing device 530 can perform a second pose update based on markers in the image dataset, and can also update the first transformation relationship between the hand coordinate system and the puncture needle coordinate system. For example, the processing device 530 can identify the hand and puncture needle in the image dataset, determine the relative pose between the hand and the puncture needle, and thus determine or update the first transformation relationship between the hand coordinate system and the puncture needle coordinate system. In some embodiments, the processing device 530 can update the first transformation relationship between the hand coordinate system and the puncture needle coordinate system every time both marker tracking and hand tracking are effective. With this configuration, when marker tracking is ineffective or the puncture needle cannot be identified due to factors such as occlusion, the processing device 530 can treat the relative spatial position between the puncture needle and the hand as fixed, and thus can determine the pose of the puncture needle model in the virtual world coordinate system based solely on the hand tracking results and the latest first transformation relationship. This configuration enhances the robustness and continuity of the tracking system in complex and non-ideal operating environments.

[0094] In some embodiments, in response to determining that marker tracking is valid, the processing device 530 can also update the fifth transformation relationship between the Earth coordinate system and the virtual world coordinate system. For example, in response to determining that marker tracking is valid, the processing device 530 can update the rotational transformation relationship in the fifth transformation relationship. Specifically, since the inertial measurement unit has accumulated errors when calculating rotational changes, that is, there is a certain degree of rotational attitude drift, and the degree of rotational attitude drift increases over time, it affects the accuracy of updating the pose of the puncture needle model based on inertial measurement data. However, the attitude of the marker set on the puncture needle can be directly calculated based on algorithms such as PnP, and will not have accumulated errors due to historical motion, providing a drift-free direction reference in the Earth coordinate system. Therefore, in response to determining that marker tracking is valid, the processing device 530 can update the rotational transformation relationship in the fifth transformation relationship based on the current rotational attitude of the marker, thereby ensuring the accuracy of the fifth transformation relationship between the current Earth coordinate system and the virtual world coordinate system. The updated fifth transformation relationship can be used for the next first pose update of the puncture needle model based on the inertial measurement dataset, thereby improving the accuracy of the first pose update.

[0095] In some embodiments, the processing device 530 may update the fifth transformation relationship between the Earth coordinate system and the virtual world coordinate system each time marker tracking is successful. With this configuration, when performing the first pose update, the processing device 530 can determine the pose of the puncture needle model in the virtual world coordinate system based on the inertial measurement dataset and the latest fifth transformation relationship, thus improving the accuracy of the first pose update.

[0096] Step 930: In response to determining that the marker tracking is invalid but the hand tracking is valid, the second pose update is performed based on the hand in the image dataset.

[0097] In some embodiments, when marker tracking is invalid but hand tracking is valid, the processing device 530 can perform a second pose update based on the hand in the image dataset. To perform the second pose update based on the hand in the image dataset, the processing device 530 can determine a fourth transformation relationship between the hand coordinate system and the virtual world coordinate system based on the image dataset, obtain the latest first transformation relationship between the hand coordinate system and the puncture needle coordinate system, and determine the updated pose of the puncture needle model in the virtual world coordinate system based on the fourth transformation relationship and the latest first transformation relationship.

[0098] To determine the fourth transformation relationship between the hand coordinate system and the virtual world coordinate system, the processing device 530 can extract and reconstruct a 3D model of the hand from the image dataset to obtain the local pose of the hand. It then uses an environmental awareness camera and motion sensors on the head-mounted device to perform self-localization of the head-mounted device. Finally, it fuses the local pose of the hand with the self-localization data of the head-mounted device to obtain the pose of the hand in the virtual world coordinate system, i.e., the fourth transformation relationship between the hand coordinate system and the virtual world coordinate system. In some embodiments, the processing device 530 can determine the fourth transformation relationship (or left-hand transformation relationship) between the hand coordinate system corresponding to the left hand and the virtual world coordinate system (or right-hand transformation relationship) based on the image dataset. When performing a second pose update based on the hand in the image dataset, the processing device 530 can first determine whether the hand operating the puncture needle in the image dataset is the left or right hand, and then perform the second pose update based on the corresponding fourth transformation relationship.

[0099] The latest first transformation relationship between the hand coordinate system and the puncture needle coordinate system refers to the first transformation relationship after the most recent update. For example, as described in step 920, when both marker tracking and hand tracking are effective, the processing device 530 can perform a second pose update based on the markers in the image dataset and update the first transformation relationship between the hand coordinate system and the puncture needle coordinate system based on the hand in the image dataset. Therefore, the updated first transformation relationship determined in the most recent execution of step 920 can be used as the current latest first transformation relationship.

[0100] Furthermore, the processing device 530 can determine the updated pose of the puncture needle model in the virtual world coordinate system based on the fourth transformation relation and the latest first transformation relation.

[0101] Some embodiments of this specification utilize the hand as a tracking marker when marker tracking is invalid, avoiding system interruptions or resets caused by invalid marker tracking, and effectively maintaining the continuity and robustness of the tracking process. Furthermore, some embodiments of this specification can update the first transformation relationship between the hand coordinate system and the puncture needle coordinate system each time both marker tracking and hand tracking are effective, and use the latest first transformation relationship for a second pose update when marker tracking is invalid but hand tracking is effective, improving the accuracy of the second pose update.

[0102] In some embodiments, when performing a second pose update based on the hand in the image dataset, the processing device 530 may update only the position parameters of the puncture needle model in the virtual world coordinate system based on the hand in the image dataset, and update the attitude parameters of the puncture needle model in the virtual world coordinate system using the most recently obtained inertial measurement dataset. For example, the processing device 530 may determine a fourth transformation relationship between the hand coordinate system and the virtual world coordinate system based on the image dataset. This fourth transformation relationship may include only the position transformation relationship between the hand coordinate system and the virtual world coordinate system, or it may include both the position transformation relationship and the attitude transformation relationship between the hand coordinate system and the virtual world coordinate system. When performing the second pose update, the position parameters of the puncture needle model in the virtual world coordinate system may be updated using only the position transformation relationship between the hand coordinate system and the virtual world coordinate system, and the attitude parameters of the puncture needle model in the virtual world coordinate system may be updated using the most recently obtained inertial measurement dataset.

[0103] In some embodiments, the fourth transformation relationship may include the position transformation relationship and the attitude transformation relationship between the hand coordinate system and the virtual world coordinate system. When performing the second pose update, the position parameters and attitude parameters of the puncture needle model in the virtual world coordinate system may be updated by using the position transformation relationship and the attitude transformation relationship between the hand coordinate system and the virtual world coordinate system respectively, and the attitude parameters may be further corrected based on the inertial measurement dataset.

[0104] Some embodiments in this specification utilize hand data from image datasets to update the position parameters of the puncture needle model in the virtual world coordinate system. This effectively avoids the position drift problem caused by using inertial measurement data for position parameter updates, ensuring that the movement of the puncture needle model in the virtual world coordinate system is precisely synchronized with the movement of the hand in the real world. Simultaneously, since the inertial measurement data collected by the inertial measurement unit mounted on the puncture needle reliably reflects the rotational attitude of the puncture needle, using inertial measurement data to update or correct the attitude parameters of the puncture needle model in the virtual world coordinate system can improve the accuracy of attitude parameter updates. Furthermore, the high-frequency characteristics of inertial measurement data solve the attitude lag problem caused by image tracking processing delays, thus improving the timeliness of attitude parameter updates.

[0105] In some embodiments, in response to determining that both marker tracking and hand tracking are invalid, the processing device 530 may not perform a second pose update based on the image dataset. For example, when both marker tracking and hand tracking are invalid, the processing device 530 may update the puncture needle model in the virtual world coordinate system based only on the inertial measurement dataset, discarding the image dataset.

[0106] It should be noted that the above description of process 900 is for illustrative purposes only and does not limit the scope of this specification. Those skilled in the art can make various modifications and changes to process 900 under the guidance of this specification. However, these modifications and changes are still within the scope of this specification. In some embodiments, process 900 may also include one or more other steps. In some embodiments, one or more steps of process 900 may be omitted. For example, process 900 may include a step for determining whether the puncture needle is operated by the left or right hand.

[0107] Figure 10 This is an exemplary flowchart illustrating first pose update based on an inertial measurement dataset according to some embodiments of this specification. In some embodiments, Figure 10 One or more operations in the process 1000 shown can be performed Figure 5 This is performed in the puncture needle tracking system 500 shown. For example, Figure 10 The process 1000 shown can be stored as instructions in storage device 550 and invoked and / or executed by processing device 530. In some embodiments, Figure 7 Step 720 can be executed based on process 1000. In some embodiments, process 1000 can be executed by processing device 530 (e.g., update module 620). Figure 10 As shown, process 1000 may include the following steps.

[0108] Step 1010: Based on the inertial measurement dataset, determine the third pose of the puncture needle in the Earth coordinate system.

[0109] In some embodiments, to determine the third pose of the puncture needle in the Earth coordinate system, the processing device 530 can calculate the pose of the inertial measurement unit (IMU) in the Earth coordinate system based on an inertial measurement dataset. For example, the processing device 530 can calculate the rotational attitude (e.g., attitude quaternion) of the IMU using angular velocity data collected by an integrating gyroscope, and calculate the position change of the IMU based on acceleration data, thereby obtaining the pose of the IMU in the Earth coordinate system. Further, the processing device 530 can determine the third pose of the puncture needle in the Earth coordinate system based on the pose of the IMU in the Earth coordinate system and the transformation relationship between the IMU coordinate system and the puncture needle coordinate system. For example, since the IMU is positioned at a fixed, known location on the puncture needle, the relative pose between the IMU and the puncture needle can be determined, i.e., the transformation relationship between the IMU coordinate system and the puncture needle coordinate system can be determined. Based on this transformation relationship, the third pose of the puncture needle in the Earth coordinate system can be directly derived given the known pose of the IMU in the Earth coordinate system.

[0110] Step 1020: Based on the third pose and the latest fifth transformation relationship between the Earth coordinate system and the virtual world coordinate system, determine the updated pose of the puncture needle model in the virtual world coordinate system.

[0111] The latest fifth transformation relationship between the Earth coordinate system and the virtual world coordinate system refers to the fifth transformation relationship after the most recent update. For example, as... Figure 9 As described in step 920, when marker tracking is effective, the processing device 530 can update the fifth transformation relationship between the Earth coordinate system and the virtual world coordinate system. Therefore, the updated fifth transformation relationship determined during the most recent execution of step 920 can be used as the current latest fifth transformation relationship.

[0112] Based on the third pose of the puncture needle in the Earth coordinate system and the latest fifth transformation relationship between the Earth coordinate system and the virtual world coordinate system, the processing device 530 can determine the updated pose of the puncture needle model corresponding to the puncture needle in the virtual world coordinate system.

[0113] The embodiments in this specification utilize multiple high-frequency acquired inertial measurement datasets to interpolate and extrapolate the motion of the puncture needle. This overcomes tracking interruptions caused by image acquisition delays and occlusion of reference objects, improving the smoothness and real-time performance of pose updates and ensuring the continuous and stable tracking process of the puncture needle. However, since the position information calculated based on inertial measurement data has significant drift, prolonged use of inertial measurement datasets for puncture needle model pose updates may affect the accuracy of the update results. Therefore, in some embodiments, the processing device 530 can update the first pose based on the historical execution count of the first pose update.

[0114] Specifically, in response to each received inertial measurement dataset, the processing device 530 can determine the number of times the first pose update has been performed since the last second pose update, that is, the number of times the pose update has been performed based on the inertial measurement dataset since the last pose update based on the image dataset. Further, the processing device 530 can determine whether the number of times the first pose update has been performed exceeds a threshold. The threshold can be a preset empirical value. In response to determining that the number of executions exceeds the threshold, the processing device 530 can update only the attitude parameters of the puncture needle model in the virtual world coordinate system based on the inertial measurement dataset, without updating the position parameters of the puncture needle model in the virtual world coordinate system. In response to determining that the number of executions does not exceed the threshold, the processing device 530 can update both the attitude and position parameters of the puncture needle model in the virtual world coordinate system based on the inertial measurement dataset. Because the position information calculated based on inertial measurement data has significant drift, and this drift increases over time, the reliability of the position information obtained from the inertial measurement dataset is low after the number of times the first pose update is executed exceeds a threshold. In this case, only the attitude parameters of the puncture needle model in the virtual world coordinate system can be updated based on the inertial measurement dataset, without updating the position parameters. This reduces the impact of position drift on the accuracy of the first pose update. In some embodiments, the processing device 530 can also detect whether the inertial measurement unit is stationary before updating the first pose. If it determines that the inertial measurement unit is stationary, the first pose update is not performed.

[0115] It should be noted that the above description of process 1000 is for illustrative purposes only and does not limit the scope of this specification. Those skilled in the art can make various modifications and changes to process 1000 under the guidance of this specification. However, these modifications and changes remain within the scope of this specification. In some embodiments, process 1000 may also include one or more other steps.

[0116] Some embodiments of this specification also provide a computer-readable storage medium that can store computer instructions. When a computer reads the computer instructions from the storage medium, the computer can execute the corresponding process of the puncture needle tracking method as described in any of the foregoing embodiments.

[0117] The basic concepts have been described above. Obviously, for those skilled in the art, the detailed disclosure above is merely illustrative and does not constitute a limitation of this specification. Although not explicitly stated herein, those skilled in the art may make various modifications, improvements, and corrections to this specification. Such modifications, improvements, and corrections are suggested in this specification and therefore remain within the spirit and scope of the exemplary embodiments described herein.

[0118] Furthermore, this specification uses specific terms to describe embodiments thereof. For example, "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic associated with at least one embodiment of this specification. Therefore, it should be emphasized and noted that references to "an embodiment," "one embodiment," or "an alternative embodiment" in different locations throughout this specification do not necessarily refer to the same embodiment. Moreover, certain features, structures, or characteristics in one or more embodiments of this specification can be appropriately combined.

[0119] Furthermore, those skilled in the art will understand that various aspects of this specification can be described and illustrated in several patentable ways or situations, including any new and useful combination of processes, machines, products, or substances, or any new and useful improvements thereof. Accordingly, various aspects of this specification can be implemented entirely by hardware, entirely by software (including firmware, resident software, microcode, etc.), or by a combination of hardware and software. All of the above hardware or software may be referred to as a “data block,” “module,” “engine,” “unit,” “component,” or “system.” Furthermore, various aspects of this specification may be represented as a computer product located on one or more computer-readable media, including computer-readable program code.

[0120] Computer storage media may contain a propagated data signal containing computer program code, for example, on baseband or as part of a carrier wave. This propagated signal may take various forms, including electromagnetic, optical, and suitable combinations thereof. Computer storage media can be any computer-readable medium other than a computer-readable storage medium, which can be connected to an instruction execution system, apparatus, or device to enable communication, propagation, or transmission of a program for use. The program code located on the computer storage medium can be propagated through any suitable medium, including radio, cable, fiber optic cable, RF, or similar media, or any combination of the above media.

[0121] The computer program code required for the operation of each part of this manual can be written in any one or more programming languages, including object-oriented programming languages ​​such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C#, VB.NET, Python, etc.; conventional procedural programming languages ​​such as C, Visual Basic, Fortran 2003, Perl, COBOL 2002, PHP, ABAP; dynamic programming languages ​​such as Python, Ruby, and Groovy; or other programming languages. This program code can run entirely on the user's computer, or as a standalone software package on the user's computer, or partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer through any network, such as a local area network (LAN) or wide area network (WAN), or connected to an external computer (e.g., via the Internet), or in a cloud computing environment, or used as a service such as Software as a Service (SaaS).

[0122] Furthermore, unless expressly stated in the claims, the order of processing elements and sequences, the use of numbers and letters, or other names described in this specification are not intended to limit the order of the processes and methods described herein. Although various examples have been discussed in the foregoing disclosure of some embodiments of the invention that are currently considered useful, it should be understood that such details are for illustrative purposes only, and the appended claims are not limited to the disclosed embodiments; rather, the claims are intended to cover all modifications and equivalent combinations that conform to the spirit and scope of the embodiments described herein. For example, while the system components described above can be implemented using hardware devices, they can also be implemented solely using software solutions, such as installing the described system on existing servers or mobile devices.

[0123] Similarly, it should be noted that, in order to simplify the description disclosed herein and thus aid in the understanding of one or more embodiments of the invention, the foregoing description of embodiments in this specification may sometimes combine multiple features into a single embodiment, drawing, or description thereof. However, this method of disclosure does not imply that the subject matter of this specification requires more features than those mentioned in the claims. In fact, the embodiments contain fewer features than all the features of a single embodiment disclosed above.

[0124] In some embodiments, numbers describing the quantity of components and attributes are used. It should be understood that such numbers used in the description of embodiments are modified in some examples with the terms "approximately," "approximately," or "generally." Unless otherwise stated, "approximately," "approximately," or "generally" indicates that the numbers are allowed to vary by ±20%. Accordingly, in some embodiments, the numerical parameters used in the specification and claims are approximate values, which may be changed depending on the characteristics required by individual embodiments. In some embodiments, numerical parameters should take into account specified significant digits and employ a general method of digit reservation. Although the numerical ranges and parameters used to confirm their breadth of range in some embodiments of this specification are approximate values, in specific embodiments, such values ​​are set as precisely as feasible.

[0125] For each patent, patent application, patent application publication, and other material such as articles, books, specifications, publications, and documents referenced in this specification, the entire contents of which are incorporated herein by reference. This excludes historical application documents that are inconsistent with or conflict with the content of this specification, as well as documents that limit the broadest scope of the claims in this specification (currently or subsequently appended to this specification). It should be noted that in the event of any inconsistency or conflict between the descriptions, definitions, and / or terminology used in the supplementary materials to this specification and the content of this specification, the descriptions, definitions, and / or terminology used in this specification shall prevail.

[0126] Finally, it should be understood that the embodiments described in this specification are merely illustrative of the principles of the embodiments described herein. Other variations may also fall within the scope of this specification. Therefore, alternative configurations of the embodiments described herein are intended to be illustrative rather than limiting, and should be considered consistent with the teachings of this specification. Accordingly, the embodiments described herein are not limited to those explicitly introduced and described herein.

Claims

1. A puncture needle tracking method characterized by, The puncture needle is provided with an inertial measurement unit, and the method comprises: During the puncture of the puncture needle on the target object, a plurality of inertial measurement data sets collected by the inertial measurement unit and a plurality of image data sets of a reference object collected by an image acquisition device are received; In response to receiving each of the inertial measurement data sets, a first pose update of a puncture needle model corresponding to the puncture needle in a virtual world coordinate system is performed based on the inertial measurement data set; and In response to receiving each of the image data sets, a second pose update of the puncture needle model in the virtual world coordinate system is performed based on the image data set.

2. The puncture needle tracking method of claim 1, wherein, The plurality of inertial measurement data sets are collected at a first frequency, and the plurality of image data sets are collected at a second frequency, and the first frequency is greater than the second frequency.

3. The puncture needle tracking method of claim 1, wherein, The reference object comprises a marker provided on the puncture needle and a hand of a user, and the second pose update of the puncture needle model in the virtual world coordinate system based on the image data set comprises: Based on the image data set, it is determined whether the marker tracking and the hand tracking are valid; In response to determining that the marker tracking and the hand tracking are both valid, the second pose update is performed based on the marker in the image data set, and a first conversion relationship between a hand coordinate system and a puncture needle coordinate system is updated; or In response to determining that the marker tracking is invalid but the hand tracking is valid, the second pose update is performed based on the hand in the image data set.

4. The puncture needle tracking method of claim 3, wherein, The image acquisition device corresponds to a head coordinate system, and the second pose update based on the marker in the image data set comprises: Based on the image data set, a first pose of the marker in the head coordinate system is determined; Based on the first pose of the marker in the head coordinate system and a second conversion relationship between the head coordinate system and the virtual world coordinate system, a second pose of the marker in the virtual world coordinate system is determined; and Based on the second pose and a third conversion relationship between a marker coordinate system and a puncture needle coordinate system, an updated pose of the puncture needle model in the virtual world coordinate system is determined.

5. The puncture needle tracking method of claim 3, wherein, The second pose update based on the hand in the image data set comprises: Based on the hand in the image data set, a position parameter of the puncture needle model in the virtual world coordinate system is updated; and Based on the inertial measurement data set, an attitude parameter of the puncture needle model in the virtual world coordinate system is updated.

6. The puncture needle tracking method of claim 3, wherein, The second pose update based on the hand in the image data set comprises: Based on the image data set, a fourth conversion relationship between a hand coordinate system and the virtual world coordinate system is determined; The latest first conversion relationship between the hand coordinate system and the puncture needle coordinate system is obtained; and Based on the fourth conversion relationship and the latest first conversion relationship, an updated pose of the puncture needle model in the virtual world coordinate system is determined.

7. The puncture needle tracking method of claim 3, wherein, Further comprising: In response to determining that the marker tracking is valid, a fifth conversion relationship between a terrestrial coordinate system and the virtual world coordinate system is updated, The first pose update of the puncture needle model corresponding to the puncture needle in the virtual world coordinate system based on the inertial measurement data set comprises: determining a third pose of the puncture needle in the earth coordinate system based on the inertial measurement data set; and determining an updated pose of the puncture needle model in the virtual world coordinate system based on the third pose and the latest fifth conversion relationship.

8. The puncture needle tracking method of claim 3, further comprising, in response to judging that both the marker tracking and the hand tracking are invalid, not performing the second pose update based on the image data set.

9. The puncture needle tracking method of claim 1, wherein, The first pose update of the puncture needle model corresponding to the puncture needle in the virtual world coordinate system based on the inertial measurement data set in response to receiving each of the inertial measurement data set comprises: in response to receiving each of the inertial measurement data set, determining a number of times of execution of the first pose update after the last time of performing the second pose update; judging whether the number of times of execution exceeds a threshold value; in response to judging that the number of times of execution exceeds the threshold value, updating a pose parameter of the puncture needle model in the virtual world coordinate system based on the inertial measurement data set; in response to judging that the number of times of execution does not exceed the threshold value, updating a pose parameter and a position parameter of the puncture needle model in the virtual world coordinate system based on the inertial measurement data set.

10. A computer-readable storage medium, characterized in that, The storage medium stores computer instructions, and after a computer reads the computer instructions in the storage medium, the computer executes the puncture needle tracking method according to any one of claims 1-9.

Images