Robot surgical navigation system and method based on interactive registration

Abstract

The invention provides a robot surgical navigation system and method based on interactive registration, and relates to the technical field of medical robots, the system comprises a robot main hand, an imaging device, a pose sensor, a control device and a display device; wherein the control device comprises an initial registration module, and the initial registration module is used for carrying out pose adjustment on the virtual model in the three-dimensional medical image displayed by the display device; the virtual fusion module is used for superposing the virtual model into a real-time picture captured by the imaging device; the interactive registration module is used for carrying out pose adjustment on the virtual model in the combined image presented by the display device; and the dynamic tracking module enables the virtual model to be in space alignment with the target object in the real-time picture. Natural and flexible man-machine interaction can be realized through the robot master hand, and continuous and accurate alignment of the virtual model and a real target object can be realized in an augmented reality environment, so that the stability, precision and practicability of intraoperative guidance are improved.

Description

Technical Field

[0002] This invention relates to the technical field of medical robots, and in particular to a robotic surgical navigation system and method based on interactive registration. Background Technology

[0004] In computer-assisted surgery, augmented reality (AR) technology has been gradually introduced into clinical applications to overcome the disconnect between traditional imaging examinations (such as CT and MRI) and real-time intraoperative vision. This technology aims to fuse and overlay three-dimensional virtual models acquired before or during surgery with real-time video of the surgical field, thereby helping doctors "see through" the tissue surface and intuitively observe key anatomical structures underneath, such as blood vessels, nerves, or tumors, to improve the accuracy and safety of the surgery.

[0005] However, existing AR-assisted surgical systems still have some problems in practical applications. For example, how to achieve accurate registration between virtual models and real anatomical structures; in addition, once the position of the imaging device changes during surgery, the established registration relationship will become invalid, making it difficult to achieve continuous and accurate overlay in dynamic scenes. Summary of the Invention

[0007] The purpose of this application is to provide a robotic surgical navigation system and method based on interactive registration, which can achieve natural and flexible human-computer interaction through the robot master hand, reduce the operational difficulty of spatial registration, and achieve continuous and accurate alignment between virtual models and real target objects in augmented reality environment, thereby improving the stability, accuracy and practicality of intraoperative guidance.

[0008] In a first aspect, embodiments of this application provide a robotic surgical navigation system based on interactive registration, comprising: The robot's main arm is used to receive user input. Imaging devices are used to capture real-time images containing target objects; A pose sensor is disposed on the imaging device to acquire the real-time pose of the imaging device; The control device is communicatively connected to the robot's main hand, the imaging device, and the pose sensor. The display device is communicatively connected to the control device; The control device includes: An initial registration module is used to adjust the pose of the virtual model in the three-dimensional medical image presented by the display device in response to the user's operation on the robot's main hand, so that the virtual model is registered with the target object in the three-dimensional medical image to determine the initial pose of the virtual model, wherein the virtual model can characterize the internal anatomical structure of the target object; The virtual fusion module is used to overlay the virtual model onto the real-time image captured by the imaging device according to the initial pose, and generate and output the combined image to the display device; An interactive registration module is used to adjust the pose of the virtual model in the combined image presented by the display device in response to the user's operation through the robot's main hand, so that the virtual model is registered with the target object in the real-time screen; The binding module is used to establish a binding relationship between the virtual model and the target object in the real-time screen when the interactive registration module completes the registration. The dynamic tracking module is used to map the pose change of the imaging device to the pose update of the virtual model through coordinate transformation based on the real-time pose data of the imaging device obtained by the pose sensor after the binding relationship is established, so as to keep the virtual model spatially aligned with the target object in the real-time image.

[0009] Furthermore, the control device also includes a virtual modeling module, which comprises: A model generation unit is used to process a three-dimensional medical image containing the target object to generate the virtual model; The feature marking unit is used to pre-define multiple first key points for registration on the virtual model.

[0010] Furthermore, the first key point is located at the edge of the anatomical structure of the virtual model, and multiple first key points are distributed at intervals on the edge of the anatomical structure.

[0011] Furthermore, the first key point is specified by receiving user input instructions.

[0012] Furthermore, the interactive registration module includes: A key point marking unit is used to mark a second key point on the target object in the real-time image, and the second key point corresponds one-to-one with a predefined first key point in the virtual model; An auxiliary registration unit is used to adjust the pose of the virtual model in response to the user's operation through the robot's main hand, so that the first key point on the virtual model and the second key point marked in the real-time image meet preset alignment conditions.

[0013] Furthermore, the interactive registration module also includes a visual feedback unit, used to highlight the first key point and the second key point when adjusting the pose of the virtual model, and / or update the distance information between them in real time.

[0014] Furthermore, the key point marking unit is a manual marking unit, used to mark a second key point on the target object in the real-time image in response to a marking instruction input by the user through the robot's main hand.

[0015] Furthermore, the key point marking unit is a feature extraction unit, used to run an image recognition algorithm to automatically identify the anatomical features of the target object in the real-time image, and extract and mark the second key point corresponding to the predefined first key point in the virtual model.

[0016] Furthermore, the pose sensor employs an optical sensor or an electromagnetic tracker.

[0017] Furthermore, the robot's main hand is an operating handle with multiple degrees of freedom; The control device has a preset operation mapping relationship, which is used to convert the physical motion parameters of the operating handle in at least one degree of freedom into pose adjustment commands for the virtual model. The physical motion parameters include at least one of displacement, rotation, and opening / closing degree.

[0018] Furthermore, the control device is configured to provide at least two switchable operating modes, including: In the pose adjustment mode, in response to the movement or rotation of the operating handle, the operation mapping relationship is configured to cause the virtual model to perform a translation or rotation that is consistent with the movement direction of the operating handle and the movement amplitude is in a preset proportion. In the scaling control mode, in response to the opening and closing action of the trigger component on the operating handle used for simulating gripping, the virtual model produces a corresponding scaling motion.

[0019] Furthermore, the control device also includes an interactive cropping module; The interactive cropping module is used to respond to the user's cropping command, generate a closed area drawn by the user on the combined image of the display device, and crop the virtual model in the combined image based on the closed area to display a portion of the model located within the closed area on the display device.

[0020] Secondly, embodiments of this application provide a method for a robotic surgical navigation system based on interactive registration, comprising: In response to the user's operation on the robot's main hand, the virtual model is pose-adjusted in the three-dimensional medical image presented on the display device, so that the virtual model is registered with the target object in the three-dimensional medical image to determine the initial pose of the virtual model, wherein the virtual model can characterize the internal anatomical structure of the target object; Based on the initial pose, the virtual model is superimposed onto the real-time image captured by the imaging device to generate and output the combined image to the display device; In response to the user's operation via the robot's main hand, the virtual model is pose-adjusted in the combined image presented on the display device, so that the virtual model is registered with the target object in the real-time image; When registering the virtual model with the target object in the real-time image, a binding relationship is established between the virtual model and the target object in the real-time image; After the binding relationship is established, based on the real-time pose data of the imaging device, the pose change of the imaging device is mapped to the pose update of the virtual model through coordinate transformation, so that the virtual model maintains spatial alignment with the target object in the real-time image.

[0021] Furthermore, the step of adjusting the pose of the virtual model in the combined image to register the virtual model with the target object in the real-time image includes: Obtain multiple first key points preset on the virtual model; Mark a second key point on the target object in the real-time image, which corresponds one-to-one with the first key point; In response to the user's operation through the robot's main hand, the pose of the virtual model is adjusted so that the first key point on the virtual model and the second key point on the target object meet the preset alignment conditions.

[0022] Furthermore, it also includes: In response to a user's cropping command, a closed region drawn by the user is generated on the composite image of the display device, and the virtual model in the composite image is cropped based on the closed region to display a portion of the model located within the closed region on the display device.

[0023] Thirdly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the second aspect above.

[0024] The robotic surgical navigation system and method based on interactive registration provided in this application have at least the following beneficial effects: Compared to existing technologies, this robotic surgical navigation system based on interactive registration uses the robot's master hand as an intuitive interface, allowing users to naturally and flexibly manipulate the virtual model to complete initial registration and interactive adjustments, significantly reducing the operational difficulty of complex spatial registration. In the initial registration module, the virtual model's pose is adjusted within the 3D medical image presented on the display device, aligning the virtual model with the target object in the 3D medical image and determining the initial pose of the virtual model. This effectively reduces the initial deviation between the virtual model and the real target object when the virtual model is superimposed onto the real-time image during the virtual fusion stage. Based on this, only minor adjustments to the virtual model are needed during the interactive registration stage to quickly achieve precise alignment, greatly improving the alignment accuracy and reliability of augmented reality.

[0025] Furthermore, by establishing a binding relationship between the virtual model and the target object in the real-time image through the binding module, synchronous movement between the two can be achieved. Combined with the real-time acquisition of the imaging device's pose data by the pose sensor (such as adjusting the imaging device's distance and pitch angles), the dynamic tracking module maps the pose changes of the imaging device to the pose updates of the virtual model through coordinate transformation. This ensures that the virtual model remains spatially aligned with the target object in the real-time image, guaranteeing that even when the imaging device moves, the virtual model remains continuously and accurately aligned with the real anatomical structure. Since the virtual model can represent the internal anatomical structure of the target object, this system not only enhances the stability and practicality of intraoperative image guidance but also provides surgeons with a "fluoroscopic" ability invisible to the naked eye, allowing them to simultaneously observe superficial tissues and deep key structures during surgery. This provides strong support for the precise execution of lesion resection or localization. Attached Figure Description

[0027] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0028] Figure 1 A schematic diagram of the structure of a robotic surgical navigation system based on interactive registration provided in an embodiment of this application; Figure 2 This is a structural block diagram of the control device; Figure 3 This is a structural block diagram of the virtual modeling module; Figure 4 A structural block diagram of the interactive registration module; Figure 5A flowchart of the robot-assisted surgical navigation method provided in the embodiments of this application; Figure 6 This is a schematic diagram showing the display effect generated using the robot-assisted surgical navigation system provided in the embodiments of this application.

[0029] icon: 10-Liver; 11-Real-time view of the liver; 12-Virtual model section; 100 - Doctor's console; 101 - Robotic main hand; 102 - Display device; 200 - Patient surgical platform; 201 - Imaging device; 300 - Control device; 301 - Virtual modeling module; 302 - Initial registration module; 303 - Virtual fusion module; 304 - Interactive registration module; 305 - Binding module; 306 - Dynamic tracking module; 307 - Interactive cropping module; 311 - Model generation unit; 312 - Feature labeling unit; 341 - Key point marking unit; 342 - Auxiliary registration unit; 343 - Visual feedback unit. Detailed Implementation

[0031] The technical solutions of this application will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0032] In the field of modern medical robotics, achieving precise fusion of virtual models and real-time intraoperative images is a key step in improving the accuracy and quality of surgical navigation. However, in the actual implementation process, the inventors found that existing augmented reality-based assisted surgical systems still have several problems, such as how to achieve precise registration between virtual models and real anatomical structures; or, during the operation, once the pose of the imaging device 201 changes, the established registration relationship will become invalid, making it difficult to achieve continuous and accurate overlay in dynamic scenes.

[0033] To address the aforementioned problems, this embodiment provides a robotic surgical navigation system based on interactive registration, such as... Figure 1 and Figure 2 As shown, the robotic surgical navigation system based on interactive registration includes: Robot main arm 101 is used to receive user input. Imaging device 201 is used to capture real-time images containing the target object; A pose sensor is disposed on the imaging device 201 to acquire the real-time pose of the imaging device 201. The control device 300 is communicatively connected to the robot's main hand 101, the imaging device 201, and the pose sensor. The display device 102 is communicatively connected to the control device 300; The control device 300 includes: The initial registration module 302 is used to adjust the pose of the virtual model in the three-dimensional medical image presented by the display device 102 in response to the user's operation on the robot master hand 101, so that the virtual model is registered with the target object in the three-dimensional medical image to determine the initial pose of the virtual model, wherein the virtual model can represent the internal anatomical structure of the target object. The virtual fusion module 303 is used to overlay the virtual model onto the real-time image captured by the imaging device 201 according to the initial pose, and generate and output the combined image to the display device 102. The interactive registration module 304 is used to adjust the pose of the virtual model in the combined image presented by the display device 102 in response to the operation of the user through the robot master hand 101, so that the virtual model is registered with the target object in the real-time screen. The binding module 305 is used to establish a binding relationship between the virtual model and the target object in the real-time screen when the interactive registration module completes the registration. The dynamic tracking module 306 is used to map the pose change of the imaging device 201 to the pose update of the virtual model through coordinate transformation based on the real-time pose data of the imaging device 201 obtained by the pose sensor after the binding relationship is established, so as to keep the virtual model spatially aligned with the target object in the real-time picture.

[0034] This robotic surgical navigation system based on interactive registration uses the robot's master hand 101 as an intuitive interactive interface, allowing users to naturally and flexibly manipulate the virtual model to complete initial registration and interactive adjustments, significantly reducing the operational difficulty of complex spatial registration. In the initial registration module 302, the virtual model's pose is adjusted within the 3D medical image presented on the display device 102, aligning the virtual model with the target object in the 3D medical image and determining the initial pose of the virtual model. This effectively reduces the initial deviation between the virtual model and the real target object when the virtual model is superimposed onto the real-time image during the virtual fusion stage. Based on this, only minor adjustments to the virtual model are needed during the interactive registration stage to quickly achieve precise alignment, greatly improving the alignment accuracy and reliability of augmented reality.

[0035] Furthermore, by establishing a binding relationship between the virtual model and the target object in the real-time image through the binding module 305, synchronous movement between the two can be achieved. Combined with the pose sensor acquiring real-time pose data of the imaging device 201 (such as adjusting the distance and pitch angle of the imaging device 201), the dynamic tracking module 306 maps the pose changes of the imaging device 201 to the pose update of the virtual model through coordinate transformation. This ensures that the virtual model remains spatially aligned with the target object in the real-time image, guaranteeing that even when the imaging device 201 moves, the virtual model remains continuously and accurately aligned with the real anatomical structure. Since the virtual model can represent the internal anatomical structure of the target object, through the aforementioned settings, this system not only enhances the stability and practicality of intraoperative image guidance but also provides surgeons with a "fluoroscopic" ability invisible to the naked eye, allowing them to simultaneously observe superficial tissues and deep key structures during surgery. This provides strong support for the precise execution of lesion resection or localization.

[0036] For ease of explanation, the following will be used as follows: Figure 1 The following is an example of a robotic surgical navigation system based on interactive registration.

[0037] Figure 1 This is a schematic diagram of the structure of the robotic surgical navigation system based on interactive registration provided in this application embodiment. The robotic surgical navigation system based on interactive registration includes a doctor's console 100, a patient surgical platform 200, and a control device 300 that enables communication between the two. The three components will be described in detail below.

[0038] (1) Doctor console 100 The doctor's console 100 serves as the main control unit, providing users with an immersive operating environment. The robot's main hand 101, acting as an interactive device, is a multi-degree-of-freedom (e.g., 6 or 7) operating handle. It serves as both an input device for controlling the auxiliary instruments and a core interface for intuitive interaction with the virtual model. Users can switch between two modes as needed.

[0039] Display device 102 is responsible for providing key visual feedback to the user. It mainly presents the following two types of images: one is real-time images, such as receiving and displaying a real-time video stream containing the target object captured by imaging device 201 (such as an endoscope); the other is composite images, such as a registered image of a virtual model and a three-dimensional medical image (such as CT, MRI), or an augmented reality (AR) image that is superimposed and fused with the aforementioned real-time images.

[0040] (2) Patient surgical platform 200 The patient surgical platform 200 is deployed in the surgical area and serves as the execution terminal for directly operating on the patient. It mainly consists of the following parts: The robotic arm system comprises one or more high-precision robotic arms, with specialized surgical instruments at the end of each arm for performing operations such as cutting, suturing, and grasping. The wrist of the instrument is designed with multiple degrees of freedom to simulate the flexible movement of a human hand in order to complete complex and delicate surgical actions.

[0041] The imaging device 201 is typically mounted on a separate robotic arm, the end of which extends into the patient's body to continuously capture high-quality real-time images or video streams containing the target object (such as diseased tissue, blood vessels, or nerves). The captured image data is transmitted in real-time to a control device 300 for processing and is ultimately presented to the user via a display device 102. For example, the imaging device 201 may be a stereoscopic endoscope.

[0042] The pose sensor is fixedly installed on the imaging device 201 to sense the six-free space pose (position and attitude) of the imaging device 201 in the global coordinate system in real time and with high precision. At the same time, the acquired real-time pose data is synchronously transmitted to the control device 300.

[0043] Optionally, the pose sensor can be an infrared camera, which continuously captures infrared light emitted or reflected by optical markers on the imaging device 201. By calculating the parallax of the same set of markers from multiple cameras, the system can calculate the three-dimensional spatial pose of the set of markers (i.e., the imaging device 201) in the observation coordinate system in real time.

[0044] (3) Control device 300 The control device 300 is connected in real time to the various components of the doctor's console 100 and the patient's surgical platform 200 via wired or wireless communication. It is responsible for the system's data processing, logical operations, and coordinated control.

[0045] In this embodiment, the control device has a preset operation mapping relationship, which is used to convert the physical motion parameters of the robot master hand 101 in at least one degree of freedom into pose adjustment commands for the virtual model; the physical motion parameters include at least one of displacement, rotation and opening degree.

[0046] To provide a flexible and intuitive operating experience, the control device 300 is configured with at least two independent operating modes to interpret and respond to the input to the robot master hand 101 in different ways: The pose adjustment mode is used for six-degree-of-freedom spatial registration between the virtual model and the real target object. In this mode, the system directly maps the movement or rotation of the robot's main hand 101 to the virtual model, based on a preset operation mapping relationship, so that the virtual model achieves the same translation or rotation direction as the robot's main hand 101 and the amplitude of the movement is in a preset proportion. For example, with an operation mapping relationship of 1:1, the translation or rotation of the robot's main hand 101 is mapped equally to the synchronous movement of the virtual model. Alternatively, the operation mapping relationship can be a preset value less than 1:1 to achieve sub-millimeter-level fine-tuning.

[0047] The scaling control mode is used to adjust the visual size of the virtual model on the display screen without changing its registered spatial position. In this mode, the system maps the opening and closing movements of the trigger components on the robot's main hand 101 used for simulated grasping to scaling commands, causing the virtual model to produce corresponding scaling movements. For example, if the preset operation mapping relationship is 1:0.01, when the operator slowly increases the opening size of the trigger component, the virtual model will synchronously and uniformly enlarge according to this ratio; conversely, when the opening is reduced, the virtual model will shrink accordingly, while its registered position in space remains unchanged.

[0048] In addition, the control device also includes a mode management unit, which dynamically switches between pose adjustment mode and scaling control mode when responding to external commands or when preset switching conditions are met. For example, the user can achieve the aforementioned switching through buttons on the robot's main hand 101, preset gestures, or voice commands.

[0049] Through the above mode, the system transforms the complex spatial control of the virtual model into an interactive process that conforms to human intuition, significantly reducing the difficulty of operation and improving operational efficiency and accuracy.

[0050] The aforementioned functions of the control device 300 are achieved by its internally cooperating software and hardware modules, which will be described one by one below.

[0051] Reference Figure 2 and Figure 3 The control device 300 also includes a virtual modeling module 301, which includes a model generation unit 311 and a feature marking unit 312. The model generation unit 311 is used to process the three-dimensional medical images acquired by the patient before surgery to generate a virtual model that is consistent with the patient's actual anatomical structure. The feature marking unit 312 is connected to the model generation unit 311 and is used to preset multiple first key points on the generated virtual model (surface or interior). These first key points provide clear corresponding references for subsequent registration operations with real-time images or spatial poses.

[0052] Virtual models, based on the patient's own multimodal imaging data, fundamentally ensure the matching accuracy between the virtual model and the individual's anatomy, avoiding individual differences introduced by general models. Their main data sources include: CT 3D reconstructed images, which clearly display bone contours, medullary cavity morphology, and the course of surrounding blood vessels and nerves, providing crucial bony structural references for orthopedic and neurosurgical procedures; MRI contrast-enhanced scans, particularly suitable for soft tissue imaging, accurately delineating the spatial relationship between tumor boundaries and adjacent key tissues (such as portal vein branches), providing essential navigation information for tumor resection surgery; and CTA angiography images, presenting the distribution of vascular systems such as cerebral and hepatic vessels in three dimensions, and clearly demonstrating the precise locational relationship between lesions (such as tumors and aneurysms) and the vascular system.

[0053] In this embodiment, multiple first key points are preset at the edges of the anatomical structure of the virtual model and are distributed at intervals along the edges.

[0054] Of course, depending on the surgical planning and registration requirements, the first key point can also be marked at a feature location with clear anatomical significance that is easily identifiable in both medical imaging and real-time video. Examples include: vascular bifurcation points (such as the portal vein bifurcation), bony landmarks, and specific tumor boundary points. This provides precise corresponding targets for subsequent initial and interactive registration, effectively improving registration efficiency and accuracy.

[0055] In practical applications, the construction of the virtual model and the marking of the first keypoints can be achieved through the following two processes: First, import the patient's 3D medical image data (e.g., DICOM format) into professional 3D modeling or medical image processing software (such as 3ds Max). Then, using image processing algorithms such as thresholding and region growing, automatically or semi-automatically extract the target anatomical structures and reconstruct a high-fidelity 3D mesh model, completing the geometric construction of the virtual model. Next, import the aforementioned geometric model file into a real-time rendering and development platform (such as Unity3D). On this platform, using scripts (such as C#) or visualization tools, mark points are created programmatically or interactively at specific anatomical feature locations on the surface of the virtual model. These mark points are defined as the first keypoints, and their 3D coordinate information is stored in the system for subsequent modules to call for matching with corresponding features in real-time images or spatial poses.

[0056] Furthermore, the feature marking unit 312 supports defining the first key point by receiving user instructions.

[0057] Specifically, during the virtual modeling phase, users (such as doctors) can directly select sites with significant anatomical features on the virtual model displayed on the display device 102 using interactive devices (such as a mouse, touchscreen, or robotic hand 101). Upon receiving this input command, the system records and marks the spatial coordinates of the selected location as a first key point. This interactive method allows doctors to define registration benchmarks based on their expertise and specific surgical strategies, making key point settings more targeted and thus improving the accuracy of subsequent registration.

[0058] Next, the composition and function of the interactive registration module 304 will be described in detail.

[0059] The interactive registration module 304, through the user's interactive markings on the image and the fine-tuning control of the robot's main hand 101, jointly completes the high-precision registration between the virtual model and the real-time image.

[0060] Specifically, refer to Figure 4 The interactive registration module 304 includes a key point marking unit 341 and an auxiliary registration unit 342. The key point marking unit 341 is used to mark a second key point on the target object (such as an anatomical structure) in the real-time image. The second key point corresponds one-to-one with the first key point predefined in the virtual model, forming a corresponding point pair for registration. The auxiliary registration unit 342 is used to drive the virtual model to adjust its pose in response to the second operation command of the user through the robot master hand 101, so that the first key point on the virtual model and the corresponding second key point marked in the real-time image visually meet the preset alignment conditions (such as spatial position coincidence), thereby completing the accurate registration.

[0061] It should be noted that the selection of the second keypoint must correspond one-to-one with the predefined first keypoint on the virtual model. For example, if a first keypoint is preset at the bifurcation of the hepatic artery in the virtual liver model, the user must mark the corresponding second keypoint at the same bifurcation location of the actual hepatic artery in the real-time view. For instance, the user can mark the point on the real-time view by clicking with a mouse or selecting with a touchscreen, and the system will automatically record the position of that point in the image pixel coordinate system.

[0062] The auxiliary registration unit 342 works in conjunction with the key point marking unit 341. When the user issues a second operation command through the robot's main hand 101, this unit will drive the virtual model to perform corresponding movements: by adjusting the pose of the virtual model, the projection position of the first key point on the virtual model on the display screen will tend to coincide with the pixel coordinates of the second key point marked in the real-time screen, thereby achieving precise spatial alignment.

[0063] Users can intuitively control the movement of the virtual model in multiple degrees of freedom by operating the robot's main hand 101. The core function of the auxiliary registration unit 342 is to map the physical movement of the robot's main hand 101 to the pose changes of the virtual model in real time and one-to-one, thereby enabling users to manually guide the virtual model to move.

[0064] Under user guidance, the system ultimately aligns all pairs of first keypoints (on the virtual model) and second keypoints (in the real-time video) on the screen or display with preset alignment conditions. In practical applications, the user directly observes and continuously operates the robot's main hand 101 until all corresponding keypoints are visually aligned and overlapped. This process is known as manual fine-tuning mode. Through this interactive registration process, the system, guided intuitively by the user, achieves precise spatial alignment between the virtual model based on 3D medical images and the real anatomical structure based on 2D real-time video.

[0065] In this embodiment, key point marking can be done manually by the user through interaction with the robot's main hand 101.

[0066] For example, users can activate manual marking mode via specific gestures or voice commands. The operation of the robot's main hand 101 then switches to moving a two-dimensional cursor (such as a crosshair) on the control screen; the user moves the robot's main hand 101, precisely positioning the cursor on a specific anatomical feature of the target object in the real-time image (such as a blood vessel bifurcation point). The user issues a marking command through a confirmation action (such as pressing a button). The system then executes: generating a prominent visual marker at the marked point in the image; recording the pixel coordinates of the point and pairing it with the corresponding first keypoint in the virtual model to form a registration reference pair. The user repeats this process until all predefined first keypoints have their corresponding second keypoints marked in the real-time image. This process combines the operational precision of the robot's main hand 101 with the user's professional judgment, providing a reliable data foundation for subsequent high-precision virtual-real registration.

[0067] Furthermore, to assist users in completing registration accurately and intuitively, the system provides the following real-time visual feedback when users adjust the virtual model's pose: Keypoint highlighting: The first keypoint on the virtual model and the second keypoint in the real-time image are rendered in striking 3D (e.g., a glowing sphere) and 2D (e.g., a highlighted ring) styles, respectively, facilitating clear tracking and comparison by the user. Real-time distance feedback: The system calculates and displays the projected distance of each pair of keypoints on the screen in real time. This information can be dynamically displayed using digital labels. Dynamic auxiliary connections: Dynamic connections are drawn between each pair of keypoints. The length of the connection visually indicates the magnitude of the error, while the direction suggests the direction the model should move. When all point pairs coincide, the connection disappears, providing a clear completion signal. By integrating the above visual guidance, the system transforms manual registration, which relies on subjective judgment, into a data-driven, visually assisted, and intuitive process, significantly reducing the user's cognitive load and improving the efficiency and accuracy of registration.

[0068] In addition to the manual marking method described above, keypoint marking can also be automatically completed by the feature extraction unit. This unit runs a deep learning-based image recognition algorithm to automatically identify the anatomical features of the target object in real-time footage and extract the second keypoint corresponding to the predefined first keypoint in the virtual model.

[0069] Specifically, a pre-trained convolutional neural network is used, with its model weights integrated into the system. This model is trained using a large number of labeled intraoperative real-time images and can accurately identify the location of specific anatomical features. The processing flow is as follows: after the real-time image stream is input, the model performs inference and automatically locates the significant anatomical features (such as vascular bifurcation points and contour vertices) of the target object (such as the liver) in the image. The model outputs the two-dimensional pixel coordinates of these feature points. The system performs anatomical matching between the identified feature points and the predefined first keypoints in the virtual model according to a pre-set keypoint mapping table, and establishes a stable data association for the successfully matched point pairs.

[0070] This automation method significantly reduces the user's workload, improves the efficiency and consistency of the registration process, and reduces human error introduced by manual marking deviations, providing a more reliable foundation for subsequent accurate registration.

[0071] After key point registration is completed, the system needs to solidify the spatial correspondence between the virtual model and the real anatomical structure. This function is implemented by the binding module 305. Specifically, the binding module 305 is used to: respond to a user's confirmation command, capture the registration status of the virtual model and the target object in the real-time image, and calculate the coordinate transformation matrix from the virtual model coordinate system to the imaging device coordinate system or the world coordinate system to establish a binding relationship. Based on the established binding relationship, regardless of how the imaging device 201 moves, the system can drive the virtual model to synchronously update its pose in real time, ensuring that it always maintains precise spatial alignment with the real anatomical structure (i.e., the target object) in the displayed image.

[0072] Specifically, users can trigger this operation by pressing a button on the robot's main arm 101, issuing a voice command, or clicking the "Confirm Registration" button on the interface. This operation marks the completion of the interactive registration phase.

[0073] Based on the above-mentioned registration and binding, the control device 300 also includes an interactive cropping module 307, which allows users to focus on displaying specific areas of the virtual model through interactive drawing in the augmented reality view, thereby purifying the surgical field of vision and highlighting key information.

[0074] Specifically, the user activates the cropping function by issuing a command (such as pressing a button) through the robot's main hand 101. Then, by manipulating the screen cursor driven by the main hand, the user directly draws a custom closed area (such as a circle, ellipse, or polygon) on the composite image. The lines of this closed area can be solid or dashed. The interactive cropping module 307 performs real-time graphic cropping of the superimposed virtual model based on the drawn closed area. That is, the system converts the two-dimensional screen area into a three-dimensional cropping space, applies real-time rendering technology to hide all virtual model parts outside this area, and simultaneously, the display device 102 only clearly displays the model parts located within the area, maintaining correct superposition with the real-time background.

[0075] For example, surgical navigation is performed in the anatomical region containing the liver 10. Figure 6 As shown, a virtual 3D model of the liver 10 has been accurately overlaid and registered onto a real-time video image captured by an endoscope. The user manipulates the robot's main hand to draw a closed region on the composite image, indicated by an elliptical dashed line in the figure. The virtual model portion outside the boundary of the elliptical dashed line is hidden, with only the portion shown as described in the figure. Figure 6The real-time liver view portion 11 is shown, while the virtual model portion 12 within the elliptical dashed boundary retains its internal structure and is continuously displayed. Therefore, users can observe the internal vascular structures of the liver 10 (such as hepatic veins and portal vein branches) or the relative position of lesions layer by layer, achieving more precise anatomical navigation. In minimally invasive surgery, this interactive clipping module 307 solves the problem of visual confusion caused by excessive augmented reality information, achieving focused augmented reality display. For example, doctors can select only the tumor and surrounding vascular model, bringing the following advantages: eliminating the occlusion of the real surgical field by irrelevant virtual information, significantly reducing visual interference; allowing doctors to focus their attention on the core anatomical area, improving the efficiency and accuracy of surgical judgment; and allowing users to define, modify, or cancel the clipping area at any time to adapt to the visualization needs of different surgical stages.

[0076] Reference Figure 5 This embodiment also provides a method for a robotic surgical navigation system based on interactive registration, including: S100, in response to the user's operation on the robot's main hand 101, the virtual model is adjusted in pose in the three-dimensional medical image presented by the display device 102, so that the virtual model is registered with the target object in the three-dimensional medical image, so as to determine the initial pose of the virtual model, wherein the virtual model can represent the internal anatomical structure of the target object.

[0077] The user operates the robot's main hand 101 to adjust the pose of the virtual model in the display device 102, making it visually perfectly aligned (registered) with the target object in the 3D medical image. This step determines the initial pose T0 of the virtual model in the system's world coordinate system, providing an accurate spatial starting point for subsequent virtual-real fusion.

[0078] S200: Based on the initial pose, the virtual model is superimposed onto the real-time image captured by the imaging device 201, and the combined image is generated and output to the display device 102.

[0079] Based on the initial pose T0, the system overlays the virtual model onto the real-time image background captured by the imaging device 201 (such as an endoscope) through graphics rendering, generating and outputting a preliminary virtual-real fusion combined image to the display device 102.

[0080] S300, in response to the user's operation through the robot master arm 101, adjusts the pose of the virtual model in the combined image presented by the display device 102, so that the virtual model is registered with the target object in the real-time screen.

[0081] Users can fine-tune the virtual model in the combined image by operating the robot's main hand 101, so that it can be highly registered with the real target anatomical structure in the real-time image on the two-dimensional image plane.

[0082] S400: When the interactive registration module 304 completes the registration, a binding relationship is established between the virtual model and the target object in the real-time screen.

[0083] When the interactive registration module 304 confirms that the virtual model and the target object in the real-time image are accurately registered, the binding module 305 automatically captures the stable spatial relationship between the two and calculates a fixed coordinate transformation matrix T from the virtual model coordinate system to the coordinate system of the real target object (such as the world coordinate system or the imaging coordinate system). map The matrix data is stored as a spatial binding basis between the two, thereby ensuring that the virtual model can move synchronously with the real target in real time and accurately based on this relationship during subsequent motion or changes in viewing perspective.

[0084] S500 After the binding relationship is established, based on the real-time pose data of the imaging device 201 obtained by the pose sensor, the pose change of the imaging device 201 is mapped to the pose update of the virtual model through coordinate transformation, so that the virtual model and the target object in the real-time picture are kept spatially aligned.

[0085] During the surgery, the system enters a dynamic tracking loop: continuously acquiring the real-time pose of the imaging device 201 measured by the pose sensor. s Based on the aforementioned spatial binding matrix T map And the system's pre-calibrated coordinate transformation chain, calculates in real time the correct pose T that the virtual model should present under the current imaging viewpoint. new According to T new The spatial transformation state of the virtual model is updated in real time, and the rendering engine is driven to generate a new combined image and output it to the display device 102.

[0086] With the aforementioned settings, the system can dynamically maintain precise spatial alignment between the virtual model and the real anatomical structure when the imaging device 201 moves or rotates at will, thereby providing doctors with a positionally stable and visually coherent augmented reality navigation screen on the display device 201, effectively supporting precise surgical operations.

[0087] Furthermore, the virtual model is registered with the target object in the real-time image, which includes the following key steps: The control device 300 reads multiple preset first key points on the virtual model. These points are marked preoperatively on anatomically significant feature locations (such as vascular bifurcation points and bony landmarks) based on 3D medical images, providing a reliable reference benchmark for registration. Second key points, corresponding one-to-one with the first key points, are marked on the target object in the real-time view. The user adjusts the pose of the virtual model by operating the robot's main hand 101, aiming to ensure that the first key points on the virtual model and the second key points in the real-time view meet preset alignment conditions on the screen. Registration is considered complete when the average error of all key point pairs is below a preset threshold, or when they visually overlap sufficiently. The system can provide visual or tactile feedback to assist in the judgment.

[0088] Next, the system allows users to focus on specific areas of the virtual model in the augmented reality view through interactive drawing.

[0089] The user activates the cropping mode by issuing a command (such as pressing a function key) through the robot's main hand 101. In this mode, the main hand controls a drawing cursor. The user moves the cursor over the composite image to draw a trajectory. Once the trajectory forms a closed shape (such as a circle or polygon) and is confirmed, a cropping area is defined. The control device 300 performs real-time graphic cropping on the virtual model based on this closed area.

[0090] This interactive clipping mechanism enables visual focus in augmented reality displays, allowing surgeons to actively hide irrelevant virtual information and concentrate their vision on the core surgical area (such as the tumor and surrounding key structures). This effectively solves the visual occlusion and confusion problems that virtual models may cause, reduces cognitive load, improves the efficiency of surgical judgment and the accuracy of operation, and enhances the practical value of augmented reality navigation.

[0091] This embodiment provides a computer-readable storage medium (such as a USB flash drive, external hard drive, SSD, ROM, or RAM) on which a computer program is stored. When the program is loaded and executed by a processor (such as a CPU or GPU), it can implement all the steps of the aforementioned method.

[0092] In its implementation, the core software of the control device 300 is an application built on the Unity engine. The system achieves real-time data interaction with physical devices by integrating SDKs (such as the robot's main hand, imaging equipment, and pose sensors) or using standard communication protocols (such as serial port / USB). This includes acquiring the pose and operation commands of the robot's main hand 101, accessing the real-time video stream from the imaging device 201, and receiving tracking data from the pose sensors. All data is parsed, fused, and calculated within Unity's world coordinate system. The system's core functions (such as coordinate transformation, spatial binding, real-time rendering, and interaction logic) are efficiently implemented in the Unity environment using C# scripts and shader programs, ultimately driving precise fusion and dynamic tracking between the virtual model and the real-time image.

[0093] It should be understood that the embodiments of the present invention are not limited to the specific structures and control flows listed in the above embodiments. Those skilled in the art can make appropriate adjustments or substitutions to the implementation methods of each module, the specific parameters of the coordinate mapping relationship, the mechanical structure form, etc., without departing from the principles of the present invention, and such modifications and variations should also be considered to fall within the protection scope defined by the claims of the present invention.

[0094] In the description of this application, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used solely for the convenience of describing this application and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0095] Finally, it should be noted that the above embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The protection scope of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A robotic surgical navigation system based on interactive registration, characterized in that, include: The robot's main arm is used to receive user input. Imaging devices are used to capture real-time images containing target objects; A pose sensor is disposed on the imaging device to acquire the real-time pose of the imaging device; The control device is communicatively connected to the robot's main hand, the imaging device, and the pose sensor. The display device is communicatively connected to the control device; The control device includes: An initial registration module is used to adjust the pose of the virtual model in the three-dimensional medical image presented by the display device in response to the user's operation on the robot's main hand, so that the virtual model is registered with the target object in the three-dimensional medical image to determine the initial pose of the virtual model, wherein the virtual model can characterize the internal anatomical structure of the target object; The virtual fusion module is used to overlay the virtual model onto the real-time image captured by the imaging device according to the initial pose, and generate and output the combined image to the display device; An interactive registration module is used to adjust the pose of the virtual model in the combined image presented by the display device in response to the user's operation through the robot's main hand, so that the virtual model is registered with the target object in the real-time screen; The binding module is used to establish a binding relationship between the virtual model and the target object in the real-time screen when the interactive registration module completes the registration. The dynamic tracking module is used to map the pose change of the imaging device to the pose update of the virtual model through coordinate transformation based on the real-time pose data of the imaging device obtained by the pose sensor after the binding relationship is established, so as to keep the virtual model spatially aligned with the target object in the real-time image.

2. The robotic surgical navigation system based on interactive registration according to claim 1, characterized in that, The control device further includes a virtual modeling module, which includes: A model generation unit is used to process a three-dimensional medical image containing the target object to generate the virtual model; The feature marking unit is used to pre-define multiple first key points for registration on the virtual model.

3. The robotic surgical navigation system based on interactive registration according to claim 2, characterized in that, The first key point is located at the edge of the anatomical structure of the virtual model, and multiple first key points are distributed at intervals on the edge of the anatomical structure.

4. The robotic surgical navigation system based on interactive registration according to claim 2 or 3, characterized in that, The first key point is specified by receiving user input instructions.

5. The robotic surgical navigation system based on interactive registration according to claim 1, characterized in that, The interactive registration module includes: A key point marking unit is used to mark a second key point on the target object in the real-time image, and the second key point corresponds one-to-one with a predefined first key point in the virtual model; An auxiliary registration unit is used to adjust the pose of the virtual model in response to the user's operation through the robot's main hand, so that the first key point on the virtual model and the second key point marked in the real-time image meet preset alignment conditions.

6. The robotic surgical navigation system based on interactive registration according to claim 5, characterized in that, The interactive registration module also includes a visual feedback unit, which is used to highlight the first key point and the second key point when adjusting the pose of the virtual model, and / or update the distance information between them in real time.

7. The robotic surgical navigation system based on interactive registration according to claim 5, characterized in that, The key point marking unit is a manual marking unit, used to mark a second key point on the target object in the real-time image in response to the marking command input by the user through the robot's main hand.

8. The robotic surgical navigation system based on interactive registration according to claim 5, characterized in that, The key point marking unit is a feature extraction unit used to run an image recognition algorithm to automatically identify the anatomical features of the target object in the real-time image and extract and mark the second key point corresponding to the first key point predefined in the virtual model.

9. The robotic surgical navigation system based on interactive registration according to claim 1, characterized in that, The pose sensor is either an optical sensor or an electromagnetic tracker.

10. The robotic surgical navigation system based on interactive registration according to claim 1, characterized in that, The robot's main hand is an operating handle with multiple degrees of freedom; The control device has a preset operation mapping relationship, which is used to convert the physical motion parameters of the operating handle in at least one degree of freedom into pose adjustment commands for the virtual model. The physical motion parameters include at least one of displacement, rotation, and opening / closing degree.

11. The robotic surgical navigation system based on interactive registration according to claim 10, characterized in that, The control device is configured to provide at least two switchable operating modes, including: In the pose adjustment mode, in response to the movement or rotation of the operating handle, the operation mapping relationship is configured to cause the virtual model to perform a translation or rotation that is consistent with the movement direction of the operating handle and the movement amplitude is in a preset proportion. In the scaling control mode, in response to the opening and closing action of the trigger component on the operating handle used for simulating gripping, the virtual model produces a corresponding scaling motion.

12. The robotic surgical navigation system based on interactive registration according to claim 1, characterized in that, The control device also includes an interactive cropping module; The interactive cropping module is used to respond to the user's cropping command, generate a closed area drawn by the user on the combined image of the display device, and crop the virtual model in the combined image based on the closed area to display a portion of the model located within the closed area on the display device.

13. A method for a robotic surgical navigation system based on interactive registration, characterized in that, include: In response to the user's operation on the robot's main hand, the virtual model is pose-adjusted in the three-dimensional medical image presented on the display device, so that the virtual model is registered with the target object in the three-dimensional medical image to determine the initial pose of the virtual model, wherein the virtual model can characterize the internal anatomical structure of the target object; Based on the initial pose, the virtual model is superimposed onto the real-time image captured by the imaging device to generate and output the combined image to the display device; In response to the user's operation via the robot's main hand, the virtual model is pose-adjusted in the combined image presented on the display device, so that the virtual model is registered with the target object in the real-time image; When registering the virtual model with the target object in the real-time image, a binding relationship is established between the virtual model and the target object in the real-time image; After the binding relationship is established, based on the real-time pose data of the imaging device, the pose change of the imaging device is mapped to the pose update of the virtual model through coordinate transformation, so that the virtual model maintains spatial alignment with the target object in the real-time image.

14. The method according to claim 13, characterized in that, The step of adjusting the pose of the virtual model in the combined image to register the virtual model with the target object in the real-time image includes: Obtain multiple first key points preset on the virtual model; Mark a second key point on the target object in the real-time image, which corresponds one-to-one with the first key point; In response to the user's operation through the robot's main hand, the pose of the virtual model is adjusted so that the first key point on the virtual model and the second key point on the target object meet the preset alignment conditions.

15. The method according to claim 14, characterized in that, Also includes: In response to a user's cropping command, a closed region drawn by the user is generated on the composite image of the display device, and the virtual model in the composite image is cropped based on the closed region to display a portion of the model located within the closed region on the display device.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it performs the steps of the method as described in any one of claims 13-15 above.

Images