Registration method of exterior mirror system and exterior mirror system

Abstract

The invention provides a registration method of an exterior mirror system and the exterior mirror system, and the method comprises the steps: adjusting the imaging parameters of an image collection module and the pose of a mechanical arm, and enabling the image collection module at the tail end of the mechanical arm to cover feature points in an imaging depth-of-field range for registration; the positions of the feature points in a mechanical arm coordinate system are obtained according to imaging parameters of the image acquisition module and the pose of the mechanical arm; and obtaining the positions of the at least three feature points in the mechanical arm coordinate system by referring to the above steps for registration and registration. By adjusting the imaging parameters of the image acquisition module and the pose of the mechanical arm, the image acquisition module covers the feature points in the imaging depth-of-field range for registration, and at the moment, the feature points are the specific positions limited by the imaging depth-of-field range in front of the lens; the position of the target in the mechanical arm coordinate system can be determined by combining the posture of the mechanical arm, the registration process can be completed without adding extra hardware, and the product cost is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of medical devices, and in particular to a registration method for an exoscopic system and an exoscopic system. Background Art

[0002] The exoscopic system is an important medical auxiliary equipment. It is connected to an imaging device at the end of the robotic arm to perform magnified microscopic imaging of the target area. The existing exoscopic system has some areas that need improvement:

[0003] On the one hand, the existing exoscopic systems have a single function and cannot interact with other devices, or the interaction process is not intelligent enough. For example, some existing exoscopic systems only present the images collected by the exoscopic system on the screen. The user selects the area for fine focus by clicking on the image on the screen (similar to the point screen focus on a smartphone), or adjusts the magnification through multi-finger operation. This adjustment process only involves the parameter adjustment of the exoscopic lens. When the user needs to adjust the exoscopic system to a suitable viewing angle to image the surgical area (for example, to make room for the doctor to operate), the position of the robotic arm still needs to be manually adjusted.

[0004] On the other hand, if the exterior mirror system is linked with other equipment, it is necessary to obtain the conversion relationship between the exterior mirror robot arm space and other spaces (that is, a registration process needs to be performed). The existing registration method usually requires additional hardware, which increases product costs.

[0005] To at least partially address the above-mentioned drawbacks, the present invention provides a registration method for an exterior mirror system and an exterior mirror system. Summary of the Invention

[0006] The present invention provides a registration method for an exterior mirror system and an exterior mirror system, which are used to solve the defect in the prior art that the exterior mirror system is not intelligent enough and cannot be linked with other devices.

[0007] In a first aspect, the present invention provides a registration method for an exterior mirror system, characterized by comprising:

[0008] Adjusting the imaging parameters of the image acquisition module and the posture of the robotic arm so that the image acquisition module at the end of the robotic arm covers the feature points within the imaging depth of field range used for registration;

[0009] Obtaining the position of the feature point in the robotic arm coordinate system according to the imaging parameters of the image acquisition module and the posture of the robotic arm;

[0010] Refer to the above steps to obtain the positions of at least three feature points in the robot arm coordinate system for registration.

[0011] Furthermore, the width of the imaging depth of field range for registration is no more than 3 mm.

[0012] Optionally, adjusting the imaging parameters of the image acquisition module and the posture of the robotic arm so that the image acquisition module at the end of the robotic arm covers the feature points within the imaging depth of field range for registration includes:

[0013] Adjust the image acquisition module to a preset magnification, and adjust the position of the robotic arm so that the distance between the image acquisition module and the feature point is within a preset working distance range;

[0014] By adjusting the focal length of the image acquisition module, the registration imaging depth of field range of the image acquisition module covers the feature points.

[0015] Furthermore, obtaining the position of the feature point in the robotic arm coordinate system according to the imaging parameters of the image acquisition module and the posture of the robotic arm includes:

[0016] Obtain the position of the feature point in the image acquisition module coordinate system according to the adjusted focal length;

[0017] Obtaining a conversion relationship between the image acquisition module coordinate system and the robotic arm coordinate system according to the adjusted robotic arm posture;

[0018] The position of the feature point in the robotic arm coordinate system is obtained according to the position of the feature point in the image acquisition module coordinate system and the conversion relationship between the image acquisition module coordinate system and the robotic arm coordinate system.

[0019] Optionally, adjusting the imaging parameters of the image acquisition module and the posture of the robotic arm so that the image acquisition module at the end of the robotic arm covers the feature points within the imaging depth of field range for registration includes:

[0020] Adjusting the image acquisition module to a preset magnification and a preset focal length;

[0021] By adjusting the posture of the robotic arm, the registration imaging depth of field range of the image acquisition module covers the feature points.

[0022] Furthermore, obtaining the position of the feature point in the robotic arm coordinate system according to the imaging parameters of the image acquisition module and the posture of the robotic arm includes:

[0023] Obtaining the position of the feature point in the image acquisition module coordinate system according to a preset focal length;

[0024] Obtaining a conversion relationship between the image acquisition module coordinate system and the robotic arm coordinate system according to the adjusted robotic arm posture;

[0025] The position of the feature point in the robotic arm coordinate system is obtained according to the position of the feature point in the image acquisition module coordinate system and the conversion relationship between the image acquisition module coordinate system and the robotic arm coordinate system.

[0026] Optionally, the feature points include patient feature points; after completing the registration, the method further includes: registering the feature points in the robotic arm coordinate system with the feature points in the patient model to obtain a registration relationship between the robotic arm coordinate system and the patient model coordinate system.

[0027] Optionally, the method further includes: adjusting the posture of the robotic arm according to the patient model viewing angle adjusted by the user, so that the image acquisition module can capture images of the patient at a corresponding viewing angle.

[0028] Optionally, the method further includes: adjusting the focus state of the image acquisition module according to the region of interest selected by the user in the patient model, and automatically focusing on the region of interest in the surgical space.

[0029] Optionally, the method further includes: displaying at least a portion of the structure in the patient model superimposed on the image captured by the image capture module based on the registration relationship and the position of the image capture module.

[0030] Optionally, the method further comprises:

[0031] The position of the feature point in the navigation coordinate system is collected by the navigation probe;

[0032] A registration relationship between the robotic arm coordinate system and the navigation coordinate system is obtained according to the position of the feature point in the robotic arm coordinate system and the position of the feature point in the navigation coordinate system.

[0033] Optionally, the method further comprises:

[0034] Track the spatial position of medical tools through the navigation module;

[0035] The posture of the robotic arm and / or the imaging parameters of the image acquisition module are adjusted according to the spatial posture of the medical tool, and the area indicated by the medical tool is automatically focused.

[0036] Optionally, the method further comprises:

[0037] Controlling the movement of the robotic arm to adjust the image acquisition module to the direction indicated by the medical tool;

[0038] The imaging depth of field of the image acquisition module is adjusted so as to cover the area indicated by the tip of the medical tool.

[0039] Optionally, the medical tool is provided with a command input module and a signal transmission module;

[0040] The user inputs a focus control instruction through the instruction input module to control the focus process of the image acquisition module;

[0041] The signal transmission unit is connected to the control unit and is used to transmit the control instructions input by the user to the processing module.

[0042] Optionally, after the user inputs a first instruction through the instruction input module, the current spatial position of the medical tool is obtained through the navigation module, and the image acquisition module is controlled to adjust to the direction currently indicated by the medical tool and / or adjusted so that the imaging depth of field range of the image acquisition module covers the area currently indicated by the medical tool.

[0043] Optionally, after the user inputs a second instruction through the instruction input module, the spatial position of the medical tool is obtained in real time through the navigation module, and the image acquisition module is controlled to follow the direction indicated by the medical tool in real time and / or adjust so that the imaging depth of field range of the image acquisition module covers the area indicated by the medical tool.

[0044] In a second aspect, the present invention further provides an exterior mirror system, comprising: a processing module, a robotic arm, an image acquisition module, and a display module;

[0045] The end of the robotic arm is connected to the image acquisition module, and is used to adjust the posture of the image acquisition module;

[0046] The processing module can execute any of the aforementioned registration methods for the exterior mirror system to complete the registration process.

[0047] In the present invention, the exterior mirror system can complete registration by itself and establish a conversion relationship between the real space and the three-dimensional model.

[0048] Optionally, the exterior mirror system of the present invention further includes a navigation module and a navigation probe.

[0049] The present invention provides a registration method for an exterior mirror system and an exterior mirror system, which have at least the following beneficial effects:

[0050] 1. By adjusting the imaging parameters of the image acquisition module and the posture of the robotic arm, the image acquisition module can cover the feature points within the imaging depth of field range for registration (that is, a clear image of the feature points is captured). At this time, the feature points are located at a specific position limited by the imaging depth of field range for registration in front of the lens. Combined with the posture of the robotic arm, the position of the target in the robotic arm coordinate system can be determined. This method can collect spatial points for registration without adding additional hardware, establish a conversion relationship between real space and three-dimensional models, and reduce product costs.

[0051] 2. In some implementations, the imaging parameters of the image acquisition module are fixed, and the feature points are “collected” by adjusting the posture of the robotic arm.

[0052] 3. In some implementations, the distance between the image acquisition module and the feature points is adjusted to within a preset working distance range, so that it has the basis for forming the "imaging depth of field range for registration", and the imaging parameters are adjusted to cover the feature points with the imaging depth of field range for registration, thereby avoiding the defect of the high difficulty of "capturing feature points by adjusting the position of the robotic arm".

[0053] 4. In some implementations, a conversion relationship is established between the robotic arm coordinate system and the patient model coordinate system. The user can flexibly and conveniently adjust the viewing angle and focus position of the exoscope by operating the patient model, without being limited to the local field of view corresponding to the image captured by the image acquisition module.

[0054] 5. In some implementations, a conversion relationship between the robotic arm coordinate system and the navigation coordinate system is established, and the navigation probe in the surgical navigation scene is "borrowed" to guide the focus of the exoscope and adjust the position of the robotic arm. The operation is simple and convenient, and efficient focusing can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0056] Figure 1 This is one of the structural schematic diagrams of an exterior mirror system provided by the present invention;

[0057] Figure 2 It is a flow chart of a registration method for an exterior mirror system provided by the present invention;

[0058] Figure 3 This is the second structural schematic diagram of an exterior mirror system provided by the present invention. DETAILED DESCRIPTION

[0059] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0060] The following combination Figure 1-Figure 3The present invention describes a registration method for an exterior mirror system and an exterior mirror system. In order to facilitate understanding of the present invention, first combine Figure 1 An exterior mirror system of the present invention is described as follows. Figure 1 As shown, the exterior mirror system includes: a processing module 100 , a robotic arm 200 , an image acquisition module 300 and a display module 400 .

[0061] The base of the robotic arm 200 is set at a desired position, for example, on a hospital bed, on the ground, or on an operating trolley. The operating trolley can be moved to a desired position and fixed. The distal end of the robotic arm 200 is a movable end, on which an image acquisition module 300 is provided. The image acquisition module 300 may include one imaging unit or two or more imaging units. The robotic arm 200 can adjust the position and posture of the image acquisition module 300. The image acquisition module 300 can acquire a high-definition magnified image of the target area and present it to medical staff through the display module 400.

[0062] Figure 2 This is a flow chart of a registration method for an exterior mirror system provided by the present invention, such as Figure 2 As shown, the method includes:

[0063] S1. Adjust the imaging parameters of the image acquisition module and the position of the robotic arm so that the image acquisition module at the end of the robotic arm covers the feature points within the imaging depth range used for registration;

[0064] S2. Obtain the position of the feature point in the robotic arm coordinate system according to the imaging parameters of the image acquisition module and the posture of the robotic arm;

[0065] S3. Obtain the positions of at least three feature points in the robot arm coordinate system according to the above steps for registration.

[0066] Specifically, the registration imaging depth of field range in the present invention refers to a narrow imaging depth of field range used during the process of "establishing the transformation relationship (i.e., the registration relationship) between the robot arm coordinate system and other coordinate systems." This "registration imaging depth of field range" can be achieved by adjusting the imaging parameters of the image acquisition module, such as increasing the magnification, reducing the distance between the image acquisition module and the imaging target (i.e., the working distance), or increasing the focal length.

[0067] Step S1 adjusts the imaging parameters of the image acquisition module and the posture of the robotic arm so that the imaging depth of field range for registration of the image acquisition module covers the feature point currently to be acquired. When the image acquisition module acquires a clear image of the feature point, it means that the feature point is located within the "imaging depth of field range for registration", that is, the feature point is located in a narrow specific area defined by the imaging depth of field range for registration in front of the lens, and its position is determinable. Step S2 determines the position of the feature point relative to the image acquisition module based on the imaging parameters, and then calculates the conversion relationship between the image acquisition module coordinate system and the robotic arm base coordinate system in combination with the posture of the robotic arm. On this basis, the position of the feature point in the robotic arm coordinate system can be determined.

[0068] Refer to the above process to collect the positions of at least three feature points in the robotic arm coordinate system, and combine the positions of each feature point in other coordinate systems (such as the patient model coordinate system). By aligning multiple groups of feature points, the transformation relationship between the robotic arm coordinate system and other coordinate systems is calculated to complete the registration process.

[0069] This embodiment adjusts the imaging parameters of the image acquisition module and the posture of the robotic arm so that the image acquisition module covers the feature points within the imaging depth of field range used for registration. At this time, the feature points are located at specific positions limited by the imaging depth of field range in front of the lens. Combined with the posture of the robotic arm, the position of the target in the robotic arm coordinate system can be determined. This method can complete the registration process without adding additional hardware, thereby reducing product costs.

[0070] Based on the previous embodiment, in some embodiments, the width of the imaging depth of field range for registration is no greater than 3 mm.

[0071] Specifically, the narrower the depth of field used for registration, the higher the accuracy of the position of the captured feature points. In this embodiment, the width of the depth of field used for registration is no greater than 3 mm, so that the accuracy of the captured feature points can meet the registration accuracy requirements of the exterior mirror system scene.

[0072] Based on any embodiment, in some embodiments, S1 includes:

[0073] S111, adjusting the image acquisition module to a preset magnification, and adjusting the posture of the robotic arm so that the distance between the image acquisition module and the feature point is within a preset working distance range;

[0074] S112: Adjust the focal length of the image acquisition module so that the registration imaging depth of field range of the image acquisition module covers the feature points.

[0075] Specifically, in step S111, the image acquisition module is adjusted to a preset magnification. For example, the image acquisition module of the exterior mirror system supports a magnification of 1 to 30 times. During the registration process, the magnification is adjusted to a preset 30 times. For example, in another example, the magnification is adjusted to a preset 25 times. In step S11, the distance between the image acquisition module and the feature point is adjusted to within a preset working distance range. For example, the image acquisition module itself supports a working distance range of 200 to 600 mm. The range of 200 to 250 mm is used as the preset working distance range. During the registration process, the posture of the image acquisition module is adjusted so that the distance between it and the feature point is within the range of 200 to 250 mm. Thereafter, step S112 adjusts the focal length of the image acquisition module so that the feature point appears clearly in the image captured by the image acquisition module.

[0076] It should be understood that the steps of "adjusting the magnification" and "adjusting the distance between the image acquisition module and the feature point" in step S111 both provide the basis for the image acquisition module to "form the imaging depth of field range for registration," and their execution is not necessarily sequential. When a clear image of the feature point is acquired in step S112, the feature point is located within the image acquisition module's current "imaging depth of field range for registration," and its position can be accurately determined.

[0077] In some implementations, the robotic arm provides a follow-up control mode, and the user can drag the robotic arm to adjust the robotic arm's posture; in other implementations, the robotic arm provides a command control mode, and the user adjusts the robotic arm's posture by inputting commands through operating software (for example, the user adjusts / fine-tunes the robotic arm's posture through up, down, left, right, and angle adjustment commands provided by the software interface); in other embodiments, the user adjusts the robotic arm's posture by inputting voice commands; in other embodiments, the user adjusts the robotic arm's posture by inputting gesture commands; in other implementations, the user adjusts the robotic arm's posture through dedicated control hardware (such as a control handle).

[0078] In this embodiment, the imaging depth of field is limited by adjusting the magnification of the image acquisition module and the distance between the image acquisition module and the feature point, providing a basis for forming the "registration imaging depth of field range." The imaging depth of field is then adjusted by focusing until a clear image of the feature point is captured. This creates the registration imaging depth of field range that covers the feature point. This simple and efficient adjustment process allows the registration imaging depth of field to more easily cover the feature point, reducing operational complexity for medical personnel and improving the efficiency of feature point acquisition.

[0079] Based on the previous embodiment, in some embodiments, S2 includes:

[0080] S211, obtaining the position of the feature point in the image acquisition module coordinate system according to the adjusted focal length;

[0081] S212, obtaining a conversion relationship between the image acquisition module coordinate system and the robotic arm coordinate system according to the adjusted robotic arm posture;

[0082] S213 , obtaining the position of the feature point in the robotic arm coordinate system according to the position of the feature point in the image acquisition module coordinate system and the conversion relationship between the image acquisition module coordinate system and the robotic arm coordinate system.

[0083] Specifically, the focal length when the "registration imaging depth of field covers the feature point" is denoted as F, and the distance of the feature point in front of the image acquisition module lens is denoted as D. In some implementations, the relationship between F and D can be pre-calibrated. The magnification factor does not affect the calculation of the feature point position, but only affects the width of the registration depth of field, that is, the accuracy of the image acquisition point. Step S21 determines the position of the feature point in the image acquisition module coordinate system based on the actual focal length during the registration process and the calibrated relationship.

[0084] The "conversion relationship between the image acquisition module coordinate system and the robotic arm coordinate system" is related to the robotic arm posture. The robotic arm posture can be described by a set of joint angles. Step S212 can determine the "conversion relationship between the image acquisition module coordinate system and the robotic arm coordinate system" based on the adjusted robotic arm posture combined with forward kinematics. On this basis, step S213 can convert the feature points in the image acquisition module coordinate system to the robotic arm coordinate system.

[0085] Based on any embodiment, in some embodiments, S1 includes:

[0086] S121, adjusting the image acquisition module to a preset magnification and a preset focal length;

[0087] S122. Adjust the posture of the robotic arm so that the registration imaging depth of field of the image acquisition module covers the feature points.

[0088] Specifically, after adjusting the image acquisition module to a preset magnification and a preset focal length, the depth of field range for registration imaging is defined, capable of clearly imaging a specific position in front of its lens. Step S122 adjusts the robotic arm so that the depth of field range for registration imaging of the image acquisition module covers the feature point. The position of the feature point in the image acquisition module coordinate system is the position of the imaging depth of field range in the image acquisition module coordinate system. It is understandable that thereafter, it is not necessary to execute S121, and more feature points can be directly collected by referring to S122. The relative positional relationship between the feature point and the image acquisition module in each acquisition posture is the same, which can reduce the amount of calculation, or in other words, reduce the amount of calibration, and eliminate the need to determine multiple sets of relationship data between the "focal length F" and the distance D of the feature point in front of the image acquisition module lens.

[0089] In some implementations, the robotic arm provides a follow-up control mode, and the user can drag the robotic arm to adjust the robotic arm's posture; in other implementations, the robotic arm provides a command control mode, and the user adjusts the robotic arm's posture by inputting commands through operating software (for example, the user adjusts / fine-tunes the robotic arm's posture through up, down, left, right, and angle adjustment commands provided by the software interface); in other embodiments, the user adjusts the robotic arm's posture by inputting voice commands; in other embodiments, the user adjusts the robotic arm's posture by inputting gesture commands; in other implementations, the user adjusts the robotic arm's posture through dedicated control hardware (such as a control handle).

[0090] In this embodiment, the image acquisition module is adjusted to a preset magnification and a preset focal length to form a registration imaging depth of field range, and then the robotic arm is adjusted so that the registration imaging depth of field range of the image acquisition module covers the feature points, thereby reducing the amount of calibration data and the amount of data calculation.

[0091] Based on the previous embodiment, in some embodiments, S2 includes:

[0092] S221, obtaining the position of the feature point in the image acquisition module coordinate system according to the preset focal length;

[0093] S222. Obtaining a conversion relationship between the image acquisition module coordinate system and the robotic arm coordinate system according to the adjusted robotic arm posture;

[0094] S223 . Obtain the position of the feature point in the robotic arm coordinate system according to the position of the feature point in the image acquisition module coordinate system and the conversion relationship between the image acquisition module coordinate system and the robotic arm coordinate system.

[0095] Specifically, the relationship between the focal length F and the distance D between the feature point and the image acquisition module lens in the state of "the registration imaging depth of field range covering the feature point" can be determined through calibration. It is understood that at least one set of the above data can be calibrated to support the registration method of this embodiment.

[0096] The "conversion relationship between the image acquisition module coordinate system and the robotic arm coordinate system" is related to the robotic arm posture. The robotic arm posture can be described by a set of joint angles. Step S222 can determine the "conversion relationship between the image acquisition module coordinate system and the robotic arm coordinate system" based on the adjusted robotic arm posture combined with forward kinematics. On this basis, step S223 can convert the feature points in the image acquisition module coordinate system to the robotic arm coordinate system.

[0097] Based on any embodiment, in some embodiments, the feature points include patient feature points;

[0098] After S3, the method further includes: registering the feature points in the robotic arm coordinate system with the feature points in the patient model to obtain a registration relationship between the robotic arm coordinate system and the patient model coordinate system.

[0099] Specifically, the patient's feature points, such as the patient's eye corners, nose tip, bone markers on the patient's body surface, and developable markers attached to the patient's body surface, are collected in the robotic arm coordinate system and the feature points in the patient model coordinate system, and the conversion relationship between the robotic arm coordinate system and the patient model coordinate system (i.e., the registration relationship) is obtained.

[0100] This embodiment establishes a connection between the robotic arm coordinate system of the exoscopic system and the patient model coordinate system. The registration process is simple and efficient, and no additional hardware is required.

[0101] Based on the previous embodiment, in some embodiments, the method further includes:

[0102] According to the patient model viewing angle adjusted by the user, the position of the robotic arm is adjusted so that the image acquisition module can capture images of the patient at the corresponding viewing angle.

[0103] Specifically, the patient model can be displayed by the display module (eg Figure 1 The user can observe the patient model's viewing angle displayed on the display module and adjust the patient model to the desired viewing angle by inputting commands. Since the registration has been completed, the "observation angle" in the patient model coordinate system can be mapped to the robotic arm coordinate system according to the registration relationship. On this basis, the robotic arm can adjust the image acquisition module to the corresponding viewing angle to capture images of the patient. As an example, the nurse rotates the patient model displayed on the display screen to the viewing angle of the "patient's left face" by using the mouse, and then clicks the "exoscopic viewing angle synchronization" button. The robotic arm can then adjust its posture so that the image acquisition module captures images from the viewing angle of the "patient's left face" in the operating space.

[0104] It should be noted that the "patient model" and the "patient image data collected by the image acquisition module" are different contents, which can be displayed through different display modules. Of course, they can also be displayed in split screens in the same display module, or one of them can be used as the main display content and the other can be displayed through a "small window".

[0105] In this embodiment, the viewing angle of the exterior mirror can be adjusted conveniently and efficiently without the medical staff having to drag the robotic arm.

[0106] Based on the foregoing embodiment, in some embodiments, the method further includes:

[0107] According to the region of interest selected by the user in the patient model, the focus state of the image acquisition module is adjusted to automatically focus on the region of interest in the surgical space.

[0108] Specifically, the patient model can be displayed by the display module (eg Figure 1 Using the display on the surgical trolley, the user can observe the patient model's perspective displayed on the display module and select a region of interest (ROI). Since the ROI in the patient model's coordinate system has already been registered, it can be mapped to the robotic arm's coordinate system based on the registration relationship. Based on this, the exoscopic system focuses the image acquisition module on the corresponding ROI. For example, a nurse can use the mouse to select the "back of the head" of the patient model as the ROI. Then, by clicking the "Focus on ROI" button, the exoscopic system adjusts the image acquisition module to focus on the patient's "back of the head" in the surgical space.

[0109] In this embodiment, the user can flexibly adjust the focus of the exterior mirror to the area of ​​interest, making the operation more convenient. Of course, this embodiment can also be combined with the previous embodiment to simultaneously adjust the position (viewing angle) and focus position of the image acquisition module.

[0110] Based on the foregoing embodiment, in some embodiments, the method further includes:

[0111] According to the registration relationship and the position of the image acquisition module, at least part of the structure in the patient model is superimposed and displayed with the image acquired by the image acquisition module.

[0112] Specifically, the patient model contains the patient's tissue structure information, such as invisible organ structures, tumor blood vessels, etc. This embodiment can superimpose the invisible structure on the image captured by the image acquisition module, making it easier for doctors to understand the tissue structure near the area of ​​interest and improving safety. More specifically: first, the structure to be displayed is mapped to the robot arm coordinate system based on the registration relationship, and then the structure to be displayed is converted to the image acquisition module coordinate system based on the conversion relationship between the robot arm coordinate system and the image acquisition module coordinate system. According to the structure to be displayed in the image acquisition module coordinate system, the image to be displayed under the corresponding viewing angle is generated and superimposed with the image captured by the image acquisition module.

[0113] This embodiment overlays and displays the structures in the patient model and the images acquired by the image acquisition module according to the registration relationship, thereby providing more intuitive information guidance to the doctor.

[0114] Based on the foregoing embodiment, in some embodiments, the method further includes:

[0115] The position of the feature point in the navigation coordinate system is collected by the navigation probe;

[0116] According to the position of the feature point in the manipulator coordinate system and the position of the feature point in the navigation coordinate system, a registration relationship between the manipulator coordinate system and the navigation coordinate system is obtained.

[0117] Specifically, the navigation probe is tracked by the navigation module to determine the location of its tip. The navigation probe collects the locations of feature points in the navigation coordinate system and then aligns them with the locations of the previously collected feature points in the robotic arm coordinate system, resulting in a registration relationship between the robotic arm coordinate system and the navigation coordinate system. It is understood that at least three sets of feature points are required to calculate the registration relationship between the two coordinate systems.

[0118] Based on the previous embodiment, in some embodiments, the method further includes:

[0119] Track the spatial position of medical tools through the navigation module;

[0120] The posture of the robotic arm and / or the imaging parameters of the image acquisition module are adjusted according to the spatial posture of the medical tool, and the area indicated by the medical tool is automatically focused.

[0121] Specifically, the aforementioned medical tools can be modified surgical instruments, such as scalpels, forceps, suction devices, electrocoagulation needles, and other surgical instruments with added tracking markers. In this case, the medical tools can be used to perform surgery and their positions can also be tracked by the navigation module to guide the autonomous focusing process of the exoscopic system. Of course, the medical tools can also be dedicated probe tools that can guide the autonomous focusing process. Autonomous focusing can be achieved simply by adjusting the position of the robotic arm, for example, by adjusting the image acquisition module to the direction indicated by the medical tool and achieving focus by adjusting the distance between the image acquisition module and the position indicated by the medical tool; autonomous focusing can also be achieved simply by adjusting the imaging parameters of the image acquisition module; autonomous focusing can also be achieved by adjusting both the position of the robotic arm and the imaging parameters of the image acquisition module.

[0122] This embodiment can provide a focus target more efficiently and autonomously control the exoscopic system to focus on the indicated part without interrupting the doctor's operating rhythm.

[0123] Based on the previous embodiment, in one embodiment, the method further includes:

[0124] Control the movement of the robotic arm and adjust the image acquisition module to the direction indicated by the medical tool;

[0125] The imaging depth of field of the image acquisition module is adjusted so as to cover the area indicated by the tip of the medical tool.

[0126] Specifically, for example, if a medical tool has a rod-shaped body, the length of the rod-shaped body can be used as the direction of the medical tool's indication. This allows for convenient and intuitive direction indication during use, facilitating adjustment of the image acquisition module to the corresponding viewing angle. For another example, a direction at a certain spatial angle θ relative to the direction of the medical tool's body can be used as the direction of the medical tool's indication. In this case, the image acquisition module captures images along the indicated direction, and its viewing angle is not obstructed by the medical tool. For another example, if a medical tool has a curved body, the direction indicated by the curved tip can be used as the direction of the medical tool's indication. The curved tip can indicate an area within a narrow passage, facilitating the image acquisition module's focused imaging of locations deep within the narrow passage.

[0127] The aforementioned "area indicated by the tip of the medical instrument" refers to a region within a certain range of the tip, for example, a 5 mm diameter region centered on the tip of the medical instrument. During focusing, the image acquisition module's depth of field must encompass this region. Preferably, the image acquisition module's focus is adjusted so that the center of its depth of field falls near the tip of the medical instrument.

[0128] During the adjustment process, the image acquisition module can be adjusted to the direction indicated by the medical tool by controlling the movement of the robotic arm, and the distance between the image acquisition module and the area indicated by the tip of the medical tool can be adjusted so that the imaging depth of field range of the image acquisition module covers the area indicated by the tip of the medical tool.

[0129] It is understood that after adjusting the robotic arm to the direction indicated by the medical tool, the focal length of the image acquisition module can be adjusted so that the imaging depth of field of the image acquisition module covers the area indicated by the tip of the medical tool. Since the zoom lens has a certain zoom range, the distance between the image acquisition module and the area indicated by the tip of the medical tool after the position adjustment must meet the limitations of the zoom range. For example, in one example, the zoom range of the image acquisition module is 200-600 mm. The position of the image acquisition module is adjusted to a distance between 200 and 600 mm from the area indicated by the tip of the medical tool, and the focal length of the image acquisition module is adjusted so that the imaging depth of field of the image acquisition module covers the area indicated by the tip of the medical tool.

[0130] This embodiment uses medical tools to indicate the direction and uses a robotic arm to autonomously adjust the position of the image acquisition module so that it reaches a suitable image acquisition posture. The focus target is given by the position indicated by the medical tool, so that this focusing stage can focus directly according to the distance (the distance to the focus target can be calculated), without calculating the image clarity for focusing, thereby improving the focusing efficiency.

[0131] Based on the aforementioned embodiments, in some embodiments, the medical tool is provided with a command input module and a signal transmission module;

[0132] The user inputs the focus control command through the command input module to control the focus process of the image acquisition module;

[0133] The signal transmission unit is connected to the control unit and is used to transmit the control instructions input by the user to the processing module.

[0134] The command input module may be a button, a rotary knob, a dip switch, a micro-touch screen, or the like. The user may input commands through the command input module. For example, each time a button is pressed, autonomous focus is performed based on the current spatial position of the medical tool. After execution, the user must wait for the next command to be acquired. Only after the new command is acquired will autonomous focus be performed again based on the new spatial position of the medical tool. For another example, the rotary knob may have three positions. When the user rotates the knob to the first position, the autonomous focus function is turned off. When the user rotates the knob to the second position, a command is transmitted and autonomous focus is performed. When the user rotates the knob to the third position, autonomous focus continues based on the spatial position of the medical tool.

[0135] Based on the previous embodiment, in some embodiments, after the user inputs the first instruction through the instruction input module, the current spatial position of the medical tool is obtained through the navigation module, and the image acquisition module is controlled to adjust to the direction currently indicated by the medical tool and / or adjusted so that the imaging depth of field range of the image acquisition module covers the area currently indicated by the medical tool.

[0136] Specifically, this embodiment performs single focus based on a first user-input command. That is, each time a first user-input command is received, autonomous focus is performed based on the current position of the medical tool. The robot arm is controlled to move, the image acquisition module is adjusted to the direction indicated by the medical tool, and / or the focus state of the image acquisition module is adjusted so that the imaging depth of field covers the area indicated by the medical tool, thereby achieving clear imaging of the area indicated by the medical tool. After executing autonomous focus once, the system waits for the next first command to be received. Only after the new command is received will autonomous focus be performed again based on the new spatial position of the medical tool.

[0137] This embodiment performs single focusing according to the first instruction input by the user, thereby improving the stability of the image captured by the exterior mirror system and improving the observation effect.

[0138] Based on the aforementioned embodiments, in some embodiments, after the user inputs the second instruction through the instruction input module, the spatial posture of the medical tool is obtained in real time through the navigation module, and the image acquisition module is controlled to follow the direction indicated by the medical tool in real time and / or adjust so that the imaging depth of field range of the image acquisition module covers the area indicated by the medical tool.

[0139] Specifically, in this embodiment, real-time focusing is performed according to the second instruction input by the user. When the second instruction input by the user is received, the spatial posture of the medical tool is continuously acquired through the navigation module and / or the movement of the robotic arm is controlled to continuously adjust the image acquisition module to the direction most recently indicated by the medical tool, and the image acquisition module is controlled to continuously focus on the position indicated by the medical tool.

[0140] It can be understood that "real-time" in this embodiment does not mean that the image acquisition module strictly complies with the direction indicated by the medical tool or that the image acquisition module strictly focuses on the position indicated by the medical tool, but means that the exoscopic system attempts to focus autonomously at a certain time interval. In this application, "real-time" means attempting to focus autonomously at a time interval of no more than 2s, for example 2s, 1s, 0.1s, 0.02s.

[0141] This embodiment performs real-time focusing according to the second instruction input by the user, thereby further improving the usability of the exterior mirror system.

[0142] An exterior mirror system provided by the present invention is described below. The exterior mirror system described below and the registration method of the exterior mirror system described above can be referenced to each other.

[0143] Figure 1 Schematic diagram of the structure of an exterior mirror system provided by the present invention, such as Figure 1 As shown, the system includes: a processing module 100 , a robotic arm 200 , an image acquisition module 300 , and a display module 400 .

[0144] In the use state, one end of the robot arm 200 is fixed, for example, fixed to a position such as a bed, the ground, a wall, a ceiling, an operating table, etc. ( Figure 1 (The figure only illustrates the state in which the control center 100 is located inside the operating trolley.) The other end of the robotic arm 200 is connected to the image acquisition module 300. The robotic arm 200 has multiple degrees of freedom, such as 4, 5, or 6 degrees of freedom. This allows the position and posture of the image acquisition module 300 to be adjusted and fixed to a suitable viewing angle to facilitate the acquisition of images of the surgical area.

[0145] The image acquisition module 300 can be a monocular camera, a binocular camera, or a multi-camera, and can realize visible light imaging, infrared imaging, fluorescence imaging, or a combination of several imaging modes.

[0146] The processing module 100 completes the registration process according to any of the registration methods for the exterior mirror system described above. Specifically, the processing module 100 adopts a general computer architecture, which can load and run a program to implement the steps of any of the registration methods for the exterior mirror system described above.

[0147] It can be understood that the above-mentioned processing module 100, robotic arm 200, and image acquisition module 300 can be independently set in physical form, or can be flexibly combined. For example, the processing module 100 can be integrated into the base of the robotic arm 200, or for example, the processing module 100 and the image acquisition module 300 can be integrated together.

[0148] Furthermore, the exoscopic system may also include a display module 400 for displaying the images captured by the image acquisition module 300. The doctor can observe the clear, magnified image of the surgical area displayed by the module to obtain more information support for the surgery. The display module can be a display screen, a projector, a wearable display device, etc.

[0149] Reference Figure 3 Based on the previous embodiment, the exterior mirror system of this embodiment further includes a navigation module 500 and a navigation probe 600.

[0150] Specifically, the navigation probe 600 can be tracked by the navigation module 500 to determine the location of its tip. The positions of feature points collected by the navigation probe 600 in the navigation coordinate system are then aligned with the positions of the previously collected feature points in the robotic arm coordinate system, resulting in a registration relationship between the robotic arm coordinate system and the navigation coordinate system. It will be appreciated that at least three sets of feature points are required to calculate the registration relationship between the two coordinate systems.

[0151] Afterwards, the navigation probe 600 can be used to indicate the area of ​​interest and guide the exoscopic system to focus on the corresponding position, or tracking marks can be set on surgical instruments, such as scalpels, tweezers, suction devices, electrocoagulation needles and other instruments. The navigation module determines the position indicated by the surgical instrument through the tracking mark, and then converts the indicated position to the robotic arm coordinate system based on the registration relationship between the robotic arm coordinate system and the navigation coordinate system. On this basis, the posture of the robotic arm and / or the imaging parameters of the image acquisition module are adjusted to automatically focus on the area indicated by the surgical instrument.

[0152] This embodiment can provide a focus target more efficiently and autonomously control the exoscopic system to focus on the indicated part without interrupting the doctor's operating rhythm.

[0153] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0154] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0155] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A registration method for an exterior mirror system, characterized in that: include: Adjusting the imaging parameters of the image acquisition module and the posture of the robotic arm so that the image acquisition module at the end of the robotic arm covers the feature points within the imaging depth of field range used for registration; Obtaining the position of the feature point in the robotic arm coordinate system according to the imaging parameters of the image acquisition module and the posture of the robotic arm; Refer to the above steps to obtain the positions of at least three feature points in the robot arm coordinate system for registration.

2. The registration method of the exterior mirror system according to claim 1, characterized in that: The width of the imaging depth of field range used for registration is no more than 3 mm.

3. The registration method of the exterior mirror system according to claim 1, characterized in that: The step of adjusting the imaging parameters of the image acquisition module and the posture of the robotic arm so that the image acquisition module at the end of the robotic arm covers the feature points within the imaging depth of field range for registration includes: Adjust the image acquisition module to a preset magnification, and adjust the position of the robotic arm so that the distance between the image acquisition module and the feature point is within a preset working distance range; By adjusting the focal length of the image acquisition module, the registration imaging depth of field range of the image acquisition module covers the feature points.

4. The registration method of the exterior mirror system according to claim 3, characterized in that: The obtaining of the position of the feature point in the robotic arm coordinate system according to the imaging parameters of the image acquisition module and the posture of the robotic arm includes: Obtain the position of the feature point in the image acquisition module coordinate system according to the adjusted focal length; Obtaining a conversion relationship between the image acquisition module coordinate system and the robotic arm coordinate system according to the adjusted robotic arm posture; The position of the feature point in the robotic arm coordinate system is obtained according to the position of the feature point in the image acquisition module coordinate system and the conversion relationship between the image acquisition module coordinate system and the robotic arm coordinate system.

5. The registration method of the exterior mirror system according to claim 1, characterized in that: The step of adjusting the imaging parameters of the image acquisition module and the posture of the robotic arm so that the image acquisition module at the end of the robotic arm covers the feature points within the imaging depth of field range for registration includes: Adjusting the image acquisition module to a preset magnification and a preset focal length; By adjusting the posture of the robotic arm, the registration imaging depth of field range of the image acquisition module covers the feature points.

6. The registration method of the exterior mirror system according to claim 5, characterized in that: The obtaining of the position of the feature point in the robotic arm coordinate system according to the imaging parameters of the image acquisition module and the posture of the robotic arm includes: Obtaining the position of the feature point in the image acquisition module coordinate system according to a preset focal length; Obtaining a conversion relationship between the image acquisition module coordinate system and the robotic arm coordinate system according to the adjusted robotic arm posture; The position of the feature point in the robotic arm coordinate system is obtained according to the position of the feature point in the image acquisition module coordinate system and the conversion relationship between the image acquisition module coordinate system and the robotic arm coordinate system.

7. The registration method of the exterior mirror system according to claim 1, characterized in that: The characteristic points include patient characteristic points; After completing the registration, the method further includes: registering the feature points in the robotic arm coordinate system with the feature points in the patient model to obtain a registration relationship between the robotic arm coordinate system and the patient model coordinate system.

8. The registration method of the exterior mirror system according to claim 7, characterized in that: The method also includes: According to the patient model viewing angle adjusted by the user, the posture of the robotic arm is adjusted so that the image acquisition module can capture images of the patient at the corresponding viewing angle.

9. The registration method of the exterior mirror system according to claim 7, characterized in that: The method also includes: According to the region of interest selected by the user in the patient model, the focus state of the image acquisition module is adjusted to automatically focus on the region of interest in the surgical space.

10. The registration method of the exterior mirror system according to claim 7, characterized in that: The method also includes: According to the registration relationship and the position of the image acquisition module, at least a portion of the structure in the patient model is superimposed and displayed with the image acquired by the image acquisition module.

11. The registration method of the exterior mirror system according to claim 1, characterized in that: The method also includes: The position of the feature point in the navigation coordinate system is collected by the navigation probe; A registration relationship between the robotic arm coordinate system and the navigation coordinate system is obtained according to the position of the feature point in the robotic arm coordinate system and the position of the feature point in the navigation coordinate system.

12. The registration method of the exterior mirror system according to claim 11, characterized in that: The method also includes: Track the spatial position of medical tools through the navigation module; The posture of the robotic arm and / or the imaging parameters of the image acquisition module are adjusted according to the spatial posture of the medical tool, and the area indicated by the medical tool is automatically focused.

13. The exterior mirror system according to claim 12, characterized in that: The method also includes: Controlling the movement of the robotic arm to adjust the image acquisition module to the direction indicated by the medical tool; The imaging depth of field of the image acquisition module is adjusted so as to cover the area indicated by the tip of the medical tool.

14. The registration method of the exterior mirror system according to claim 11, characterized in that: The medical tool is provided with a command input module and a signal transmission module; The user inputs a focus control instruction through the instruction input module to control the focus process of the image acquisition module; The signal transmission unit is connected to the control unit and is used to transmit the control instructions input by the user to the processing module.

15. An exterior mirror system, characterized in that: include: Processing module, robotic arm, image acquisition module, display module; The end of the robotic arm is connected to the image acquisition module, and is used to adjust the posture of the image acquisition module; The processing module executes the registration method of the exterior mirror system according to any one of claims 1 to 14 to complete the registration process.

Images