Endoscopic image fusion method and electronic device

Abstract

The present application provides an endoscopic image fusion method and electronic device, wherein the method includes: collecting a fluorescent image of the target tissue through a fluorescent binocular endoscope, and collecting a white light image of the target tissue through a white light binocular endoscope; separating the target tissue from the fluorescent image, and calculating the three-dimensional coordinates of the target tissue in the coordinate system associated with the fluorescent binocular endoscope; converting the three-dimensional coordinates of the target tissue in the coordinate system associated with the fluorescent binocular endoscope into the three-dimensional coordinates of the target tissue in the coordinate system associated with the white light binocular endoscope according to the positive kinematics of the manipulator; reprojecting the target tissue onto the image plane of the white light binocular endoscope to obtain a reprojected image; superimposing the reprojected image and the white light image to obtain a target image. Through the above scheme, the problems of high cost and low accuracy of existing image registration are solved, and the technical effect of efficient and low-cost image registration is achieved.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular relates to an endoscopic image fusion method and electronic equipment. Background Art

[0002] Currently, image fusion methods for fluorescence endoscopy and white light endoscopy are mainly aimed at combining fluorescence images and white light images to provide more comprehensive and accurate diagnosis and treatment guidance. To achieve accurate fusion, image registration and image superposition are required.

[0003] Commonly used image registration methods for fluorescence endoscopy and white light endoscopy are:

[0004] 1) Mechanical alignment: A special mechanical structure, such as a beam splitter, is used to ensure that the angles of light received by the fluorescence and white-light image sensors are consistent. Alternatively, the same optical sensor is used to receive the fluorescence and white-light images in a time-sharing manner, thereby ensuring that the positions of the feature points in the captured fluorescence and white-light images are completely consistent.

[0005] 2) Feature point matching-based method: Using computer vision and image processing techniques, feature point pairs are extracted from the fluorescence image and the white light image. Then, the geometric transformation relationship between the matched feature points is calculated to obtain the mapping relationship between the two images.

[0006] 3) Mutual Information-Based Methods: Mutual information is a metric that measures the statistical correlation between two images. Mutual information-based methods achieve image registration by maximizing the mutual information between the images. Common optimization algorithms include gradient descent and genetic algorithms. These algorithms iteratively adjust image transformation parameters until the peak mutual information is reached.

[0007] 4) Deep learning-based methods: Use neural networks to learn the relationship between fluorescence images and white light images. By applying convolutional neural networks (CNNs) or generative adversarial networks (GANs), the feature representation and geometric transformation relationship between the images are obtained, thereby achieving high-precision image registration.

[0008] However, the above-mentioned image registration methods have problems to varying degrees. For example, the registration method based on mechanical structure alignment has relatively high requirements on the endoscope structure and is only applicable to integrated fluorescence and white light endoscopes, and requires the endoscope's lens and sensor to be consistent. However, integrated fluorescence and white light endoscopes often have complex structures, high prices, and resolutions that are lower than those of single-spectrum endoscopes. Registration methods based on feature point matching and mutual information often have difficulty handling complex scenes. Complex scenes may have multiple targets, occlusions, illumination changes, and other problems. These factors may affect the accuracy of feature extraction, and methods based on feature point matching are usually sensitive to image changes. If the image undergoes changes such as rotation, scaling, translation, or deformation, the feature extraction results may be affected, and feature matching or mutual information registration are usually time-consuming. Methods based on deep learning are, on the one hand, time-consuming, and on the other hand, their generalization capabilities are difficult to guarantee.

[0009] There is currently no effective solution to the problems of high cost and low accuracy in existing image registration. Summary of the Invention

[0010] The purpose of this application is to provide an endoscopic image fusion method and electronic equipment, which can achieve accurate and efficient fusion of fluorescent white light images during image registration.

[0011] The present application provides an endoscopic image fusion method, which is applied to an operating robot. The operating robot includes: a first robotic arm and a second robotic arm. The first robotic arm is provided with a fluorescent binocular endoscope, and the second robotic arm is provided with a white light binocular endoscope. The method includes:

[0012] Acquire a fluorescent image of the target tissue using a fluorescent binocular endoscope, and acquire a white light image of the target tissue using a white light binocular endoscope;

[0013] Separating the target tissue from the fluorescent image and calculating the three-dimensional coordinates of the target tissue in a coordinate system associated with the fluorescent binocular endoscope;

[0014] According to the forward kinematics of the manipulator, the three-dimensional coordinates of the target tissue in the coordinate system associated with the fluorescent binocular endoscope are converted into the three-dimensional coordinates of the target tissue in the coordinate system associated with the white light binocular endoscope;

[0015] reprojecting the target tissue onto an image plane of the white-light binocular endoscope according to the three-dimensional coordinates of the target tissue in a coordinate system associated with the white-light binocular endoscope to obtain a reprojected image;

[0016] The re-projected image and the white light image are superimposed and displayed to obtain a target image.

[0017] In one embodiment, according to the forward kinematics of the manipulator, converting the three-dimensional coordinates of the target tissue in the coordinate system associated with the fluorescent binocular endoscope into the three-dimensional coordinates of the target tissue in the coordinate system associated with the white light binocular endoscope includes:

[0018] Determining the three-dimensional coordinates of the target tissue in the coordinate system associated with the fluorescent binocular endoscope according to the depth information of the target tissue in the coordinate system associated with the fluorescent binocular endoscope;

[0019] Determining a positional relationship of the target tissue relative to the end of the robotic arm corresponding to the fluorescent binocular endoscope according to the three-dimensional coordinates of the target tissue in the coordinate system associated with the fluorescent binocular endoscope;

[0020] According to the positional relationship of the target tissue relative to the end of the robotic arm corresponding to the fluorescent binocular endoscope, based on the forward kinematics of the robotic arm, converting the positional relationship of the target tissue relative to the end of the robotic arm corresponding to the white light binocular endoscope;

[0021] The position relationship of the target tissue relative to the end of the robotic arm corresponding to the white light binocular endoscope is used to determine the three-dimensional coordinates of the target tissue in the camera coordinate system of the white light binocular endoscope.

[0022] In one embodiment, reprojecting the target tissue onto an image plane of the white-light binocular endoscope according to the three-dimensional coordinates of the target tissue in a coordinate system associated with the white-light binocular endoscope to obtain a reprojected image includes:

[0023] Determining the position of the target tissue relative to the white-light binocular endoscope according to the three-dimensional coordinates of the target tissue in the coordinate system associated with the white-light binocular endoscope and according to a pre-calibrated camera result;

[0024] According to the posture of the target tissue relative to the white-light binocular endoscope, the target tissue is reprojected onto the image plane of the white-light binocular endoscope to obtain a reprojected image.

[0025] In one embodiment, camera calibration is performed as follows:

[0026] Fix the target binocular endoscope through the endoscope fixing platform;

[0027] Input the end angle and focal length of the target binocular endoscope;

[0028] Controlling the distance and posture of the calibration plate relative to the lens of the target binocular endoscope according to a plurality of preset posture points;

[0029] When the distance and posture of the calibration plate relative to the lens of the target binocular endoscope reach a plurality of preset posture points, controlling the target binocular endoscope to capture an image of the calibration plate;

[0030] Feature extraction is performed on the collected image of the calibration plate, and the extrinsic parameters of the target binocular endoscope are obtained based on the extracted features to achieve camera calibration.

[0031] In one embodiment, after obtaining the extrinsic parameters of the target binocular endoscope based on the extracted features to implement camera calibration, the method further includes:

[0032] Obtain multiple poses of the calibration plate and the camera extrinsics corresponding to each pose;

[0033] Calculate the relative transformation of posture and the relative transformation of external parameters;

[0034] When the difference between the relative transformation of the external parameters and the relative transformation of the posture is less than a preset threshold, it is determined that the calibration is passed;

[0035] When the difference between the relative transformation of the external parameters and the relative transformation of the posture is not less than the preset threshold, recalibration is performed.

[0036] In one embodiment, reprojecting the target tissue onto an image plane of the white-light binocular endoscope according to the three-dimensional coordinates of the target tissue in a coordinate system associated with the white-light binocular endoscope to obtain a reprojected image includes:

[0037] reprojecting the target tissue onto a left image plane of the white-light binocular endoscope according to the three-dimensional coordinates of the target tissue in the coordinate system associated with the left camera of the white-light binocular endoscope to obtain a left image;

[0038] According to the three-dimensional coordinates of the target tissue in the coordinate system associated with the right camera of the white light binocular endoscope, the target tissue is reprojected onto the right image plane of the white light binocular endoscope to obtain a right image, wherein the left image and the right image are used to present a stereoscopic image effect.

[0039] In one embodiment, the re-projected image and the white light image are superimposed and displayed to obtain a target image, including:

[0040] Rendering the target tissue in the reprojected image into a preset color;

[0041] The rendered image is fused with the white light image to obtain a target image.

[0042] In one embodiment, the present invention is applied to a surgical robot provided with a single fluorescent binocular endoscope and a single white light binocular endoscope.

[0043] An electronic device comprises a processor and a memory for storing instructions executable by the processor, wherein the steps of the above method are implemented when the processor executes the instructions.

[0044] A computer-readable storage medium stores a computer program / instruction thereon, which implements the steps of the above method when executed by a processor.

[0045] The endoscopic image fusion method and electronic device provided by the present application collect the fluorescent image of the target tissue through a fluorescent binocular endoscope and collect the white light image of the target tissue through a white light binocular endoscope; then, the target tissue can be separated from the fluorescent image, and the three-dimensional coordinates of the target tissue in the coordinate system associated with the fluorescent binocular endoscope are calculated; then, according to the positive kinematics of the manipulator, the three-dimensional coordinates of the target tissue in the coordinate system associated with the fluorescent binocular endoscope are converted into the three-dimensional coordinates of the target tissue in the coordinate system associated with the white light binocular endoscope, and then the target tissue is reprojected onto the image plane of the white light binocular endoscope to obtain a reprojected image, and the reprojected image and the white light image are superimposed and displayed to obtain the target image. Through the above scheme, based on the conversion between the coordinate systems calibrated by the camera and the positive kinematics of the manipulator, the fusion of fluorescent and white light endoscope images is performed, which solves the problems of high cost and low accuracy of existing image registration, and achieves the technical effect of efficient and low-cost image registration. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0047] Figure 1 This is a schematic diagram of the architecture of an embodiment of a laparoscopic surgical robot system provided by the present application;

[0048] Figure 2 is a schematic diagram of the architecture of the image display platform 50 provided in this application;

[0049] Figure 3 is a schematic diagram of the structure of the surgical platform 40 provided in this application;

[0050] Figure 4 It is a schematic structural diagram of the binocular endoscope provided by the present application;

[0051] Figure 5 This is a structural diagram of the endoscope automatic calibration platform provided by this application;

[0052] Figure 6 This is another structural diagram of the endoscope automatic calibration platform provided by this application;

[0053] Figure 7 This is another structural diagram of the endoscope automatic calibration platform provided by this application;

[0054] Figure 8 This is a schematic structural diagram of the scope pitch angle adjustment module provided by the present application;

[0055] Figure 9 This is a schematic diagram of the structure of the electric three-dimensional rotating pan-tilt platform provided by the present application;

[0056] Figure 10 This is a schematic diagram of the three-dimensional pan-tilt transformation provided by this application;

[0057] Figure 11 This is a schematic diagram of the automatic calibration process of the endoscope provided by this application;

[0058] Figure 12 is a schematic diagram of the image plane of the binocular camera provided in this application;

[0059] Figure 13 is a schematic diagram of the coordinate system of the binocular endoscope provided in this application;

[0060] Figure 14 Schematic diagram of the coordinate system of the forward kinematics of the robotic arm provided in this application;

[0061] Figure 15 It is a schematic diagram of the operation of the double endoscope provided by this application;

[0062] Figure 16 This is a method flow chart of the fluorescent white light endoscope image fusion method provided by the present application;

[0063] Figure 17 This is a hardware structure block diagram of an electronic device for a fluorescent white light endoscope image fusion method provided in this application;

[0064] Figure 18 This is a schematic diagram of the module structure of an embodiment of the fluorescent white light endoscopic image fusion device provided in this application. DETAILED DESCRIPTION

[0065] In order to enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0066] In this example, a laparoscopic surgical robot system is provided. Figure 1 As shown, the system comprises a doctor's operating terminal 20, a patient's operating terminal 60, an image display platform 50, a surgical instrument library 10, life support equipment 30, a surgical platform 40, and a surgeon 70. This medical device combines laparoscopic and robotic technologies. By inserting a laparoscope and medical instruments controlled by a robotic arm into the patient's body, the surgeon operates the robotic arm through a console to perform surgeries such as urology, gynecology, thoracic surgery, and general surgery. The flexibility and precision of the robotic arm allow surgeons to perform more precise surgical procedures, reducing surgical risks and complications. In this system, the endoscope system provides the surgeon with high-definition images and a magnified field of view, allowing the surgeon to more clearly observe the surgical area.

[0067] The image display platform 50 can be as follows: Figure 2 As shown, it includes: a medical image display module 303, an image processing unit 302, a medical white light binocular endoscope 301, and a medical fluorescence binocular endoscope 304. The white light endoscope described in the present invention is an endoscope that uses the reflection imaging principle of common natural light (such as white light, blue light, etc.), and the fluorescence endoscope is an endoscope that generates fluorescence to form images when stimulated (such as near-infrared excitation, ultraviolet excitation, etc.). The image display platform captures the image of the surgical area through the endoscope, and the image processing unit processes and enhances the image and transmits it to the medical image display module, providing the surgeon with a clear field of view, enabling him to accurately observe the surgical area. Furthermore, the surgeon can also process the image (for example, by enlarging the image or marking the lesion area) to meet different surgical needs and the surgeon's personal preferences.

[0068] The above-mentioned surgical platform 40 (patient side) can be as follows Figure 3As shown, it includes: an operating trolley 100, a fluorescent binocular endoscope 101, a white light binocular endoscope 102, an adjustment arm 103, a tool arm 104 and a surgical instrument 105. The operating trolley usually includes multiple robotic arms (including tool arms and adjustment arms), and the end of the tool arm of each robotic arm is equipped with at least one surgical tool such as an endoscope or a surgical instrument. The surgeon operates the robotic arm in the operating platform through the console to move, and then operates the surgical instrument or endoscope. In the process of controlling the movement of the endoscope, the surgeon can observe the target area of ​​the operation more comprehensively. The operating platform can provide surgeons with more precise, stable and flexible surgical operations, reducing surgical trauma and bleeding. By equipping the operating platform with a fluorescent endoscope and a white light endoscope, the doctor's field of view can be expanded, and a more significant and clear lesion area can be observed.

[0069] For example, a surgical robot may include: a first robotic arm and a second robotic arm, the first robotic arm is provided with a fluorescent binocular endoscope, and the second robotic arm is provided with a white light binocular endoscope, that is, two robotic arms are used to carry a white light endoscope and a fluorescent endoscope respectively, and the two are not integrated into the same endoscope or the same robotic arm.

[0070] Among them, binocular endoscopes can be used as Figure 4 As shown, it includes: an endoscope base 11 that accommodates various electronic components, a light source path 12, an electronic cable path 13, and a camera lens 14. Figure 4 In the figure, 15 represents the front end of the endoscope's lens at different angles, and 16 represents the binocular lens of the endoscope. A binocular endoscope captures images of organs and tissues through its lens and transmits them to the observer's field of view. The light source system provides sufficient illumination and light of different wavelengths and characteristics, thereby displaying the different optical properties of the target area and enabling the observer to clearly see image details. The binocular endoscope simulates the human binocular vision system, achieving depth perception and three-dimensional reconstruction of the target area through the principles of inspection and triangulation, and achieving a stereoscopic visual effect.

[0071] In this example, a fluorescent white-light endoscopic image fusion method is provided. By using a single fluorescent binocular endoscope and a single white-light binocular endoscope, combined with camera calibration and robotic arm kinematics, the registration and fusion of fluorescent images and white-light images are realized in laparoscopic surgical robot surgery. That is, a method for fusing endoscopic fluorescent images and white-light images in the above-mentioned surgical robot application scenario is provided. By designing a set of endoscopic camera automatic calibration devices, and combining knowledge such as camera calibration and robotic arm kinematics, the images of the fluorescent endoscope and the white-light endoscope are corrected and registered, so that the fluorescent lesion tissue is displayed on the high-definition color endoscopic image in a color-coding-based manner, so that the doctor can observe the lesion and surrounding tissue at the same time, thereby better guiding diagnosis and surgery.

[0072] Specifically, the fluorescent white light endoscope image fusion method may include the following steps:

[0073] Step 1: Acquire a fluorescence image of the target tissue using a fluorescence binocular endoscope, and acquire a white light image of the target tissue using a white light binocular endoscope;

[0074] Step 2: Segment the lesion tissue from the fluorescence image based on the fluorescence characteristics and obtain the pixel coordinates of the lesion;

[0075] Step 3: Calculate the three-dimensional coordinates of the lesion in the fluorescence endoscope camera coordinate system;

[0076] Step 4: Calculate the three-dimensional coordinates of the lesion tissue under the white light endoscope based on the forward kinematics of the robotic arm;

[0077] Step 5: Reproject the lesion onto the white light image plane based on the three-dimensional coordinates of the lesion tissue under the white light endoscope;

[0078] Step 6: Render the reprojected image as the target fluorescent color and fuse it onto the white light image;

[0079] Step 7: Output the final fused image with lesion annotation for the doctor's reference.

[0080] Specifically, the registration and superposition of the fluorescence binocular endoscope image and the white light binocular endoscope image through camera calibration and robot arm kinematics may include the following steps:

[0081] Step 1: Calibrate the fluorescence endoscope and the white light endoscope through the endoscope automatic calibration device to obtain the camera internal and external parameters of the fluorescence endoscope and the camera internal and external parameters of the white light endoscope. The intrinsic parameters of the camera are parameters related to the camera's own characteristics, that is, the parameters inside the camera, such as: the focal length, pixel size, etc. of the camera; the extrinsic parameters of the camera determine the position and orientation of the camera in a certain three-dimensional space. The extrinsic parameters of the camera are parameters in the world coordinate system, such as: the position and rotation direction of the camera.

[0082] Among them, the endoscope automatic calibration platform can be used as Figure 5 The platform includes a binocular endoscope 201, an endoscope fixing and pitch angle adjustment device 202, a motorized screw module 203, a pan / tilt fixed base 204, a three-dimensional rotating motorized pan / tilt 205, and a checkerboard calibration plate 206. The platform, through the motorized screw module 203 and the three-dimensional rotating motorized pan / tilt 205, can control the movement and rotation of the calibration plate according to the calculated pose nodes. The control motor has an encoder to measure the motor's rotation angle. The endoscope fixing and pitch angle adjustment device 202 can adjust the angle of the endoscope lens relative to the calibration plate, making it more widely applicable to calibrating endoscopes with different lens angles.

[0083] The above-mentioned endoscope automatic calibration platform can also be used as follows Figure 6 As shown, the endoscope pitch angle adjustment device 202 is replaced with a rotary motor with an encoder, and the above-mentioned endoscope automatic calibration platform can also be as shown. Figure 7 As shown, the screw linear motion device 203 is replaced with a wire wheel drive. The specific structure can be selected according to actual needs and conditions, and this application does not limit this.

[0084] The above-mentioned mirror pitch angle adjustment module 202 can be as follows Figure 8 As shown, it includes: a sleeve 21 and a base 22 for fixing the endoscope, and an electric push rod 23 for adjusting the pitch angle. After the endoscope is fixed, the electric push rod can be extended and retracted according to the angle of the lens of the fixed endoscope, thereby adjusting the angle of the endoscope lens relative to the calibration plate to ensure that the endoscope at different angles can fully capture the image of the calibration plate.

[0085] The above-mentioned electric three-dimensional rotating platform 205 can be Figure 9 As shown, the system comprises a pan-tilt base 24, motors 25a, 25b, and 25c with angle encoders, and a calibration plate mounting base 26. After securing the calibration plate, the pan-tilt base, driven by the motor, moves the plate to different positions, triggering the endoscope to capture images at different positions, and recording the positions between these positions. This approach avoids the low accuracy associated with manually moving the calibration plate, improves the uniformity of the plate's position distribution in space, and improves consistency between different calibrations, while also achieving automated calibration.

[0086] The 3D gimbal can change its position as follows Figure 10 As shown, it includes: a calibration plate coordinate system 31 described by calibration pose node 1, a calibration plate coordinate system 32 described by calibration pose node 2, a pose transformation 33 between different calibration plate coordinate systems, and different pose nodes of the calibration plate during the calibration process. When the three-dimensional gimbal rotates the calibration plate to a node each time, it triggers the endoscope to capture an image of the calibration plate and record the poses of different nodes.

[0087] Based on the above-mentioned endoscope automatic calibration platform, Figure 11 As shown, the automatic calibration of the endoscope is performed as follows:

[0088] S1: Fix the binocular endoscope to be calibrated on the endoscope automatic calibration platform;

[0089] S2: Input the endoscope's end angle, lens manufacturing focal length and other product parameters;

[0090] S3: Design several pose nodes of the calibration plate during the calibration process according to the parameters;

[0091] S4: The distance between the calibration plate and the lens is controlled by the electric lead screw module, and the posture of the calibration plate relative to the lens is controlled by the three-dimensional rotating electric pan-tilt head;

[0092] S5: After the calibration plate moves to the planned pose node, the endoscope is controlled to capture an image of the calibration plate;

[0093] S6: Extract features from the acquired image and optimize it using optimization algorithms such as Zhang’s camera correction;

[0094] S7: Output the camera intrinsic parameters of the endoscope lens and the external parameters for different calibration plate poses.

[0095] Furthermore, whether the calibration is passed can be determined in one of the following ways:

[0096] Method 1: Determine whether the calibration is successful based on the calculated camera extrinsic parameters Tc and the calibration plate Tb pose. Specifically, the relative transformation can be calculated according to the following formula:

[0097]

[0098]

[0099] Calculate the average error:

[0100]

[0101] Determine the average error Is it greater than a given threshold? If it is greater than or equal to the given threshold, it is determined that recalibration is required. If it is less than the given threshold, it is determined that the calibration is passed.

[0102] Method 2: Input the internal and external parameters of the calibrated endoscope dual lens, calculate and decompose the essential matrix based on the internal and external parameters of the endoscope dual lens, and perform epipolar constraint calibration; project the 3D grid points of the calibration plate onto the image planes of the left and right dual cameras, and calculate the reprojection error; if the error is less than the set threshold, the calibration is determined to be successful.

[0103] After epipolar correction based on method 2, the binocular camera diagram can be as follows Figure 12 As shown, it may include: a left virtual image plane 409 and a right virtual image plane 410. The purpose of the binocular camera epipolar correction is to align the images of the left and right cameras on the same plane, so that when performing stereo vision processing, the same three-dimensional point can find a matching point in the same row of pixels on the left and right image planes, which simplifies the feature point matching process and improves the matching accuracy and efficiency.

[0104] Step 2: Determine the relative positions of the fluorescence endoscope camera, the white light endoscope camera coordinate system, and the end-point of the laparoscopic surgical robot through the robot hand-eye calibration.

[0105] Among them, the coordinate system of the binocular endoscope can be as follows Figure 13 As shown, it includes: endoscope left camera coordinate system 401, endoscope right camera coordinate system 402, left camera image plane 403, right camera image plane 404, left image coordinate system 405, right camera coordinate system 406, and world coordinate system 407. Based on this coordinate system, the relationship between the world coordinate system and the pixels of the left and right cameras can be derived, thereby realizing target depth perception based on the principles of parallax and triangulation.

[0106] Step 3: Derived the relative pose between the camera coordinate systems of the fluorescence endoscope and the white light endoscope through the forward kinematics of the laparoscopic surgical robot.

[0107] The forward kinematics of the manipulator can be expressed as Figure 14 As shown, in Figure 14 The figure shows the pose transformation relationship between the robot arm base coordinate system 411 and the endoscope camera coordinate system 412. Combining the robot arm's design parameters and DH calibration, the tool arm's forward kinematics equations are constructed. These equations are then updated using the angle feedback from each joint motor. The robot arm is mapped from joint space to end-of-arm Cartesian space, thereby obtaining the pose of the endoscope camera relative to the robot arm base in real time. Combined with binocular depth perception, the pose of the target tissue relative to the robot arm base can be determined.

[0108] Specifically, in this example, a schematic diagram of a dual endoscope operation is provided, such as Figure 15 As shown, the system includes: an operating table 413, lesion tissue 414, a fluorescence endoscope 415, and a white-light endoscope 416, and the process of transforming the image of the lesion tissue under the fluorescence endoscope into the image under the white-light endoscope. At a certain moment, the fluorescence binocular endoscope captures the image of the target tissue, separates the lesion from the image background by the fluorescence characteristics of the lesion, and obtains the depth information of the lesion. Combined with the forward kinematics of the manipulator, the position relationship of the lesion relative to the manipulator base is calculated, and then the position relationship of the lesion relative to the white-light endoscope camera coordinate system is obtained and projected onto the left and right image planes. At the same time, the white-light binocular endoscope captures the target tissue image, and then the two sets of left and right images are superimposed and displayed separately, thereby realizing the fusion of the fluorescence image and the white-light image.

[0109] In the above example, a single existing fluorescence binocular endoscope and a single white-light binocular endoscope are utilized to provide clear color images of lesion tissue without having to replace them with a fluorescence-white-light integrated endoscope, thus expanding the scope and effectiveness of the existing equipment. The fluorescence endoscope captures a depth image of the lesion and transforms it into the camera coordinate system of the white-light endoscope. This image is then reprojected onto the left and right image planes of the white-light endoscope. The reprojected image is rendered as the target fluorescence color and fused onto the white-light image, thereby displaying the fluorescent lesion on the high-definition color image, providing the physician with clearer and more intuitive information. Specifically, by designing an automatic calibration device for endoscope cameras, fully automatic calibration of endoscopes with different lens angles is achieved, improving the efficiency and consistency of endoscope calibration. By combining camera calibration with knowledge of robotic arm kinematics, precise correction and registration of endoscopic images is achieved. With accurate camera calibration and robotic arm kinematic models, image registration between the fluorescence binocular endoscope and the white-light binocular endoscope can be achieved, improving the accuracy of image fusion. Moreover, the computational complexity of image registration is less than that of existing image registration methods, and the registration results can provide a good initial value for image registration based on optimization methods such as gradient descent, thereby improving the final image registration accuracy.

[0110] Figure 16 It is a method flow chart of an embodiment of the fluorescent white light endoscopic image fusion method provided by the present application. Although the present application provides the method operation steps or device structure as shown in the following embodiments or drawings, more or fewer operation steps or module units may be included in the method or device based on routine or no creative labor. In the steps or structures where there is no necessary causal relationship logically, the execution order of these steps or the module structure of the device is not limited to the execution order or module structure described in the embodiments of the present application and shown in the drawings. When the method or module structure is applied to an actual device or terminal product, it can be connected in accordance with the method or module structure shown in the embodiment or drawings for sequential execution or parallel execution (for example, a parallel processor or multi-threaded processing environment, or even a distributed processing environment).

[0111] Specifically, such as Figure 16 As shown, the above-mentioned fluorescent white light endoscope image fusion method can be applied to an operating robot, wherein the operating robot includes: a first robotic arm and a second robotic arm, wherein the first robotic arm is provided with a fluorescent binocular endoscope, and the second robotic arm is provided with a white light binocular endoscope, and can include the following steps:

[0112] Step 1601: collecting a fluorescence image of the target tissue through a fluorescence binocular endoscope and collecting a white light image of the target tissue through a white light binocular endoscope;

[0113] Step 1602: Separate the target tissue from the fluorescence image, and calculate the three-dimensional coordinates of the target tissue in the coordinate system associated with the fluorescence binocular endoscope;

[0114] Step 1603: Convert the three-dimensional coordinates of the target tissue in the coordinate system associated with the fluorescent binocular endoscope into the three-dimensional coordinates of the target tissue in the coordinate system associated with the white light binocular endoscope according to the forward kinematics of the robotic arm;

[0115] Specifically, according to the forward kinematics of the manipulator, converting the three-dimensional coordinates of the target tissue in the coordinate system associated with the fluorescent binocular endoscope into the three-dimensional coordinates of the target tissue in the coordinate system associated with the white light binocular endoscope may include:

[0116] S1: determining the three-dimensional coordinates of the target tissue in the coordinate system associated with the fluorescent binocular endoscope according to the depth information of the target tissue in the coordinate system associated with the fluorescent binocular endoscope;

[0117] S2: determining a position relationship of the target tissue relative to the end of the robotic arm corresponding to the fluorescent binocular endoscope according to the three-dimensional coordinates of the target tissue in the coordinate system associated with the fluorescent binocular endoscope;

[0118] S3: converting the positional relationship of the target tissue relative to the end of the robotic arm corresponding to the fluorescent binocular endoscope into the positional relationship of the target tissue relative to the end of the robotic arm corresponding to the white light binocular endoscope based on the forward kinematics of the robotic arm;

[0119] S4: The position relationship of the target tissue relative to the end of the robotic arm corresponding to the white light binocular endoscope is determined to determine the three-dimensional coordinates of the target tissue in the camera coordinate system of the white light binocular endoscope.

[0120] That is, the conversion is made from the fluorescent binocular endoscope to the robotic arm end corresponding to the fluorescent binocular endoscope, and then from the robotic arm end corresponding to the fluorescent binocular endoscope to the robotic arm end corresponding to the white light binocular endoscope, and then from the robotic arm end corresponding to the white light binocular endoscope to the white light binocular endoscope, thereby obtaining the three-dimensional coordinates of the target tissue in the camera coordinate system of the white light binocular endoscope to achieve remapping.

[0121] Step 1604: reprojecting the target tissue onto an image plane of the white-light binocular endoscope according to the three-dimensional coordinates of the target tissue in the coordinate system associated with the white-light binocular endoscope to obtain a reprojected image;

[0122] When the target tissue is reprojected onto the image plane of the white-light binocular endoscope according to the three-dimensional coordinates of the target tissue in the coordinate system associated with the white-light binocular endoscope to obtain a reprojected image, the position of the target tissue relative to the white-light binocular endoscope can be determined according to the three-dimensional coordinates of the target tissue in the coordinate system associated with the white-light binocular endoscope and the pre-calibrated camera results; according to the position of the target tissue relative to the white-light binocular endoscope, the target tissue is reprojected onto the image plane of the white-light binocular endoscope to obtain a reprojected image.

[0123] Because in this example, both the white light endoscope and the fluorescence endoscope are binocular endoscopes, and therefore, both have a left image plane and a right image plane. Therefore, when the target tissue is reprojected onto the image plane of the white light binocular endoscope according to the three-dimensional coordinates of the target tissue in the coordinate system associated with the white light binocular endoscope to obtain a reprojected image, the target tissue can be reprojected onto the left image plane of the white light binocular endoscope according to the three-dimensional coordinates of the target tissue in the coordinate system associated with the left camera of the white light binocular endoscope to obtain a left image; and the target tissue can be reprojected onto the right image plane of the white light binocular endoscope according to the three-dimensional coordinates of the target tissue in the coordinate system associated with the right camera of the white light binocular endoscope to obtain a right image, wherein the left image and the right image are used to present a stereoscopic image effect to the operator (for example, a doctor).

[0124] Step 1605: The re-projected image and the white light image are superimposed and displayed to obtain a target image.

[0125] Specifically, superimposing the reprojected image and the white-light image to obtain the target image may include rendering the target tissue in the reprojected image in a preset color and fusing the rendered image with the white-light image to obtain the final target image. For example, if the target tissue is rendered in green, the reprojected image may be rendered in fluorescent green and then fused with the white-light image.

[0126] For the above-mentioned white light binocular endoscope and fluorescent binocular endoscope, camera calibration is required to ensure the conversion and correspondence of time pose and coordinate system. Specifically, camera calibration can be performed as follows:

[0127] S1: Fix the target binocular endoscope through the endoscope fixing platform;

[0128] S2: Input the end angle and focal length of the target binocular endoscope;

[0129] S3: controlling the distance and posture of the calibration plate relative to the lens of the target binocular endoscope according to a plurality of preset posture points;

[0130] S4: When the calibration plate reaches a distance and a posture relative to the lens of the target binocular endoscope and reaches a plurality of preset posture points, controlling the target binocular endoscope to capture an image of the calibration plate;

[0131] S5: extracting features from the collected image of the calibration plate, and obtaining the external parameters of the target binocular endoscope based on the extracted features to achieve camera calibration.

[0132] Furthermore, considering that a single calibration may not be accurate and successful during camera calibration, after obtaining the external parameters of the target binocular endoscope based on the extracted features to realize camera calibration, multiple postures of the calibration plate and the external parameters corresponding to each posture can be obtained; the relative transformation of the posture and the relative transformation of the external parameters are calculated; if the difference between the relative transformation of the external parameters and the relative transformation of the posture is less than a preset threshold, it is determined that the calibration is passed; if the difference between the relative transformation of the external parameters and the relative transformation of the posture is not less than a preset threshold, recalibration is performed.

[0133] The above-mentioned fluorescent white light endoscope image fusion method can be applied to a surgical robot equipped with a single fluorescent binocular endoscope and a single white light binocular endoscope, thereby providing doctors with clearer images and operation support during surgery.

[0134] In the above example, a fluorescence and white-light binocular endoscope image registration and fusion method combining camera calibration and robotic arm kinematics knowledge is provided. This method improves the efficiency of registration and ensures the accuracy of image registration through accurate camera calibration and accurate robotic arm forward kinematics. It achieves the fusion of fluorescence images and white-light images, reduces equipment costs, expands the scope of use of the original equipment, and provides a good initial value for subsequent image registration based on iterative optimization such as gradient descent, thereby improving the final image registration accuracy and reducing the number of iterations.

[0135] The method embodiments provided in the above embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on an electronic device as an example, Figure 17 This is a hardware structure diagram of an electronic device for a fluorescent white light endoscope image fusion method provided by this application. Figure 17 As shown, the electronic device may include one or more (only one is shown in the figure) processors 02 (the processor 02 may include but is not limited to a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 04 for storing data, and a transmission module 06 for communication functions. It will be understood by those skilled in the art that Figure 17The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 17 More or fewer components than shown, or with Figure 17 Different configurations shown.

[0136] The memory 04 can be used to store software programs and modules of application software, such as the program instructions / modules corresponding to the fluorescent white light endoscopic image fusion method in the embodiment of the present application. The processor 02 executes various functional applications and data processing by running the software programs and modules stored in the memory 04, that is, implementing the fluorescent white light endoscopic image fusion method of the above-mentioned application. The memory 04 may include a high-speed random access memory and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 04 may further include a memory remotely located relative to the processor 02, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0137] The transmission module 06 is used to receive or send data via a network. Specific examples of the aforementioned network may include a wireless network provided by a communications provider of the electronic device. In one embodiment, the transmission module 06 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission module 06 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0138] At the software level, the fluorescent white light endoscope image fusion device is applied to an operating robot, which includes a first robotic arm and a second robotic arm. The first robotic arm is provided with a fluorescent binocular endoscope, and the second robotic arm is provided with a white light binocular endoscope. The device can be used as follows: Figure 18 Shown, including:

[0139] An acquisition module 1801 is configured to acquire a fluorescent image of the target tissue using a fluorescent binocular endoscope and a white light image of the target tissue using a white light binocular endoscope;

[0140] a separation module 1802 for separating target tissue from the fluorescent image and calculating the three-dimensional coordinates of the target tissue in a coordinate system associated with the fluorescent binocular endoscope;

[0141] a conversion module 1803 for converting the three-dimensional coordinates of the target tissue in the coordinate system associated with the fluorescent binocular endoscope into the three-dimensional coordinates of the target tissue in the coordinate system associated with the white light binocular endoscope according to the forward kinematics of the manipulator;

[0142] a reprojection module 1804 for reprojecting the target tissue onto an image plane of the white-light binocular endoscope according to the three-dimensional coordinates of the target tissue in the coordinate system associated with the white-light binocular endoscope to obtain a reprojected image;

[0143] The superposition module 1805 is configured to superimpose the re-projected image and the white light image to obtain a target image.

[0144] In one embodiment, the above-mentioned conversion module 1803 can specifically determine the three-dimensional coordinates of the target tissue in the coordinate system associated with the fluorescent binocular endoscope based on the depth information of the target tissue in the coordinate system associated with the fluorescent binocular endoscope; determine the posture relationship of the target tissue relative to the end of the robotic arm corresponding to the fluorescent binocular endoscope based on the three-dimensional coordinates of the target tissue in the coordinate system associated with the fluorescent binocular endoscope; convert the posture relationship of the target tissue relative to the end of the robotic arm corresponding to the fluorescent binocular endoscope into the posture relationship of the target tissue relative to the end of the robotic arm corresponding to the white light binocular endoscope based on the forward kinematics of the robotic arm; determine the three-dimensional coordinates of the target tissue in the camera coordinate system of the white light binocular endoscope based on the posture relationship of the target tissue relative to the end of the robotic arm corresponding to the white light binocular endoscope.

[0145] In one embodiment, the above-mentioned reprojection module 1804 can specifically determine the position of the target tissue relative to the white-light binocular endoscope according to the three-dimensional coordinates of the target tissue in the coordinate system associated with the white-light binocular endoscope and the pre-calibrated camera results; and reproject the target tissue onto the image plane of the white-light binocular endoscope according to the position of the target tissue relative to the white-light binocular endoscope to obtain a reprojected image.

[0146] In one embodiment, camera calibration can be performed as follows: fix the target binocular endoscope through an endoscope fixing platform; input the end angle and focal length of the target binocular endoscope; control the distance and posture of the calibration plate relative to the lens of the target binocular endoscope according to a plurality of preset posture points; when the distance and posture of the calibration plate relative to the lens of the target binocular endoscope reach a plurality of preset posture points, control the target binocular endoscope to capture an image of the calibration plate; perform feature extraction on the captured image of the calibration plate, and obtain the external parameters of the target binocular endoscope based on the extracted features to achieve camera calibration.

[0147] In one embodiment, after the extrinsic parameters of the target binocular endoscope are obtained based on the extracted features to realize camera calibration, multiple postures of the calibration plate and the camera extrinsic parameters corresponding to each posture can also be obtained; the relative transformation of the posture and the relative transformation of the extrinsic parameters are calculated; when the difference between the relative transformation of the extrinsic parameters and the relative transformation of the posture is less than a preset threshold, it is determined that the calibration is passed; when the difference between the relative transformation of the extrinsic parameters and the relative transformation of the posture is not less than a preset threshold, recalibration is performed.

[0148] In one embodiment, the above-mentioned reprojection module 1804 can specifically reproject the target tissue onto the left image plane of the white light binocular endoscope according to the three-dimensional coordinates of the target tissue in the coordinate system associated with the left camera of the white light binocular endoscope to obtain a left image; and reproject the target tissue onto the right image plane of the white light binocular endoscope according to the three-dimensional coordinates of the target tissue in the coordinate system associated with the right camera of the white light binocular endoscope to obtain a right image, wherein the left image and the right image are used to present a stereoscopic image effect.

[0149] In one embodiment, the superposition module 1805 may render the target tissue in the re-projected image as a preset fluorescent color; and fuse the image rendered with a slight preset fluorescent color with the white light image to obtain the target image.

[0150] In one embodiment, the above-mentioned device can be applied to a surgical robot provided with a single fluorescent binocular endoscope and a single white light binocular endoscope.

[0151] The embodiments of the present application also provide a specific implementation of an electronic device capable of implementing all steps in the fluorescent white light endoscopic image fusion method in the above embodiment. The electronic device specifically includes the following contents: a processor, a memory, a communication interface, and a bus; wherein the processor, the memory, and the communication interface communicate with each other via the bus; the processor is used to call a computer program in the memory, and when the processor executes the computer program, all steps in the fluorescent white light endoscopic image fusion method in the above embodiment are implemented. For example, when the processor executes the computer program, the following steps are implemented:

[0152] Step 1: Acquire a fluorescence image of the target tissue using a fluorescence binocular endoscope, and acquire a white light image of the target tissue using a white light binocular endoscope;

[0153] Step 2: Separating the target tissue from the fluorescence image and calculating the three-dimensional coordinates of the target tissue in the coordinate system associated with the fluorescence binocular endoscope;

[0154] Step 3: According to the forward kinematics of the robotic arm, the three-dimensional coordinates of the target tissue in the coordinate system associated with the fluorescent binocular endoscope are converted into the three-dimensional coordinates of the target tissue in the coordinate system associated with the white light binocular endoscope;

[0155] Step 4: reprojecting the target tissue onto an image plane of the white-light binocular endoscope according to the three-dimensional coordinates of the target tissue in the coordinate system associated with the white-light binocular endoscope to obtain a reprojected image;

[0156] Step 5: Overlay and display the re-projected image and the white light image to obtain a target image.

[0157] The embodiments of the present application also provide a computer-readable storage medium capable of implementing all steps of the fluorescent white-light endoscopic image fusion method in the above-mentioned embodiment. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, all steps of the fluorescent white-light endoscopic image fusion method in the above-mentioned embodiment are implemented. For example, when the processor executes the computer program, the following steps are implemented:

[0158] Step 1: Acquire a fluorescence image of the target tissue using a fluorescence binocular endoscope, and acquire a white light image of the target tissue using a white light binocular endoscope;

[0159] Step 2: Separating the target tissue from the fluorescence image and calculating the three-dimensional coordinates of the target tissue in the coordinate system associated with the fluorescence binocular endoscope;

[0160] Step 3: According to the forward kinematics of the robotic arm, the three-dimensional coordinates of the target tissue in the coordinate system associated with the fluorescent binocular endoscope are converted into the three-dimensional coordinates of the target tissue in the coordinate system associated with the white light binocular endoscope;

[0161] Step 4: reprojecting the target tissue onto an image plane of the white-light binocular endoscope according to the three-dimensional coordinates of the target tissue in the coordinate system associated with the white-light binocular endoscope to obtain a reprojected image;

[0162] Step 5: Overlay and display the re-projected image and the white light image to obtain a target image.

[0163] From the above description, it can be seen that the embodiment of the present application collects the fluorescent image of the target tissue through a fluorescent binocular endoscope and collects the white light image of the target tissue through a white light binocular endoscope; then, the target tissue can be separated from the fluorescent image, and the three-dimensional coordinates of the target tissue in the coordinate system associated with the fluorescent binocular endoscope are calculated; then, according to the positive kinematics of the manipulator, the three-dimensional coordinates of the target tissue in the coordinate system associated with the fluorescent binocular endoscope are converted into the three-dimensional coordinates of the target tissue in the coordinate system associated with the white light binocular endoscope, and then the target tissue is reprojected onto the image plane of the white light binocular endoscope to obtain a reprojected image, and the reprojected image and the white light image are superimposed and displayed to obtain the target image. Through the above scheme, based on the conversion between the coordinate systems calibrated by the camera and the positive kinematics of the manipulator, the fusion of the fluorescent white light endoscope images is performed, which solves the problems of high cost and low accuracy of the existing image registration, and achieves the technical effect of efficient and low-cost image registration.

[0164] The various embodiments in this specification are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other. Each embodiment focuses on the differences between the other embodiments. In particular, the hardware + program embodiments are generally similar to the method embodiments, so their description is relatively simple. For relevant portions, refer to the description of the method embodiments.

[0165] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0166] Although the present application provides method operation steps as described in the embodiments or flowcharts, more or fewer operation steps may be included based on conventional or non-creative work. The order of steps listed in the embodiments is only one way of executing the steps among many steps and does not represent the only execution order. When the actual device or client product is executed, it can be executed in sequence or in parallel according to the method shown in the embodiments or the drawings (for example, in a parallel processor or multi-threaded processing environment).

[0167] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, an in-vehicle human-computer interaction device, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0168] Although the present specification embodiment provides the method operation steps as described in the embodiment or flow chart, more or less operation steps may be included based on conventional or non-creative means. The order of steps listed in the embodiment is only one way in the order of execution of many steps and does not represent a unique execution order. When the device or terminal product in practice is executed, it can be performed in sequence or in parallel according to the method shown in the embodiment or the accompanying drawings (such as a parallel processor or a multi-threaded processing environment, or even a distributed data processing environment). The term "comprise", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, product or equipment including a series of elements not only include those elements, but also include other elements not clearly listed, or also include elements inherent to such process, method, product or equipment. In the absence of more restrictions, it is not excluded that there are other identical or equivalent elements in the process, method, product or equipment including the elements.

[0169] For the convenience of description, the above devices are described in terms of functions divided into various modules. Of course, when implementing the embodiments of this specification, the functions of each module can be implemented in the same or multiple software and / or hardware, or the modules that implement the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0170] Those skilled in the art will also appreciate that, in addition to implementing the controller in pure computer-readable program code, it is entirely possible to implement the same functionality by logically programming the method steps in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, embedded microcontrollers, and the like. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be considered structures within the hardware component. Alternatively, the devices for implementing various functions can be considered both software modules implementing the method and structures within the hardware component.

[0171] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0172] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0173] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0174] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0175] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0176] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0177] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0178] Embodiments of this specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. Embodiments of this specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In distributed computing environments, program modules may be located in local and remote computer storage media, including storage devices.

[0179] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between the various embodiments can be referenced across them. Each embodiment focuses on the differences from the other embodiments. In particular, since the system embodiments are generally similar to the method embodiments, their description is relatively simple. For relevant parts, reference can be made to the description of the method embodiments. Throughout this specification, reference to the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of the embodiments in this specification. In this specification, the schematic representations of these terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, those skilled in the art may combine and integrate the different embodiments or examples, and features of different embodiments or examples, described in this specification, without conflict.

[0180] The above description is merely an example of the embodiments of this specification and is not intended to limit the embodiments of this specification. For those skilled in the art, various modifications and variations of the embodiments of this specification are possible. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of this specification shall be included within the scope of the claims of the embodiments of this specification.

Claims

1. An endoscopic image fusion method, characterized in that: Applied to an operating robot, the operating robot includes: a first robotic arm and a second robotic arm, the first robotic arm is provided with a fluorescent binocular endoscope, and the second robotic arm is provided with a white light binocular endoscope, the method includes: Acquire a fluorescent image of the target tissue using a fluorescent binocular endoscope, and acquire a white light image of the target tissue using a white light binocular endoscope; Separating the target tissue from the fluorescent image and calculating the three-dimensional coordinates of the target tissue in a coordinate system associated with the fluorescent binocular endoscope; According to the forward kinematics of the manipulator, the three-dimensional coordinates of the target tissue in the coordinate system associated with the fluorescent binocular endoscope are converted into the three-dimensional coordinates of the target tissue in the coordinate system associated with the white light binocular endoscope; reprojecting the target tissue onto an image plane of the white-light binocular endoscope according to the three-dimensional coordinates of the target tissue in a coordinate system associated with the white-light binocular endoscope to obtain a reprojected image; The re-projected image and the white light image are superimposed and displayed to obtain a target image.

2. The method according to claim 1, characterized in that According to the forward kinematics of the manipulator, the three-dimensional coordinates of the target tissue in the coordinate system associated with the fluorescent binocular endoscope are converted into the three-dimensional coordinates of the target tissue in the coordinate system associated with the white light binocular endoscope, including: Determining the three-dimensional coordinates of the target tissue in the coordinate system associated with the fluorescent binocular endoscope according to the depth information of the target tissue in the coordinate system associated with the fluorescent binocular endoscope; Determining a positional relationship of the target tissue relative to the end of the robotic arm corresponding to the fluorescent binocular endoscope according to the three-dimensional coordinates of the target tissue in the coordinate system associated with the fluorescent binocular endoscope; According to the positional relationship of the target tissue relative to the end of the robotic arm corresponding to the fluorescent binocular endoscope, based on the forward kinematics of the robotic arm, converting the positional relationship of the target tissue relative to the end of the robotic arm corresponding to the white light binocular endoscope; The position relationship of the target tissue relative to the end of the robotic arm corresponding to the white light binocular endoscope is used to determine the three-dimensional coordinates of the target tissue in the camera coordinate system of the white light binocular endoscope.

3. The method according to claim 1, characterized in that Reprojecting the target tissue onto an image plane of the white-light binocular endoscope according to the three-dimensional coordinates of the target tissue in a coordinate system associated with the white-light binocular endoscope to obtain a reprojected image includes: Determining the position of the target tissue relative to the white-light binocular endoscope according to the three-dimensional coordinates of the target tissue in the coordinate system associated with the white-light binocular endoscope and according to a pre-calibrated camera result; According to the posture of the target tissue relative to the white-light binocular endoscope, the target tissue is reprojected onto the image plane of the white-light binocular endoscope to obtain a reprojected image.

4. The method according to claim 3, characterized in that Perform camera calibration as follows: Fix the target binocular endoscope through the endoscope fixing platform; Input the end angle and focal length of the target binocular endoscope; Controlling the distance and posture of the calibration plate relative to the lens of the target binocular endoscope according to a plurality of preset posture points; When the distance and posture of the calibration plate relative to the lens of the target binocular endoscope reach a plurality of preset posture points, controlling the target binocular endoscope to capture an image of the calibration plate; Feature extraction is performed on the collected image of the calibration plate, and the external parameters of the target binocular endoscope are obtained based on the extracted features to achieve camera calibration.

5. The method according to claim 4, characterized in that After obtaining the external parameters of the target binocular endoscope based on the extracted features to achieve camera calibration, the method further includes: Obtain multiple poses of the calibration plate and the camera extrinsics corresponding to each pose; Calculate the relative transformation of posture and the relative transformation of external parameters; When the difference between the relative transformation of the external parameters and the relative transformation of the posture is less than a preset threshold, it is determined that the calibration is passed; When the difference between the relative transformation of the external parameters and the relative transformation of the posture is not less than the preset threshold, recalibration is performed.

6. The method according to claim 1, characterized in that Reprojecting the target tissue onto an image plane of the white-light binocular endoscope according to the three-dimensional coordinates of the target tissue in a coordinate system associated with the white-light binocular endoscope to obtain a reprojected image includes: reprojecting the target tissue onto a left image plane of the white-light binocular endoscope according to the three-dimensional coordinates of the target tissue in the coordinate system associated with the left camera of the white-light binocular endoscope to obtain a left image; According to the three-dimensional coordinates of the target tissue in the coordinate system associated with the right camera of the white light binocular endoscope, the target tissue is reprojected onto the right image plane of the white light binocular endoscope to obtain a right image, wherein the left image and the right image are used to present a stereoscopic image effect.

7. The method according to claim 1, characterized in that The reprojected image and the white light image are superimposed and displayed to obtain a target image, comprising: Rendering the target tissue in the reprojected image into a preset color; The rendered image is fused with the white light image to obtain a target image.

8. The method according to any one of claims 1 to 7, characterized in that It is used in a surgical robot equipped with a single fluorescent binocular endoscope and a single white light binocular endoscope.

9. An electronic device comprising a processor and a memory for storing instructions executable by the processor, characterized in that: When the processor executes the instructions, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images