Endoscope image fusion method and electronic equipment

Abstract

The invention provides an endoscope image fusion method and electronic equipment, and the method comprises the steps: collecting a fluorescent image of a target tissue through a fluorescent binocular endoscope, and collecting a white light image of the target tissue through a white light binocular endoscope; separating a target tissue from the fluorescence image, and calculating a three-dimensional coordinate of the target tissue in a coordinate system associated with the fluorescence binocular endoscope; according to the forward kinematics of the mechanical arm, the three-dimensional coordinates of the target tissue in the coordinate system associated with the fluorescent binocular endoscope are converted into the three-dimensional coordinates of the target tissue in the coordinate system associated with the white light binocular endoscope; re-projecting the target tissue to an image plane of the white light binocular endoscope to obtain a re-projected image; and displaying the re-projection image and the white light image in an overlapping manner to obtain a target image. According to the scheme, the problems of high cost and low accuracy of existing image registration are solved, and the technical effect of carrying out image registration with high efficiency and low cost is achieved.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular relates to an endoscopic image fusion method and electronic equipment. Background Art

[0002] At present, the image fusion method of fluorescence endoscope and white light endoscope is mainly to combine fluorescence image and white light image to provide more comprehensive and accurate diagnosis and treatment guidance. In order to achieve accurate fusion, image registration and image superposition are required.

[0003] Commonly used image registration methods for fluorescence endoscopy and white light endoscopy are:

[0004] 1) Method based on mechanical structure alignment: A special mechanical structure is used, such as a beam splitter, to ensure that the angles of light received by the sensors of the fluorescence image and the white light image are consistent. Alternatively, the same optical sensor is used to receive the fluorescence image and the white light image in time-sharing mode, thereby ensuring that the positions of the feature points of the captured fluorescence image and the white light image are completely consistent.

[0005] 2) Feature point matching-based method: Using computer vision and image processing technology, feature point pairs are extracted from the fluorescent image and the white light image. Then, the geometric transformation relationship between the matched feature points is calculated to obtain the mapping relationship between the two images.

[0006] 3) Mutual information-based methods: Mutual information is a metric that measures the statistical correlation between two images. Mutual information-based methods achieve image registration by maximizing the mutual information between images. Commonly used optimization algorithms include gradient descent and genetic algorithms. These algorithms continuously adjust image transformation parameters in an iterative manner until the peak value of the mutual information is reached.

[0007] 4) Deep learning-based methods: Use neural networks to learn the relationship between fluorescence images and white light images. By applying convolutional neural networks (CNN) or generative adversarial networks (GAN) to learn the feature representation and geometric transformation relationship between images, high-precision image registration can be achieved.

[0008] However, the above-mentioned image registration methods have problems to varying degrees. For example, the registration method based on mechanical structure alignment has relatively high requirements on the endoscope structure and is only applicable to fluorescent and white light integrated endoscopes, and requires the endoscope's lens and sensor to be consistent. However, fluorescent and white light integrated endoscopes often have complex structures, high prices, and resolutions that are lower than those of single spectrum endoscopes. Registration methods based on feature point matching and mutual information often have difficulty in handling complex scenes. Complex scenes may have multiple targets, occlusions, illumination changes, and other problems, which may affect the accuracy of feature extraction. Methods based on feature point matching are usually sensitive to image changes. If the image undergoes changes such as rotation, scaling, translation or deformation, the feature extraction results may be affected, and feature matching or mutual information registration are usually time-consuming. Methods based on deep learning are time-consuming on the one hand, and their generalization ability is difficult to guarantee on the other.

[0009] There is no effective solution to the problems of high cost and low accuracy in existing image registration. Summary of the invention

[0010] The purpose of the present application is to provide an endoscopic image fusion method and electronic equipment, which can achieve accurate and efficient fusion of fluorescent white light images during image registration.

[0011] The present application provides an endoscope image fusion method, which is applied to an operating robot, wherein the operating robot comprises: a first mechanical arm and a second mechanical arm, wherein the first mechanical arm is provided with a fluorescent binocular endoscope, and the second mechanical arm is provided with a white light binocular endoscope, and the method comprises:

[0012] Acquire a fluorescent image of the target tissue by a fluorescent binocular endoscope, and acquire a white light image of the target tissue by a white light binocular endoscope;

[0013] Separating the target tissue from the fluorescent image and calculating the three-dimensional coordinates of the target tissue in a coordinate system associated with the fluorescent binocular endoscope;

[0014] According to the forward kinematics of the robot arm, the three-dimensional coordinates of the target tissue in the coordinate system associated with the fluorescent binocular endoscope are converted into the three-dimensional coordinates of the target tissue in the coordinate system associated with the white light binocular endoscope;

[0015] Reprojecting the target tissue onto an image plane of the white-light binocular endoscope according to the three-dimensional coordinates of the target tissue in a coordinate system associated with the white-light binocular endoscope to obtain a reprojected image;

[0016] The reprojected image and the white light image are superimposed and displayed to obtain a target image.

[0017] In one embodiment, according to the forward kinematics of the manipulator, converting the three-dimensional coordinates of the target tissue in the coordinate system associated with the fluorescent binocular endoscope into the three-dimensional coordinates of the target tissue in the coordinate system associated with the white light binocular endoscope comprises:

[0018] Determine the three-dimensional coordinates of the target tissue in the coordinate system associated with the fluorescent binocular endoscope according to the depth information of the target tissue in the coordinate system associated with the fluorescent binocular endoscope;

[0019] Determine, according to the three-dimensional coordinates of the target tissue in the coordinate system associated with the fluorescent binocular endoscope, the position and posture relationship of the target tissue relative to the end of the robotic arm corresponding to the fluorescent binocular endoscope;

[0020] According to the position and posture relationship of the target tissue relative to the end of the robotic arm corresponding to the fluorescent binocular endoscope, based on the forward kinematics of the robotic arm, converting it into the position and posture relationship of the target tissue relative to the end of the robotic arm corresponding to the white light binocular endoscope;

[0021] The position relationship of the target tissue relative to the end of the robotic arm corresponding to the white light binocular endoscope is used to determine the three-dimensional coordinates of the target tissue in the camera coordinate system of the white light binocular endoscope.

[0022] In one embodiment, according to the three-dimensional coordinates of the target tissue in the coordinate system associated with the white light binocular endoscope, the target tissue is reprojected onto the image plane of the white light binocular endoscope to obtain a reprojected image, including:

[0023] According to the three-dimensional coordinates of the target tissue in the coordinate system associated with the white light binocular endoscope and in accordance with the pre-calibrated camera result, the position and posture of the target tissue relative to the white light binocular endoscope is determined;

[0024] According to the position of the target tissue relative to the white light binocular endoscope, the target tissue is reprojected onto the image plane of the white light binocular endoscope to obtain a reprojected image.

[0025] In one embodiment, the camera calibration is performed as follows:

[0026] Fixing the target binocular endoscope via an endoscope fixing platform;

[0027] Input the end angle and focal length of the target binocular endoscope;

[0028] According to a plurality of preset posture points, controlling the distance and posture of the calibration plate relative to the lens of the target binocular endoscope;

[0029] When the distance and posture of the calibration plate relative to the lens of the target binocular endoscope reach a plurality of preset posture points, controlling the target binocular endoscope to collect an image of the calibration plate;

[0030] Feature extraction is performed on the collected image of the calibration plate, and the external parameters of the target binocular endoscope are obtained according to the extracted features to achieve camera calibration.

[0031] In one embodiment, after obtaining the external parameters of the target binocular endoscope according to the extracted features to implement camera calibration, the method further includes:

[0032] Obtain multiple poses of the calibration plate and the camera extrinsics corresponding to each pose;

[0033] Calculate the relative transformation of posture and the relative transformation of external parameters;

[0034] When the difference between the relative transformation of the external parameters and the relative transformation of the posture is less than a preset threshold, it is determined that the calibration is passed;

[0035] When the difference between the relative transformation of the external parameters and the relative transformation of the posture is not less than a preset threshold, recalibration is performed.

[0036] In one embodiment, according to the three-dimensional coordinates of the target tissue in the coordinate system associated with the white light binocular endoscope, the target tissue is reprojected onto the image plane of the white light binocular endoscope to obtain a reprojected image, including:

[0037] Reprojecting the target tissue onto a left image plane of the white light binocular endoscope according to the three-dimensional coordinates of the target tissue in the coordinate system associated with the left camera of the white light binocular endoscope to obtain a left image;

[0038] According to the three-dimensional coordinates of the target tissue in the coordinate system associated with the right camera of the white light binocular endoscope, the target tissue is reprojected onto the right image plane of the white light binocular endoscope to obtain a right image, wherein the left image and the right image are used to present a stereoscopic image effect.

[0039] In one embodiment, the reprojected image and the white light image are superimposed and displayed to obtain a target image, including:

[0040] Rendering the target tissue in the reprojected image into a preset color;

[0041] The rendered image is fused with the white light image to obtain a target image.

[0042] In one embodiment, the invention is applied to a surgical robot provided with a single fluorescent binocular endoscope and a single white light binocular endoscope.

[0043] An electronic device comprises a processor and a memory for storing instructions executable by the processor, wherein the steps of the above method are implemented when the processor executes the instructions.

[0044] A computer-readable storage medium stores a computer program / instruction, which implements the steps of the above method when executed by a processor.

[0045] The endoscopic image fusion method and electronic device provided by the present application collect the fluorescent image of the target tissue through a fluorescent binocular endoscope, and collect the white light image of the target tissue through a white light binocular endoscope; then, the target tissue can be separated from the fluorescent image, and the three-dimensional coordinates of the target tissue in the coordinate system associated with the fluorescent binocular endoscope are calculated; then, according to the positive kinematics of the manipulator, the three-dimensional coordinates of the target tissue in the coordinate system associated with the fluorescent binocular endoscope are converted into the three-dimensional coordinates of the target tissue in the coordinate system associated with the white light binocular endoscope, and then the target tissue is reprojected onto the image plane of the white light binocular endoscope to obtain a reprojected image, and the reprojected image and the white light image are superimposed and displayed to obtain the target image. Through the above scheme, based on the conversion between the coordinate systems calibrated by the camera and the positive kinematics of the manipulator, the fusion of fluorescent white light endoscope images is performed, which solves the problems of high cost and low accuracy in the existing image registration, and achieves the technical effect of efficient and low-cost image registration. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0047] Figure 1 It is a schematic diagram of the architecture of an embodiment of a laparoscopic surgical robot system provided by the present application;

[0048] Figure 2 is a schematic diagram of the architecture of the image display platform 50 provided by the present application;

[0049] Figure 3 is a schematic diagram of the structure of the surgical platform 40 provided in the present application;

[0050] Figure 4 is a schematic diagram of the structure of a binocular endoscope provided by the present application;

[0051] Figure 5 It is a structural schematic diagram of an endoscope automatic calibration platform provided by the present application;

[0052] Figure 6 It is another structural schematic diagram of the endoscope automatic calibration platform provided by the present application;

[0053] Figure 7 It is another structural schematic diagram of the endoscope automatic calibration platform provided by the present application;

[0054] Figure 8 It is a structural schematic diagram of the endoscope pitch angle adjustment module provided by the present application;

[0055] Fig. 9 It is a structural schematic diagram of the electric three-dimensional rotating pan-tilt platform provided by the present application;

[0056] Fig.10 It is a schematic diagram of the posture transformation of the three-dimensional gimbal provided by this application;

[0057] Fig.11 It is a schematic diagram of the automatic calibration process of the endoscope provided by the present application;

[0058] Fig.12 is a schematic diagram of the image plane of the binocular camera provided in this application;

[0059] Fig.13 is a schematic diagram of the coordinate system of the binocular endoscope provided in this application;

[0060] Fig.14 is a schematic diagram of the coordinate system of the forward kinematics of the robotic arm provided in this application;

[0061] Fig.15 It is a schematic diagram of the working of the double endoscope provided by the present application;

[0062] Fig.16 is a method flow chart of the fluorescent white light endoscope image fusion method provided by the present application;

[0063] Fig.17 It is a hardware structure block diagram of an electronic device for a fluorescent white light endoscope image fusion method provided by the present application;

[0064] Fig.18 It is a schematic diagram of the module structure of an embodiment of the fluorescent white light endoscope image fusion device provided in the present application. DETAILED DESCRIPTION

[0065] In order to enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0066] In this example, a laparoscopic surgical robot system is provided. Figure 1 As shown, it includes: a doctor's operating end 20, a patient's operating end 60, an image display platform 50, a surgical instrument library 10, a life support device 30, a surgical platform 40 and a surgeon 70. It is a medical device that combines laparoscopic technology with robotic technology. By inserting a laparoscope and medical instruments controlled by a robotic arm into the patient's body, the surgeon operates the robotic arm through a console to perform surgical operations to complete urology, gynecology, thoracic surgery, general surgery and other operations. Due to the flexibility and precision of the robotic arm, the surgeon can perform more delicate surgical operations and reduce the occurrence of surgical risks and complications. In this system, the endoscope system can provide doctors with high-definition images and magnified fields of view, allowing surgeons to observe the surgical area more clearly.

[0067] The image display platform 50 can be as follows: Figure 2 As shown, it includes: a medical image display module 303, an image processing unit 302, a medical white light binocular endoscope 301, and a medical fluorescence binocular endoscope 304. The white light endoscope described in the present invention is an endoscope based on the reflection imaging principle of common natural light (such as white light, blue light, etc.), and the fluorescence endoscope is an endoscope that is commonly excited (such as near-infrared excitation, ultraviolet excitation, etc.) to generate fluorescence for imaging. The image display platform captures the image of the surgical area through an endoscope, and the image is processed and enhanced by the image processing unit, and transmitted to the medical image display module, providing a clear field of view for the surgeon, so that the surgeon can accurately observe the surgical area. Furthermore, the surgeon can also process the image (for example: enlarge the image or mark the lesion area, etc.) to meet different surgical needs and the surgeon's personal preferences.

[0068] The above-mentioned surgical platform 40 (patient side) can be as follows Figure 3As shown, it includes: an operating trolley 100, a fluorescent binocular endoscope 101, a white light binocular endoscope 102, an adjustment arm 103, a tool arm 104 and a surgical instrument 105. The operating trolley usually includes multiple mechanical arms (including tool arms and adjustment arms), and the end of the tool arm of each mechanical arm is equipped with at least one surgical tool such as an endoscope or a surgical instrument. The surgeon operates the mechanical arm in the operating platform through the console to move, and then operates the surgical instrument or endoscope. In the process of controlling the movement of the endoscope, the surgeon can observe the target area of ​​the operation more comprehensively. The operating platform can provide surgeons with more precise, stable and flexible surgical operations, reducing surgical trauma and bleeding. By equipping the operating platform with a fluorescent endoscope and a white light endoscope, the doctor's field of vision can be expanded to observe more significant and clear lesion areas.

[0069] For example, a surgical robot may include: a first robotic arm and a second robotic arm, the first robotic arm is provided with a fluorescent binocular endoscope, and the second robotic arm is provided with a white light binocular endoscope, that is, two robotic arms are used to carry a white light endoscope and a fluorescent endoscope respectively, and the two are not integrated into the same endoscope or the same robotic arm.

[0070] Among them, binocular endoscopes can be used as Figure 4 As shown, it includes: an endoscope base 11 accommodating various electronic components, a light source passage 12, an electronic cable passage 13, and a camera lens 14, wherein Figure 4 In the figure, 15 indicates the front end of the endoscope lens at different angles, and 16 indicates the binocular lens of the endoscope. The binocular endoscope captures the images of organs and tissues through the lens and transmits them to the observer's field of view. The light source system provides sufficient illumination and light of different wavelengths and characteristics, thereby showing the different optical properties of the target area, allowing the observer to clearly see the image details. The binocular endoscope simulates the human binocular vision system, realizes the depth perception and three-dimensional reconstruction of the target area through the inspection principle and triangulation principle, and realizes the stereoscopic visual effect.

[0071] In this example, a fluorescent white-light endoscopic image fusion method is provided, which utilizes a single fluorescent binocular endoscope and a single white-light binocular endoscope, combined with camera calibration and robotic arm kinematics, to achieve registration and fusion of fluorescent images and white-light images in laparoscopic surgical robot surgery, that is, a method for fusing endoscopic fluorescent images and white-light images in the above-mentioned surgical robot application scenario is provided, by designing a set of endoscopic camera automatic calibration devices, and combining the knowledge of camera calibration, robotic arm kinematics, etc., to correct and align the images of the fluorescent endoscope and the white-light endoscope, so that the fluorescent diseased tissue is displayed on the high-definition color endoscopic image in a color-coding-based manner, so that the doctor can observe the lesion and surrounding tissue at the same time, thereby better guiding diagnosis and surgery.

[0072] Specifically, the fluorescent white light endoscope image fusion method may include the following steps:

[0073] Step 1: Collect the fluorescence image of the target tissue through a fluorescent binocular endoscope, and collect the white light image of the target tissue through a white light binocular endoscope;

[0074] Step 2: Segment the lesion tissue from the fluorescence image according to the fluorescence characteristics and obtain the pixel coordinates of the lesion;

[0075] Step 3: Calculate the three-dimensional coordinates of the lesion in the fluorescence endoscope camera coordinate system;

[0076] Step 4: Calculate the three-dimensional coordinates of the lesion tissue under the white light endoscope based on the forward kinematics of the robotic arm;

[0077] Step 5: Reproject the lesion onto the white light image plane according to the three-dimensional coordinates of the lesion tissue under the white light endoscope;

[0078] Step 6: Render the reprojected image as the target fluorescent color fused onto the white light image;

[0079] Step 7: Output the final fused image with lesion annotation for the doctor's reference.

[0080] Specifically, the registration and superposition of the fluorescent binocular endoscope image and the white light binocular endoscope image are performed through camera calibration and robot arm kinematics, which may include the following steps:

[0081] Step 1: Calibrate the fluorescence endoscope and the white light endoscope through the endoscope automatic calibration device to obtain the camera internal and external parameters of the fluorescence endoscope and the camera internal and external parameters of the white light endoscope, where the camera's internal parameters are parameters related to the camera's own characteristics, that is, the parameters inside the camera, such as: the camera's focal length, pixel size, etc.; the camera's external parameters determine the camera's position and orientation in a certain three-dimensional space, and the camera's external parameters are parameters in the world coordinate system, such as: the camera's position, rotation direction, etc.

[0082] Among them, the endoscope automatic calibration platform can be Figure 5 The device shown includes: a binocular endoscope 201, an endoscope fixing and pitch angle adjustment device 202, an electric screw module 203, a pan-tilt fixing base 204, a three-dimensional rotating electric pan-tilt 205, and a checkerboard calibration plate 206. The platform can control the calibration plate to move and rotate according to the calculated posture nodes through the electric screw module 203 and the three-dimensional rotating electric pan-tilt 205, wherein the control motor has an encoder to measure the rotation angle of the motor. The endoscope fixing and pitch angle adjustment device 202 can cooperate to adjust the angle of the endoscope lens relative to the calibration plate, so as to be more widely applicable to the calibration of endoscopes with different lens angles.

[0083] The above-mentioned endoscope automatic calibration platform can also be used as follows Figure 6 As shown, the endoscope pitch angle adjustment device 202 is replaced with a rotary motor with an encoder, and the above-mentioned endoscope automatic calibration platform can also be as shown in FIG. Figure 7 As shown, the lead screw linear motion device 203 is replaced with a wire wheel drive. The specific structure can be selected according to actual needs and conditions, and this application does not limit this.

[0084] The above-mentioned mirror pitch angle adjustment module 202 can be as follows Figure 8 As shown, it includes: a sleeve 21 and a base 22 for fixing the endoscope, and an electric push rod 23 for adjusting the pitch angle. After the endoscope is fixed, the electric push rod can be extended and retracted according to the lens angle of the fixed endoscope, so as to adjust the angle of the endoscope lens relative to the calibration plate, so as to ensure that the endoscope at different angles can completely capture the image of the calibration plate.

[0085] The above-mentioned electric three-dimensional rotating platform 205 can be as follows Fig. 9 As shown, it includes: a pan-tilt base 24, motors 25.a, 25.b, 25.c with angle encoders, and a calibration plate fixing base 26. After the calibration plate is fixed, the pan-tilt moves the calibration plate to different postures under the drive of the motor, triggers the endoscope to capture images at different posture nodes, and records the postures between different nodes. In this way, the problem of low accuracy in manually moving the calibration plate is avoided, the uniformity of the posture distribution of the calibration plate in space and the consistency between different calibrations are improved, and the automation of calibration is realized.

[0086] The 3D gimbal can change its position as follows: Fig.10 As shown, it includes: a calibration plate coordinate system 31 described by a calibration pose node 1, a calibration plate coordinate system 32 described by a calibration pose node 2, a pose transformation 33 between different calibration plate coordinate systems, and different pose nodes of the calibration plate during the calibration process. When the three-dimensional gimbal rotates the calibration plate to a node each time, the endoscope is triggered to capture an image of the calibration plate and record the poses of different nodes.

[0087] Based on the above endoscope automatic calibration platform, Fig.11 As shown, the automatic calibration of the endoscope is performed as follows:

[0088] S1: Fix the binocular endoscope to be calibrated in the endoscope automatic calibration platform;

[0089] S2: Input the end angle of the endoscope, lens manufacturing focal length and other product parameters;

[0090] S3: designing several pose nodes of the calibration board during the calibration process according to the parameters;

[0091] S4: The distance between the calibration plate and the lens is controlled by the electric lead screw module, and the posture of the calibration plate relative to the lens is controlled by the three-dimensional rotating electric pan-tilt head;

[0092] S5: After the calibration plate moves to the planned pose node, the endoscope is controlled to collect an image of the calibration plate;

[0093] S6: Extract features from the acquired images and optimize them using optimization algorithms such as Zhang’s camera correction;

[0094] S7: Output the camera intrinsic parameters of the endoscope lens and the external parameters for different calibration plate postures.

[0095] Furthermore, whether the calibration is passed can be determined in one of the following ways:

[0096] Method 1: Determine whether the calibration is successful based on the calculated camera extrinsic parameters Tc and the calibration plate Tb pose. Specifically, the relative transformation can be calculated according to the following formula:

[0097]

[0098]

[0099] Calculate the average error:

[0100]

[0101] Determine the mean error Is it greater than a given threshold? If it is greater than or equal to the given threshold, it is determined that recalibration is required; if it is less than the given threshold, it is determined that the calibration has passed.

[0102] Method 2: Input the internal and external parameters of the calibrated dual-lens endoscope, calculate and decompose the essential matrix based on the internal and external parameters of the dual-lens endoscope, and perform epipolar constraint calibration; project the three-dimensional grid points of the calibration plate onto the image plane of the left and right dual cameras, and calculate the reprojection error; if the error is less than the set threshold, the calibration is determined to be successful.

[0103] After epipolar correction based on method 2, the schematic diagram of the binocular camera can be as follows Fig.12 As shown, it may include: a left virtual image plane 409 and a right virtual image plane 410. The purpose of the binocular camera epipolar correction is to align the images of the left and right cameras on the same plane, so that when performing stereo vision processing, the same three-dimensional point can find a matching point in the same row of pixels on the left and right image planes, which simplifies the feature point matching process and improves the matching accuracy and efficiency.

[0104] Step 2: Determine the relative position of the fluorescent endoscope camera, the white light endoscope camera coordinate system and the end of the laparoscopic surgery robot through the robot hand-eye calibration;

[0105] Among them, the coordinate system of the binocular endoscope can be as follows Fig.13 As shown, it includes: endoscope left camera coordinate system 401, endoscope right camera coordinate system 402, left camera image plane 403, right camera image plane 404, left image coordinate system 405, right camera coordinate system 406, and world coordinate system 407. Based on the coordinate system, the relationship between the world coordinate system and the pixels of the left and right cameras can be derived, thereby realizing the target depth perception based on the parallax principle and the triangulation principle.

[0106] Step 3: Through the forward kinematics of the laparoscopic surgical robot, the relative position between the camera coordinate systems of the fluorescent endoscope and the white light endoscope is derived.

[0107] The forward kinematics of the robot arm can be expressed as Fig.14 As shown, in Fig.14 The figure shows the position transformation relationship between the robot base coordinate system 411 and the endoscope camera coordinate system 412. The forward kinematics equation of the tool arm is constructed by combining the design parameters of the robot arm and the DH calibration. The forward kinematics equation is updated by the angle feedback of each joint motor, and the robot arm is mapped from the joint space to the end Cartesian space, so as to obtain the position of the endoscope camera relative to the robot base in real time. Combined with the depth perception of the binoculars, the position of the target tissue relative to the robot base can be obtained.

[0108] Specifically, in this example, a schematic diagram of a dual endoscope operation is provided, such as Fig.15 As shown, it includes: an operating table 413, a lesion tissue 414, a fluorescent endoscope 415, a white light endoscope 416, and a transformation process of the image of the lesion tissue under the fluorescent endoscope to the image of the white light endoscope. At a certain moment, the fluorescent binocular endoscope captures the image of the target tissue, separates the lesion from the image background by the fluorescent characteristics of the lesion, and obtains the depth information of the lesion. Combined with the forward kinematics of the manipulator, the position relationship of the lesion relative to the base of the manipulator is calculated, and then the position relationship of the lesion relative to the coordinate system of the white light endoscope camera is obtained and projected to the left and right image planes. At the same time, the white light binocular endoscope captures the target tissue image, and then the two groups of left and right images are superimposed and displayed respectively, so as to realize the fusion of the fluorescent image and the white light image.

[0109] In the above example, the existing single fluorescent binocular endoscope and single white light binocular endoscope are used to provide clear color images of diseased tissue without replacing them with fluorescent white light integrated endoscopes, which improves the scope of use and use effect of the original equipment. The depth image of the lesion is captured by the fluorescent endoscope, and the depth image of the lesion is transformed into the camera coordinate system of the white light endoscope, and then reprojected to the left and right image planes of the white light endoscope, and the reprojected image is rendered as the target fluorescent color and fused on the white light image, so that the fluorescent diseased tissue is displayed on the high-definition color image to provide doctors with clearer and more intuitive information. Specifically, by designing an automatic calibration device for endoscope cameras, fully automatic calibration of endoscopes with different lens angles is achieved, and the efficiency and consistency of endoscope calibration are improved. By combining camera calibration with the kinematics knowledge of the robotic arm, accurate correction and registration of endoscopic images are achieved. Under the premise of accurate camera calibration and robotic arm kinematic model, image registration of fluorescent binocular endoscopes and white light binocular endoscopes can be achieved, which improves the accuracy of image fusion. Moreover, the computational complexity of image registration is less than that of existing image registration methods, and the registration results can provide a good initial value for image registration based on optimization methods such as gradient descent, thereby improving the final image registration accuracy.

[0110] Fig.16 It is a method flow chart of an embodiment of the fluorescent white light endoscopic image fusion method provided by the present application. Although the present application provides method operation steps or device structures as shown in the following embodiments or drawings, more or fewer operation steps or module units may be included in the method or device based on routine or no creative labor. In the steps or structures where there is no necessary causal relationship logically, the execution order of these steps or the module structure of the device is not limited to the execution order or module structure described in the embodiments of the present application and shown in the drawings. When the method or module structure is applied to an actual device or terminal product, it can be connected according to the method or module structure shown in the embodiment or drawings for sequential execution or parallel execution (for example, a parallel processor or a multi-threaded processing environment, or even a distributed processing environment).

[0111] Specifically, Fig.16 As shown, the above-mentioned fluorescent white light endoscope image fusion method can be applied to an operating robot, wherein the operating robot comprises: a first mechanical arm and a second mechanical arm, wherein the first mechanical arm is provided with a fluorescent binocular endoscope, and the second mechanical arm is provided with a white light binocular endoscope, and may include the following steps:

[0112] Step 1601: collecting a fluorescent image of the target tissue through a fluorescent binocular endoscope, and collecting a white light image of the target tissue through a white light binocular endoscope;

[0113] Step 1602: Separate the target tissue from the fluorescent image, and calculate the three-dimensional coordinates of the target tissue in the coordinate system associated with the fluorescent binocular endoscope;

[0114] Step 1603: According to the forward kinematics of the robot arm, convert the three-dimensional coordinates of the target tissue in the coordinate system associated with the fluorescent binocular endoscope into the three-dimensional coordinates of the target tissue in the coordinate system associated with the white light binocular endoscope;

[0115] Specifically, according to the forward kinematics of the manipulator, converting the three-dimensional coordinates of the target tissue in the coordinate system associated with the fluorescent binocular endoscope into the three-dimensional coordinates of the target tissue in the coordinate system associated with the white light binocular endoscope may include:

[0116] S1: determining the three-dimensional coordinates of the target tissue in the coordinate system associated with the fluorescent binocular endoscope according to the depth information of the target tissue in the coordinate system associated with the fluorescent binocular endoscope;

[0117] S2: determining the position and posture relationship of the target tissue relative to the end of the robotic arm corresponding to the fluorescent binocular endoscope according to the three-dimensional coordinates of the target tissue in the coordinate system associated with the fluorescent binocular endoscope;

[0118] S3: converting the position and posture relationship of the target tissue relative to the end of the robotic arm corresponding to the fluorescent binocular endoscope into the position and posture relationship of the target tissue relative to the end of the robotic arm corresponding to the white light binocular endoscope based on the forward kinematics of the robotic arm;

[0119] S4: The position relationship of the target tissue relative to the end of the robotic arm corresponding to the white light binocular endoscope is determined to determine the three-dimensional coordinates of the target tissue in the camera coordinate system of the white light binocular endoscope.

[0120] That is, the conversion is made from the fluorescent binocular endoscope to the robotic arm end corresponding to the fluorescent binocular endoscope, and then from the robotic arm end corresponding to the fluorescent binocular endoscope to the robotic arm end corresponding to the white light binocular endoscope, and then, from the robotic arm end corresponding to the white light binocular endoscope to the white light binocular endoscope, thereby obtaining the three-dimensional coordinates of the target tissue in the camera coordinate system of the white light binocular endoscope to achieve remapping.

[0121] Step 1604: reprojecting the target tissue onto an image plane of the white light binocular endoscope according to the three-dimensional coordinates of the target tissue in the coordinate system associated with the white light binocular endoscope to obtain a reprojected image;

[0122] When the target tissue is reprojected onto the image plane of the white light binocular endoscope according to the three-dimensional coordinates of the target tissue in the coordinate system associated with the white light binocular endoscope to obtain the reprojected image, the position and posture of the target tissue relative to the white light binocular endoscope can be determined according to the three-dimensional coordinates of the target tissue in the coordinate system associated with the white light binocular endoscope and the predetermined camera calibration result; according to the position and posture of the target tissue relative to the white light binocular endoscope, the target tissue is reprojected onto the image plane of the white light binocular endoscope to obtain the reprojected image.

[0123] Because in this example, both the white light endoscope and the fluorescent endoscope are binocular endoscopes, and therefore, both have a left image plane and a right image plane. Therefore, when the target tissue is reprojected onto the image plane of the white light binocular endoscope according to the three-dimensional coordinates of the target tissue in the coordinate system associated with the white light binocular endoscope to obtain the reprojected image, the target tissue can be reprojected onto the left image plane of the white light binocular endoscope according to the three-dimensional coordinates of the target tissue in the coordinate system associated with the left camera of the white light binocular endoscope to obtain the left image; and the target tissue can be reprojected onto the right image plane of the white light binocular endoscope according to the three-dimensional coordinates of the target tissue in the coordinate system associated with the right camera of the white light binocular endoscope to obtain the right image, wherein the left image and the right image are used to present a stereoscopic image effect to an operator (e.g., a doctor).

[0124] Step 1605: superimpose the reprojected image and the white light image to obtain a target image.

[0125] Specifically, the reprojected image and the white light image are superimposed and displayed to obtain the target image, which may include: rendering the target tissue in the reprojected image as a preset color; and fusing the rendered image with the white light image to obtain the final target image. For example, the target tissue rendering color may be green, so the reprojected image is rendered as fluorescent green and then fused with the white light image.

[0126] For the above-mentioned white light binocular endoscope and fluorescent binocular endoscope, camera calibration is required to convert and correspond the time pose and coordinate system. Specifically, the camera calibration can be performed as follows:

[0127] S1: Fix the target binocular endoscope through the endoscope fixing platform;

[0128] S2: Input the end angle and focal length of the target binocular endoscope;

[0129] S3: controlling the distance and posture of the calibration plate relative to the lens of the target binocular endoscope according to a plurality of preset posture points;

[0130] S4: When the calibration plate reaches a distance and a posture relative to the lens of the target binocular endoscope and reaches a plurality of preset posture points, controlling the target binocular endoscope to collect an image of the calibration plate;

[0131] S5: extracting features from the collected image of the calibration plate, and obtaining external parameters of the target binocular endoscope based on the extracted features to achieve camera calibration.

[0132] Furthermore, considering that a single calibration may not be accurate and successful during camera calibration, after the external parameters of the target binocular endoscope are obtained based on the extracted features to realize camera calibration, multiple postures of the calibration plate and the external parameters corresponding to each posture can be obtained; the relative transformation of the posture and the relative transformation of the external parameters are calculated; when the difference between the relative transformation of the external parameters and the relative transformation of the posture is less than a preset threshold, it is determined that the calibration is passed; when the difference between the relative transformation of the external parameters and the relative transformation of the posture is not less than a preset threshold, recalibration is performed.

[0133] The above-mentioned fluorescent white light endoscope image fusion method can be applied to a surgical robot equipped with a single fluorescent binocular endoscope and a single white light binocular endoscope, thereby providing doctors with clearer images and operation support during surgery.

[0134] In the above example, a fluorescence and white light binocular endoscope image registration and fusion method combining camera calibration and robot arm kinematics knowledge is provided, which improves the efficiency of registration. The accuracy of image registration is ensured by accurate camera calibration and accurate robot arm positive kinematics, and the fusion of fluorescence images and white light images is realized. The cost of the equipment is reduced, the scope of use of the original equipment is expanded, and a good initial value is provided for the subsequent image registration based on iterative optimization such as gradient descent, thereby improving the final image registration accuracy and reducing the number of iterations.

[0135] The method embodiments provided in the above embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on an electronic device as an example, Fig.17 This is a hardware structure block diagram of an electronic device for a fluorescent white light endoscope image fusion method provided by the present application. Fig.17 As shown, the electronic device may include one or more (only one is shown in the figure) processors 02 (the processor 02 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 04 for storing data, and a transmission module 06 for communication functions. It can be understood by those skilled in the art that Fig.17The structure shown is only for illustration and does not limit the structure of the above electronic device. Fig.17 More or fewer components as shown, or with Fig.17 Different configurations are shown.

[0136] The memory 04 can be used to store software programs and modules of application software, such as program instructions / modules corresponding to the fluorescent white light endoscopic image fusion method in the embodiment of the present application. The processor 02 executes various functional applications and data processing by running the software programs and modules stored in the memory 04, that is, the fluorescent white light endoscopic image fusion method of the above-mentioned application is realized. The memory 04 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 04 may further include a memory remotely arranged relative to the processor 02, and these remote memories can be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0137] The transmission module 06 is used to receive or send data via a network. The above-mentioned network specific example may include a wireless network provided by a communication provider of the electronic device. In one example, the transmission module 06 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission module 06 can be a radio frequency (Radio Frequency, RF) module, which is used to communicate with the Internet wirelessly.

[0138] At the software level, the fluorescent white light endoscope image fusion device is applied to an operating robot, the operating robot comprises: a first mechanical arm and a second mechanical arm, the first mechanical arm is provided with a fluorescent binocular endoscope, the second mechanical arm is provided with a white light binocular endoscope, and the device can be as follows: Fig.18 As shown, including:

[0139] An acquisition module 1801 is used to acquire a fluorescent image of a target tissue through a fluorescent binocular endoscope and to acquire a white light image of the target tissue through a white light binocular endoscope;

[0140] A separation module 1802, for separating the target tissue from the fluorescent image and calculating the three-dimensional coordinates of the target tissue in the coordinate system associated with the fluorescent binocular endoscope;

[0141] A conversion module 1803, for converting the three-dimensional coordinates of the target tissue in the coordinate system associated with the fluorescent binocular endoscope into the three-dimensional coordinates of the target tissue in the coordinate system associated with the white light binocular endoscope according to the forward kinematics of the robot arm;

[0142] A reprojection module 1804 is used to reproject the target tissue onto an image plane of the white light binocular endoscope according to the three-dimensional coordinates of the target tissue in the coordinate system associated with the white light binocular endoscope to obtain a reprojected image;

[0143] The superposition module 1805 is used to superimpose the reprojected image and the white light image to obtain a target image.

[0144] In one embodiment, the above-mentioned conversion module 1803 can specifically determine the three-dimensional coordinates of the target tissue in the coordinate system associated with the fluorescent binocular endoscope based on the depth information of the target tissue in the coordinate system associated with the fluorescent binocular endoscope; determine the posture relationship of the target tissue relative to the end of the robotic arm corresponding to the fluorescent binocular endoscope based on the three-dimensional coordinates of the target tissue in the coordinate system associated with the fluorescent binocular endoscope; convert the posture relationship of the target tissue relative to the end of the robotic arm corresponding to the fluorescent binocular endoscope into the posture relationship of the target tissue relative to the end of the robotic arm corresponding to the white light binocular endoscope based on the positive kinematics of the robotic arm; determine the three-dimensional coordinates of the target tissue in the camera coordinate system of the white light binocular endoscope based on the posture relationship of the target tissue relative to the end of the robotic arm corresponding to the white light binocular endoscope.

[0145] In one embodiment, the above-mentioned reprojection module 1804 can specifically determine the position of the target tissue relative to the white light binocular endoscope according to the three-dimensional coordinates of the target tissue in the coordinate system associated with the white light binocular endoscope and the predetermined camera calibration result; according to the position of the target tissue relative to the white light binocular endoscope, reproject the target tissue onto the image plane of the white light binocular endoscope to obtain a reprojected image.

[0146] In one embodiment, camera calibration can be performed in the following manner: fix the target binocular endoscope through an endoscope fixing platform; input the end angle and focal length of the target binocular endoscope; control the distance and posture of the calibration plate relative to the lens of the target binocular endoscope according to a plurality of preset posture points; when the distance and posture of the calibration plate relative to the lens of the target binocular endoscope reach a plurality of preset posture points, control the target binocular endoscope to capture an image of the calibration plate; perform feature extraction on the captured image of the calibration plate, and obtain the external parameters of the target binocular endoscope based on the extracted features to achieve camera calibration.

[0147] In one embodiment, after the external parameters of the target binocular endoscope are obtained based on the extracted features to realize camera calibration, multiple postures of the calibration plate and the camera external parameters corresponding to each posture can also be obtained; the relative transformation of the posture and the relative transformation of the external parameters are calculated; when the difference between the relative transformation of the external parameters and the relative transformation of the posture is less than a preset threshold, it is determined that the calibration is passed; when the difference between the relative transformation of the external parameters and the relative transformation of the posture is not less than a preset threshold, re-calibration is performed.

[0148] In one embodiment, the reprojection module 1804 can specifically reproject the target tissue onto the left image plane of the white light binocular endoscope according to the three-dimensional coordinates of the target tissue in the coordinate system associated with the left camera of the white light binocular endoscope to obtain a left image; and reproject the target tissue onto the right image plane of the white light binocular endoscope according to the three-dimensional coordinates of the target tissue in the coordinate system associated with the right camera of the white light binocular endoscope to obtain a right image, wherein the left image and the right image are used to present a stereoscopic image effect.

[0149] In one embodiment, the overlay module 1805 may render the target tissue in the reprojected image as a preset fluorescent color; and fuse the image rendered with a slight preset fluorescent color with the white light image to obtain a target image.

[0150] In one embodiment, the above-mentioned device can be applied to a surgical robot provided with a single fluorescent binocular endoscope and a single white light binocular endoscope.

[0151] The embodiments of the present application also provide a specific implementation of an electronic device capable of implementing all the steps in the fluorescent white light endoscopic image fusion method in the above embodiments, wherein the electronic device specifically includes the following contents: a processor, a memory, a communication interface, and a bus; wherein the processor, the memory, and the communication interface communicate with each other through the bus; the processor is used to call a computer program in the memory, and when the processor executes the computer program, all the steps in the fluorescent white light endoscopic image fusion method in the above embodiments are implemented. For example, when the processor executes the computer program, the following steps are implemented:

[0152] Step 1: Collect the fluorescence image of the target tissue through a fluorescent binocular endoscope, and collect the white light image of the target tissue through a white light binocular endoscope;

[0153] Step 2: separating the target tissue from the fluorescent image, and calculating the three-dimensional coordinates of the target tissue in the coordinate system associated with the fluorescent binocular endoscope;

[0154] Step 3: According to the forward kinematics of the robot arm, the three-dimensional coordinates of the target tissue in the coordinate system associated with the fluorescent binocular endoscope are converted into the three-dimensional coordinates of the target tissue in the coordinate system associated with the white light binocular endoscope;

[0155] Step 4: reprojecting the target tissue onto an image plane of the white light binocular endoscope according to the three-dimensional coordinates of the target tissue in the coordinate system associated with the white light binocular endoscope to obtain a reprojected image;

[0156] Step 5: Overlay the reprojected image and the white light image to obtain a target image.

[0157] The embodiments of the present application also provide a computer-readable storage medium capable of implementing all the steps in the fluorescent white light endoscopic image fusion method in the above embodiments. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, all the steps in the fluorescent white light endoscopic image fusion method in the above embodiments are implemented. For example, when the processor executes the computer program, the following steps are implemented:

[0158] Step 1: Collect the fluorescence image of the target tissue through a fluorescent binocular endoscope, and collect the white light image of the target tissue through a white light binocular endoscope;

[0159] Step 2: separating the target tissue from the fluorescent image, and calculating the three-dimensional coordinates of the target tissue in the coordinate system associated with the fluorescent binocular endoscope;

[0160] Step 3: According to the forward kinematics of the robot arm, the three-dimensional coordinates of the target tissue in the coordinate system associated with the fluorescent binocular endoscope are converted into the three-dimensional coordinates of the target tissue in the coordinate system associated with the white light binocular endoscope;

[0161] Step 4: reprojecting the target tissue onto an image plane of the white light binocular endoscope according to the three-dimensional coordinates of the target tissue in the coordinate system associated with the white light binocular endoscope to obtain a reprojected image;

[0162] Step 5: Overlay the reprojected image and the white light image to obtain a target image.

[0163] From the above description, it can be seen that the embodiment of the present application collects the fluorescent image of the target tissue through a fluorescent binocular endoscope, and collects the white light image of the target tissue through a white light binocular endoscope; then, the target tissue can be separated from the fluorescent image, and the three-dimensional coordinates of the target tissue in the coordinate system associated with the fluorescent binocular endoscope are calculated; then, according to the positive kinematics of the manipulator, the three-dimensional coordinates of the target tissue in the coordinate system associated with the fluorescent binocular endoscope are converted into the three-dimensional coordinates of the target tissue in the coordinate system associated with the white light binocular endoscope, and then the target tissue is reprojected onto the image plane of the white light binocular endoscope to obtain a reprojected image, and the reprojected image and the white light image are superimposed and displayed to obtain the target image. Through the above scheme, based on the conversion between the coordinate systems calibrated by the camera and the positive kinematics of the manipulator, the fusion of the fluorescent white light endoscope images is performed, which solves the problems of high cost and low accuracy in the existing image registration, and achieves the technical effect of efficient and low-cost image registration.

[0164] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the hardware + program embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0165] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0166] Although the present application provides method operation steps as described in the embodiments or flowcharts, more or fewer operation steps may be included based on conventional or non-creative labor. The order of steps listed in the embodiments is only one way of executing the order of many steps and does not represent the only execution order. When the actual device or client product is executed, it can be executed in the order of the method shown in the embodiments or the drawings or in parallel (for example, in a parallel processor or multi-threaded processing environment).

[0167] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a vehicle-mounted human-computer interaction device, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0168] Although the present specification embodiment provides the method operation steps as described in the embodiment or flow chart, more or less operation steps may be included based on conventional or non-creative means. The order of steps listed in the embodiment is only one way in the order of execution of many steps, and does not represent a unique execution order. When the device or terminal product in practice is executed, it can be executed in sequence or in parallel (such as a parallel processor or a multi-threaded processing environment, or even a distributed data processing environment) according to the method shown in the embodiment or the accompanying drawings. The term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, product or equipment including a series of elements not only includes those elements, but also includes other elements not clearly listed, or also includes elements inherent to such process, method, product or equipment. In the absence of more restrictions, it is not excluded that there are other identical or equivalent elements in the process, method, product or equipment including the elements.

[0169] For the convenience of description, the above devices are described in various modules according to their functions. Of course, when implementing the embodiments of this specification, the functions of each module can be implemented in the same or more software and / or hardware, or the module implementing the same function can be implemented by a combination of multiple sub-modules or sub-units. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0170] Those skilled in the art also know that, in addition to implementing the controller in a purely computer-readable program code, the controller can be made to implement the same function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered as a hardware component, and the devices for implementing various functions included therein can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules for implementing the method and structures within the hardware component.

[0171] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0172] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0173] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0174] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0175] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0176] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0177] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, the embodiments of this specification may take the form of complete hardware embodiments, complete software embodiments or embodiments combining software and hardware. Moreover, the embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0178] The present specification embodiments may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present specification embodiments may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0179] Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. In the description of this specification, the description of the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiment of this specification. In this specification, the schematic representation of the above terms does not necessarily target the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, in the absence of contradiction, a person skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0180] The above is only an example of the embodiment of the present specification and is not intended to limit the embodiment of the present specification. For those skilled in the art, the embodiment of the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the embodiment of the present specification shall be included in the scope of the claims of the embodiment of the present specification.

Claims

1. An endoscopic image fusion method, characterized in that: Applied in an operating robot, the operating robot comprises: a first mechanical arm and a second mechanical arm, the first mechanical arm is provided with a fluorescent binocular endoscope, the second mechanical arm is provided with a white light binocular endoscope, the method comprises: Acquire a fluorescent image of the target tissue by a fluorescent binocular endoscope, and acquire a white light image of the target tissue by a white light binocular endoscope; Separating the target tissue from the fluorescent image and calculating the three-dimensional coordinates of the target tissue in a coordinate system associated with the fluorescent binocular endoscope; According to the forward kinematics of the robot arm, the three-dimensional coordinates of the target tissue in the coordinate system associated with the fluorescent binocular endoscope are converted into the three-dimensional coordinates of the target tissue in the coordinate system associated with the white light binocular endoscope; Reprojecting the target tissue onto an image plane of the white-light binocular endoscope according to the three-dimensional coordinates of the target tissue in a coordinate system associated with the white-light binocular endoscope to obtain a reprojected image; The reprojected image and the white light image are superimposed and displayed to obtain a target image.

2. The method according to claim 1, characterized in that According to the forward kinematics of the manipulator, the three-dimensional coordinates of the target tissue in the coordinate system associated with the fluorescent binocular endoscope are converted into the three-dimensional coordinates of the target tissue in the coordinate system associated with the white light binocular endoscope, including: Determine the three-dimensional coordinates of the target tissue in the coordinate system associated with the fluorescent binocular endoscope according to the depth information of the target tissue in the coordinate system associated with the fluorescent binocular endoscope; Determine, according to the three-dimensional coordinates of the target tissue in the coordinate system associated with the fluorescent binocular endoscope, the position and posture relationship of the target tissue relative to the end of the robotic arm corresponding to the fluorescent binocular endoscope; According to the position and posture relationship of the target tissue relative to the end of the robotic arm corresponding to the fluorescent binocular endoscope, based on the forward kinematics of the robotic arm, converting it into the position and posture relationship of the target tissue relative to the end of the robotic arm corresponding to the white light binocular endoscope; The position relationship of the target tissue relative to the end of the robotic arm corresponding to the white light binocular endoscope is used to determine the three-dimensional coordinates of the target tissue in the camera coordinate system of the white light binocular endoscope.

3. The method according to claim 1, characterized in that: According to the three-dimensional coordinates of the target tissue in the coordinate system associated with the white light binocular endoscope, the target tissue is reprojected onto the image plane of the white light binocular endoscope to obtain a reprojected image, including: According to the three-dimensional coordinates of the target tissue in the coordinate system associated with the white light binocular endoscope and in accordance with the pre-calibrated camera result, the position and posture of the target tissue relative to the white light binocular endoscope is determined; According to the position of the target tissue relative to the white light binocular endoscope, the target tissue is reprojected onto the image plane of the white light binocular endoscope to obtain a reprojected image.

4. The method according to claim 3, characterized in that: Perform camera calibration as follows: Fixing the target binocular endoscope via an endoscope fixing platform; Input the end angle and focal length of the target binocular endoscope; According to a plurality of preset posture points, controlling the distance and posture of the calibration plate relative to the lens of the target binocular endoscope; When the distance and posture of the calibration plate relative to the lens of the target binocular endoscope reach a plurality of preset posture points, controlling the target binocular endoscope to collect an image of the calibration plate; Feature extraction is performed on the collected image of the calibration plate, and the external parameters of the target binocular endoscope are obtained according to the extracted features to achieve camera calibration.

5. The method according to claim 4, characterized in that After obtaining the external parameters of the target binocular endoscope according to the extracted features to realize camera calibration, the method further includes: Obtain multiple poses of the calibration plate and the camera extrinsics corresponding to each pose; Calculate the relative transformation of posture and the relative transformation of external parameters; When the difference between the relative transformation of the external parameters and the relative transformation of the posture is less than a preset threshold, it is determined that the calibration is passed; When the difference between the relative transformation of the external parameters and the relative transformation of the posture is not less than a preset threshold, recalibration is performed.

6. The method according to claim 1, characterized in that According to the three-dimensional coordinates of the target tissue in the coordinate system associated with the white light binocular endoscope, the target tissue is reprojected onto the image plane of the white light binocular endoscope to obtain a reprojected image, including: Reprojecting the target tissue onto a left image plane of the white light binocular endoscope according to the three-dimensional coordinates of the target tissue in the coordinate system associated with the left camera of the white light binocular endoscope to obtain a left image; According to the three-dimensional coordinates of the target tissue in the coordinate system associated with the right camera of the white light binocular endoscope, the target tissue is reprojected onto the right image plane of the white light binocular endoscope to obtain a right image, wherein the left image and the right image are used to present a stereoscopic image effect.

7. The method according to claim 1, characterized in that The reprojected image and the white light image are superimposed and displayed to obtain a target image, comprising: Rendering the target tissue in the reprojected image into a preset color; The rendered image is fused with the white light image to obtain a target image.

8. The method according to any one of claims 1 to 7, characterized in that It is applied to a surgical robot provided with a single fluorescent binocular endoscope and a single white light binocular endoscope.

9. An electronic device comprising a processor and a memory for storing instructions executable by the processor, characterized in that: When the processor executes the instructions, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images