Real-time Geodesy Method, Device, Electronic Device and Storage Medium for 3D Endoscope

Abstract

The present invention relates to the technical field of endoscopes, and specifically discloses a real-time geodesic measurement method, device, electronic device and storage medium for a 3D endoscope. The method includes the steps of: acquiring a left-eye image and a right-eye image captured by the 3D endoscope; preprocessing the left-eye image and the right-eye image, and then performing stereo matching to obtain a set of matching point clouds; determining a reference point cloud in the set of matching point clouds based on the position of the geodesic center in the left-eye image or the right-eye image; generating a geodesic on the display screens corresponding to the left-eye image and the right-eye image according to the set of matching point clouds, the reference point cloud, and a target geodesic distance based on a heat propagation algorithm; this method can quickly generate a geodesic without affecting the display of the picture, and in the process of generation, downsampling processing is used to preprocess the left-eye image and the right-eye image, which can reduce the amount of data processing without affecting the extraction of key features, and improve the display synchronization of the image and the geodesic in the display picture.

Description

Technical Field

[0001] The present application relates to the technical field of endoscopes, and in particular, to a real-time geodesic measurement method, device, electronic device, and storage medium for a 3D endoscope. Background Art

[0002] In recent years, as an advanced medical imaging device, the 3D endoscope has been significantly popularized in hospitals. This technology relies on a binocular camera to simulate the human eye and creates a three-dimensional surgical field of view, significantly improving the surgical precision and safety. Compared with the 2D endoscope, the 3D endoscope more accurately displays the tissue depth and spatial relationship, enhancing the doctor's operation control effect.

[0003] In surgical practice, especially for liver and lung lesion resection, it is necessary to calculate the surface distance (geodesic distance) in real time through a 3D endoscope to achieve precise resection planning and balance complete resection and reduced bleeding risk. However, the current real-time geodesic distance calculation system faces computational challenges brought by high-definition image processing. There is a problem of a large amount of data analysis for real-time geodesic measurement of high-definition images, resulting in a large delay in the geodesic measurement result and making it difficult to ensure real-time performance.

[0004] In response to the above problems, there is currently no effective technical solution. Summary of the Invention

[0005] The purpose of the present application is to provide a real-time geodesic measurement method, device, electronic device, and storage medium for a 3D endoscope to reduce the delay of geodesic calculation and ensure the real-time performance of the geodesic measurement result.

[0006] In a first aspect, the present application provides a real-time geodesic measurement method for a 3D endoscope, which is applied in a 3D endoscope. The method includes the following steps:

[0007] S1. Obtain the left-eye image and right-eye image captured by the 3D endoscope;

[0008] S2. Preprocess the left-eye image and the right-eye image, and then perform stereo matching to obtain a set of matching point clouds. The preprocessing includes downsampling processing and distortion removal and alignment processing;

[0009] S3. Based on the position of the geodesic center in the left-eye image or the right-eye image, determine the reference point cloud in the set of matching point clouds;

[0010] S4. Based on the heat propagation algorithm, generate a geodesic on the display screens corresponding to the left-eye image and the right-eye image according to the set of matching point clouds, the reference point cloud, and the target geodesic distance.

[0011] The real-time geodesic measurement method of the 3D endoscope in this application is based on the preprocessed left-eye image and right-eye image for stereo matching to obtain a set of matching point clouds, and based on the heat propagation algorithm, suitable point clouds are selected from the set of matching point clouds to generate a geodesic on the display screen. It can complete the rapid generation of the geodesic without affecting the screen display, and the downsampling process is used for preprocessing the left-eye image and right-eye image during the generation process, which can reduce the amount of data processing without affecting the extraction of key features, greatly improving the efficiency of point cloud analysis, effectively improving the generation efficiency of the geodesic, and improving the display synchronization of the left-eye image, right-eye image and the geodesic in the display screen.

[0012] The real-time geodesic measurement method of the 3D endoscope described above, wherein the distortion removal and alignment process includes the following steps:

[0013] Remap the downsampled left-eye image and right-eye image based on the mapping matrix, and the mapping matrix is obtained based on binocular camera calibration.

[0014] In this example, the remapping process can map the pixels in the image from one position to another, and can achieve effects such as image translation, rotation, scaling, and perspective transformation, so that the downsampled left-eye image and right-eye image can be paired for use to facilitate stereo matching processing.

[0015] The real-time geodesic measurement method of the 3D endoscope described above, wherein the magnification of the downsampling process is 2-6 times.

[0016] In this example, the magnification of the downsampling process is preferably 4 times. The downsampled left-eye image and right-eye image can achieve stereo matching without affecting the extraction of key features, greatly reducing the amount of data analysis in the stereo matching process and also reducing the number of point clouds in the geodesic measurement calculation.

[0017] The real-time geodesic measurement method of the 3D endoscope described above, wherein the stereo matching includes the following steps:

[0018] Perform pixel matching on the preprocessed left-eye image and right-eye image, and obtain the coordinate deviation of each pixel to generate a disparity map;

[0019] According to the disparity map and the reprojection matrix, map the pixels in the distortion-removed and aligned left-eye image and / or right-eye image into three-dimensional space to obtain the three-dimensional point clouds corresponding to each pixel, and form the set of matching point clouds.

[0020] The real-time geodesic measurement method of the 3D endoscope described above, wherein the step of generating a geodesic on the display screen corresponding to the left-eye image and the right-eye image based on the heat propagation algorithm according to the set of matching point clouds, the reference point cloud, and the target geodesic distance includes:

[0021] Obtain target point clouds based on the heat propagation algorithm, where the target point clouds are the point clouds in the matching point cloud set with a geodesic distance to the reference point cloud equal to the target geodesic distance;

[0022] Obtain the two-dimensional coordinates of all target point clouds in the left-eye image and the right-eye image;

[0023] Generate geodesics on the display screens corresponding to the left-eye image and the right-eye image according to the two-dimensional coordinates.

[0024] The real-time geodesic measurement method for a 3D endoscope, wherein the step of obtaining target point clouds based on the heat propagation algorithm includes:

[0025] Calculate the geodesic distances from different point clouds in the matching point cloud set to the reference point cloud based on the heat propagation algorithm;

[0026] Filter out the point clouds in the matching point cloud set with a geodesic distance equal to the target geodesic distance as the target point clouds.

[0027] The real-time geodesic measurement method for a 3D endoscope, wherein the step of generating geodesics on the display screens corresponding to the left-eye image and the right-eye image according to the two-dimensional coordinates includes:

[0028] Create blank layers on the display screens corresponding to the left-eye image and the right-eye image respectively;

[0029] Assign the corresponding pixels on the blank layers a value of 1 based on the two-dimensional coordinates;

[0030] Display the pixels with a value of 1 in the blank layers based on a preset color.

[0031] In a second aspect, the present application also provides a real-time geodesic measurement device for a 3D endoscope, which is applied in a 3D endoscope. The device includes:

[0032] An image acquisition module, configured to acquire a left-eye image and a right-eye image captured by a 3D endoscope;

[0033] A point cloud reconstruction module, configured to preprocess the left-eye image and the right-eye image, and then perform stereo matching to obtain a matching point cloud set. The preprocessing includes downsampling processing and distortion removal and alignment processing;

[0034] A reference module, configured to determine a reference point cloud in the matching point cloud set based on the position of the geodesic center in the left-eye image or the right-eye image;

[0035] A geodesic module, configured to generate geodesics on the display screens corresponding to the left-eye image and the right-eye image based on the heat propagation algorithm according to the matching point cloud set, the reference point cloud, and the target geodesic distance.

[0036] The real-time geodesic measurement device of the 3D endoscope of the present application performs stereo matching on the preprocessed left-eye image and right-eye image to obtain a set of matching point clouds, and filters appropriate point clouds from the set of matching point clouds based on the heat propagation algorithm to generate a geodesic on the display screen. It can complete the rapid generation of the geodesic without affecting the screen display, and during the generation process, downsampling processing is used to preprocess the left-eye image and right-eye image, which can reduce the amount of data processing without affecting the extraction of key features, greatly improving the point cloud analysis efficiency, effectively improving the generation efficiency of the geodesic, and improving the display synchronization of the left-eye image, right-eye image and geodesic in the display screen.

[0037] In a third aspect, the present application further provides an electronic device, including a processor and a memory. The memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps in the method provided in the first aspect above are run.

[0038] In a fourth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in the method provided in the first aspect above are run.

[0039] As can be seen from the above, the present application provides a real-time geodesic measurement method, device, electronic device and storage medium for a 3D endoscope. Among them, the real-time geodesic measurement method of the 3D endoscope of the present application performs stereo matching on the preprocessed left-eye image and right-eye image to obtain a set of matching point clouds, and filters appropriate point clouds from the set of matching point clouds based on the heat propagation algorithm to generate a geodesic on the display screen. It can complete the rapid generation of the geodesic without affecting the screen display, and during the generation process, downsampling processing is used to preprocess the left-eye image and right-eye image, which can reduce the amount of data processing without affecting the extraction of key features, greatly improving the point cloud analysis efficiency, effectively improving the generation efficiency of the geodesic, and improving the display synchronization of the left-eye image, right-eye image and geodesic in the display screen. Description of the Drawings

[0040] Figure 1 It is a flowchart of the real-time geodesic measurement method for the 3D endoscope provided by the embodiment of the present application.

[0041] Figure 2 It is a schematic diagram of the display screen corresponding to the left-eye image at a certain moment when the real-time geodesic measurement method for the 3D endoscope provided by the embodiment of the present application is applied to the 3D endoscope.

[0042] Figure 3 It is a schematic diagram of the display screen corresponding to the left-eye image at another moment when the real-time geodesic measurement method for the 3D endoscope provided by the embodiment of the present application is applied to the 3D endoscope.

[0043] Figure 4 This is a schematic structural diagram of the real-time geodesic measurement device of the 3D endoscope provided by the embodiment of the present application.

[0044] Figure 5 This is a schematic structural diagram of the electronic device provided by the embodiment of the present application.

[0045] Reference numerals: 201, image acquisition module; 202, point cloud reconstruction module; 203, reference module; 204, geodesic module; 301, processor; 302, memory; 303, communication bus. Detailed implementation manners

[0046] Next, the technical solutions in the embodiments of the present application will be clearly and completely described with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Usually, the components of the embodiments of the present application described and shown in the accompanying drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application to be protected, but only represents the selected embodiments of the present application. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative efforts belong to the scope of protection of the present application.

[0047] It should be noted that: similar reference numerals and letters represent similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of the present application, the terms "first", "second", etc. are only used for distinguishing descriptions and cannot be understood as indicating or implying relative importance.

[0048] In a first aspect, please refer to Figure 1 , some embodiments of the present application provide a real-time geodesic measurement method for a 3D endoscope, which is applied to a 3D endoscope. The method includes the following steps:

[0049] S1. Obtain the left-eye image and the right-eye image captured by the 3D endoscope;

[0050] S2. Preprocess the left-eye image and the right-eye image, and then perform stereo matching to obtain a set of matching point clouds. The preprocessing includes downsampling processing and distortion removal and alignment processing;

[0051] S3. Determine the reference point cloud in the set of matching point clouds based on the position of the geodesic center in the left-eye image or the right-eye image;

[0052] S4. Based on the heat propagation algorithm, generate geodesics on the display screens corresponding to the left-eye image and the right-eye image according to the set of matching points, the reference point cloud, and the target geodesic distance.

[0053] Specifically, a 3D endoscope is a medical device integrating high-precision optics and imaging technology. It adopts a dual-camera design inside and can capture and synthesize high-definition three-dimensional stereoscopic images, providing a more real and intuitive surgical field of view for doctors. Among them, the images captured by the two cameras respectively correspond to the above-mentioned left-eye image and right-eye image.

[0054] More specifically, in the embodiment of the present application, the two cameras of the 3D endoscope continuously take pictures. The left-eye image and the right-eye image in step S1 are respectively the pictures extracted in real time based on the video streams continuously captured by the two cameras. The real-time geodesic measurement method of the 3D endoscope in the embodiment of the present application analyzes and obtains geodesics in real time based on these pictures, so as to output geodesics in real time on the display screen corresponding to the display video stream; among them, steps S1 - S4 are equivalent to the process of analyzing the frame pictures of the captured video stream and generating geodesics.

[0055] It should be noted that the real-time geodesic measurement method of the 3D endoscope in the embodiment of the present application can analyze each frame of the video stream captured by the 3D endoscope to generate and update geodesics, or can perform frame skipping analysis on the video stream captured by the 3D endoscope to generate and update geodesics. Among them, the former can more effectively ensure the real-time performance and accuracy of geodesics, and the latter can save system resources and reduce the operating load of the 3D endoscope.

[0056] More specifically, generally, the left-eye image and the right-eye image in the video stream captured by the 3D endoscope will be directly displayed on the display screen of the display device of the 3D endoscope. The real-time geodesic measurement method of the 3D endoscope in the embodiment of the present application extracts the left-eye image and the right-eye image while the 3D endoscope performs display processing, and generates geodesics superimposed on the left-eye image and the right-eye image displayed on the display screen based on steps S2 - S4. Among them, the generation process of geodesics is mainly a process of analyzing based on the set of matching points established in step S2, that is, the processing processes of steps S2 - S4 will not affect the pixel data in the left-eye image and the right-eye image originally displayed on the display screen.

[0057] More specifically, the left-eye image and the right-eye image are planar images with parallax and capable of matching to form a three-dimensional visual effect. Therefore, in step S2, stereo matching can be performed based on the feature pairing relationship of pixels in the preprocessed left-eye image and right-eye image to perform point cloud reconstruction, thereby obtaining a set of matching points. Among them, this stereo matching process can be carried out using existing stereo matching algorithms for three-dimensional reconstruction, such as SSD (Sum of Squared Differences), SAD (Sum of Absolute Differences), ZNCC (Zero-mean Normalized Cross-Correlation), BM (Block Matching Algorithm), SGBM (Semi-Global Block Matching) and other algorithms. Preferably, the BM or SGBM algorithm is used.

[0058] More specifically, the distortion removal and alignment process in the preprocessing can effectively eliminate the pixel position offset caused by distortion in the left-eye image and the right-eye image, can effectively improve the matching accuracy and consistency of subsequent stereo matching, and can simplify the search range of stereo matching to improve the matching speed.

[0059] More specifically, the downsampling process in the preprocessing can generate two low-resolution images to be analyzed (corresponding to the left-eye image and the right-eye image). These two low-resolution images to be analyzed are only used for stereo matching to reconstruct and obtain a set of matching points, and are not used for screen display; this downsampling process can effectively reduce the data dimension of subsequent processing and greatly improve the generation efficiency of subsequent geodesics.

[0060] More specifically, the magnification of the downsampling process is set according to system parameters (mainly depending on the resolution of the original left-eye image and right-eye image and the type of stereo matching algorithm). This downsampling process needs to be carried out without affecting the extraction of key features to reduce the amount of data analysis in subsequent processing, thereby optimizing the allocation and use of computing resources, improving the generation rate of geodesics, and improving the timeliness of geodesics on the display screen.

[0061] It should be noted that the video stream obtained by the camera of the 3D endoscope is directly transmitted to the display device for display on the corresponding display screen. The geodesic obtained by analyzing the real-time geodesic method of the 3D endoscope in the embodiment of the present application is also displayed on the corresponding display screen; the real-time geodesic method of the 3D endoscope in the embodiment of the present application is based on the downsampling process to reduce the amount of data processing. The method of combining point cloud matching analysis to obtain geodesics can greatly reduce the generation time of geodesics, can effectively reduce the gap between the transmission time of the video stream and the generation time of geodesics, so that the geodesics can be displayed on the left-eye image and the right-eye image corresponding to the video stream almost synchronously.

[0062] More specifically, in the embodiments of the present application, the 3D endoscope outputs the original left-eye image and right-eye image in the video stream based on the dual display screens. Step S4 corresponds to generating two geodesic lines respectively paired with the left-eye image and the right-eye image, and displaying them on the corresponding display screens. In this embodiment, the user of the 3D endoscope can more accurately observe the position of the geodesic lines, so that the geodesic lines can be presented as lines distributed on the surface of the organ tissue, generating a stereoscopic vision characteristic.

[0063] More specifically, the geodesic center is the center of geodesic calculation, which can be an anchor point determined by the user of the 3D endoscope on the display screen, or the center point of the field of view automatically generated based on the interaction of the two display screens, or the focus of the user's vision of the 3D endoscope on the display screen (applicable to the case where the 3D endoscope has a vision tracking function). In the embodiments of the present application, it is preferably an anchor point determined by the user of the 3D endoscope on the display screen to cooperate with the user to mark the geodesic lines required for the resection surgery. The user can determine or change the position of the geodesic center on the display device through a control terminal such as a mouse according to the usage requirements to adjust the distribution of the geodesic lines.

[0064] More specifically, the target geodesic distance can be the geodesic distance set by the user of the 3D endoscope for geodesic calculation, or the geodesic distance preset and selectable for geodesic calculation in the 3D endoscope. In the embodiments of the present application, it is preferably the set value of the user of the 3D endoscope.

[0065] It should be noted that the geodesic center is the position point determined by the user on the display screen. Step S3 is a process of inferring the positions of the position points in the left-eye image and the right-eye image based on the position coordinates of the position points on the display screen to locate the corresponding point clouds from the matching point cloud set, and defining the point cloud as the reference point cloud.

[0066] More specifically, after determining the reference point cloud, step S4 can screen out a series of corresponding point clouds from the matching point cloud set under the guidance of the target geodesic distance, and these point clouds are continuous. Marking and displaying the positions of these point clouds on the display screens corresponding to the left-eye image and the right-eye image can form the geodesic line effect on the display screens.

[0067] More specifically, other embodiments can use a vector algorithm to replace the heat propagation algorithm to generate geodesic lines. The heat propagation algorithm is an algorithm for calculating the geodesic distance according to the propagation of heat on the vector field, which can more accurately focus on the geodesic distance relationship between different point clouds and the reference point cloud.

[0068] More specifically, the principle of geodesic calculation based on the heat propagation algorithm is as follows: Based on the heat flow equation, the geodesic distance on the mesh surface is solved through the vector field and the Poisson equation. This algorithm transforms the non-linear geodesic distance problem into the solution of a linear Poisson equation, reducing the computational complexity. The specific calculation process is as follows: Through time discretization and space discretization, the Laplacian matrix and the area diagonal matrix are constructed, and then the scalar field u and its gradient field ▽u are solved. The gradient field is normalized and reversed to obtain the geodesic distance gradient field. Finally, the divergence of this gradient field is solved, and the geodesic distance of each vertex on the mesh is obtained through the Poisson equation. In the embodiments of the present application, the mesh surface is the surface constructed by the matching point cloud set, which represents the actual surface morphology of the organ tissue captured by the corresponding left-eye image and right-eye image. Therefore, the heat propagation algorithm can quickly and conveniently calculate the geodesic distance between different position points (point clouds) on different surfaces and the geodesic center. Then, in step S4, the position points are screened using the target geodesic distance, and a continuous, three-dimensional geodesic line composed of point clouds can be obtained. The geodesic line can be superimposed and displayed on the left-eye image and right-eye image in the display screen through methods such as projection or coordinate transformation.

[0069] The real-time geodesic measurement method of the 3D endoscope in the embodiments of the present application is based on the preprocessed left-eye image and right-eye image for stereo matching to obtain a matching point cloud set, and based on the heat propagation algorithm, suitable point clouds are screened in the matching point cloud set to generate a geodesic line on the display screen. It can quickly generate the geodesic line without affecting the screen display. During the generation process, downsampling processing is used to preprocess the left-eye image and right-eye image, which can reduce the amount of data processing without affecting the extraction of key features, greatly improving the point cloud analysis efficiency, effectively improving the generation efficiency of the geodesic line, and improving the display synchronization of the left-eye image, right-eye image, and geodesic line in the display screen.

[0070] In some preferred embodiments, the preprocessing process of step S2 is as follows: First, downsampling processing is performed, and then distortion removal and alignment processing are performed.

[0071] Specifically, downsampling processing can effectively reduce the number of pixels / resolution in the corresponding image. The method of performing downsampling first and then distortion removal and alignment can reduce the computational amount in the distortion removal and alignment processing, so as to further improve the efficiency of generating the geodesic line by the real-time geodesic measurement method of the 3D endoscope in the embodiments of the present application.

[0072] In some preferred embodiments, the downsampling processing is performed based on the downsampling interpolation method.

[0073] Specifically, the sizes (resolutions) of the left-eye image and the right-eye image are the same. Assuming the size of the left-eye image is (W, H) and the downsampling ratio is n, where n > 1, W is the width, and H is the height, then the size of the downsampled left-eye image is (w = W / n, h = H / n). Among them, the downsampling interpolation method can adopt methods such as nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, etc.

[0074] It should be noted that the images displayed on the display screen are the left-eye image and the right-eye image without preprocessing.

[0075] In some preferred embodiments, the distortion removal and alignment process includes the following steps:

[0076] Remap the downsampled left-eye image and right-eye image based on the mapping matrix, and the mapping matrix is obtained based on binocular camera calibration.

[0077] Specifically, the binocular camera is the dual camera of a 3D endoscope, which needs to be calibrated before leaving the factory for use to determine the reference mapping matrix representing binocular disparity. The reference mapping matrix can directly perform distortion removal and alignment on the left-eye image and the right-eye image. However, since the resolutions of the downsampled left-eye image and right-eye image are different from the corresponding original images, the real-time geodesy method of the 3D endoscope in the embodiments of the present application needs to use a mapping matrix suitable for the resolutions of the downsampled left-eye image and right-eye image for remapping to achieve the effect of distortion removal and alignment. This mapping matrix can be obtained by converting the coordinate relationship in the corresponding image based on the reference mapping matrix obtained from binocular camera calibration.

[0078] More specifically, the remapping process can map the pixels in the image from one position to another, and can achieve effects such as image translation, rotation, scaling, and perspective transformation, so that the downsampled left-eye image and right-eye image can be paired for use, facilitating stereo matching processing.

[0079] In some preferred embodiments, the downsampling ratio is 2 - 6 times.

[0080] Specifically, the single-vision image resolution of the current common 3D endoscope is 2K or 4K, and there are a large number of pixels in the image. Therefore, in the embodiments of the present application, the downsampling ratio is preferably 4 times. The downsampled left-eye image and right-eye image obtained by this downsampling process can achieve stereo matching without affecting the extraction of key features, greatly reducing the amount of data analysis in the stereo matching process and also reducing the number of point clouds in the geodesy calculation.

[0081] It should be noted that the shooting distance of the 3D endoscope is relatively close, and the actual acquired images contain fewer semantic objects. Stereo matching and geodesic calculation mainly focus on the object parallax depth, and the analysis process mainly considers the pixel positions corresponding to the undulating features of the organ tissue edges. After the downsampling process based on 4 times, the left-eye image and the right-eye image can still effectively retain the undulating features of the organ tissue edges.

[0082] In some preferred embodiments, the stereo matching includes the following steps:

[0083] Perform pixel matching on the preprocessed left-eye image and right-eye image, and obtain the coordinate deviation of each pixel to generate a disparity map;

[0084] According to the disparity map and the reprojection matrix, map the pixels in the undistorted and aligned left-eye image and / or right-eye image into the three-dimensional space to obtain the three-dimensional point cloud corresponding to each pixel, and form a matching point cloud set.

[0085] Specifically, the reprojection matrix realizes the conversion between the world coordinate system and the image pixel coordinate system, which can be obtained based on binocular camera calibration and can be scaled and adjusted based on the aforementioned downsampling ratio; the method of performing stereo matching on binocular images according to the disparity map and the reprojection matrix to obtain the three-dimensional point cloud belongs to a conventional method, so it will not be elaborated here.

[0086] In some preferred embodiments, the steps of generating a geodesic on the display screens corresponding to the left-eye image and the right-eye image based on the heat propagation algorithm according to the matching point cloud set, the reference point cloud, and the target geodesic distance include:

[0087] S41. Obtain the target point cloud based on the heat propagation algorithm, where the target point cloud is the point cloud in the matching point cloud set whose geodesic distance from the reference point cloud is the target geodesic distance;

[0088] S42. Obtain the two-dimensional coordinates of all target point clouds in the left-eye image and the right-eye image;

[0089] S43. Generate a geodesic on the display screens corresponding to the left-eye image and the right-eye image according to the two-dimensional coordinates.

[0090] Specifically, the real-time geodesic method of the 3D endoscope in the embodiment of the present application uses step S41 to screen suitable point clouds from the matching point cloud set, and then marks and projects the corresponding position points on the display screen according to the spatial positions of these point clouds for display to quickly form the geodesic required by the user.

[0091] More specifically, the two-dimensional coordinates obtained by the conversion in step S42 correspond to the coordinate spaces of the original left-eye image and right-eye image, that is, the conversion process of the target point cloud needs to consider the resolution difference caused by the aforementioned downsampling process to obtain the two-dimensional coordinates of the point cloud in the corresponding left-eye image and right-eye image.

[0092] In some preferred embodiments, the steps of obtaining the target point cloud based on the heat propagation algorithm include:

[0093] S411. Calculate the geodesic distance from different point clouds in the matching point cloud set to the reference point cloud based on the heat propagation algorithm;

[0094] S412. Screen out the point clouds in the matching point cloud set whose geodesic distance is equal to the target geodesic distance as the target point cloud.

[0095] Specifically, based on the foregoing content, after establishing a matching point cloud set that can accurately represent the actual surface morphology of the organ tissue captured by the camera, the heat propagation algorithm can conveniently and quickly calculate the geodesic distance between different point clouds in the matching point cloud set and the reference point cloud; step S412 can obtain all target point clouds by screening point clouds based on the target geodesic distance.

[0096] More specifically, limited by the number of point clouds in the matching point cloud set, the target point clouds obtained by screening in step S412 may not form a continuous geodesic. In some embodiments, the point clouds obtained by screening can be compensated by connecting lines, and the point clouds required to form a geodesic are selected for compensation to obtain a series of point clouds that can form a continuous geodesic as the target point cloud; in some other embodiments, step S412 can also set a distance range based on the target geodesic distance, that is, screen the point clouds whose geodesic distance meets this distance range to ensure that the obtained point clouds present a continuous part of a certain width as the target point cloud.

[0097] In some preferred embodiments, the steps of generating a geodesic on the display screens corresponding to the left-eye image and the right-eye image according to two-dimensional coordinates include:

[0098] S431. Create blank layers on the display screens corresponding to the left-eye image and the right-eye image respectively;

[0099] S432. Assign the value of 1 to the corresponding pixels on the blank layer based on the two-dimensional coordinates;

[0100] S433. Display the pixels with the value of 1 in the blank layer based on a preset color.

[0101] Specifically, creating blank layers means creating two blank layers that respectively match the display screens corresponding to the left-eye image and the right-eye image. Among them, the size and position of the blank layer match the size and position of the display screen, and the initial value of all pixels in the blank layer is 0, so that all pixels in the blank layer are in a transparent state.

[0102] More specifically, the real-time geodesic measurement method of the 3D endoscope according to the embodiments of the present application is equivalent to selecting the pixels where the geodesic line is located in the blank layer by using step S432, and displaying these pixels in a preset color by using step S433 to achieve convenient display of the geodesic line.

[0103] It should be noted that the user can change the preset color according to the usage requirements to change the display effect of the geodesic line, and can also use the real-time geodesic measurement method of the 3D endoscope according to the embodiments of the present application to obtain multiple geodesic lines and display them based on different preset colors; the user can also choose to hide or display the blank layer according to the usage requirements to hide or display the geodesic line.

[0104] More specifically, the real-time geodesic measurement method of the 3D endoscope according to the embodiments of the present application draws and displays the geodesic line based on the blank layer superimposed on the left-eye image and the right-eye image in the display screen, which can avoid the influence of the geodesic line on the image display and ensure the clarity and smoothness of the screen display.

[0105] More specifically, as Figure 2 and Figure 3 shown, the display screen preferably displays the geodesic center and the geodesic line at the same time. Among them, step S3 determines the reference point cloud corresponding to the geodesic center, and step S42 can simultaneously obtain the two-dimensional coordinates corresponding to the reference point cloud and display them on the display screen based on the same display method as the geodesic line.

[0106] In a second aspect, please refer to Figure 4 , some embodiments of the present application further provide a real-time geodesic measurement device for a 3D endoscope, which is applied in a 3D endoscope. The device includes:

[0107] An image acquisition module 201, configured to acquire a left-eye image and a right-eye image captured by the 3D endoscope;

[0108] A point cloud reconstruction module 202, configured to preprocess the left-eye image and the right-eye image, and then perform stereo matching to obtain a set of matching point clouds. The preprocessing includes downsampling processing and distortion removal and alignment processing;

[0109] A reference module 203, configured to determine a reference point cloud in the set of matching point clouds based on the position of the geodesic center in the left-eye image or the right-eye image;

[0110] A geodesic line module 204, configured to generate a geodesic line on the display screens corresponding to the left-eye image and the right-eye image based on the heat propagation algorithm according to the set of matching point clouds, the reference point cloud, and the target geodesic distance.

[0111] The real-time geodesic measurement device of the 3D endoscope according to the embodiment of the present application performs stereo matching based on the preprocessed left-eye image and right-eye image to obtain a set of matching point clouds, and filters suitable point clouds from the set of matching point clouds based on the heat propagation algorithm to generate a geodesic on the display screen. It can quickly generate the geodesic without affecting the screen display, and uses downsampling processing to preprocess the left-eye image and right-eye image during the generation process, which can reduce the amount of data processing without affecting the extraction of key features, greatly improving the efficiency of point cloud analysis, effectively improving the generation efficiency of the geodesic, and improving the display synchronization of the left-eye image, right-eye image and the geodesic in the display screen.

[0112] In some preferred embodiments, the real-time geodesic measurement device of the 3D endoscope according to the embodiment of the present application is used to execute the real-time geodesic measurement method of the 3D endoscope provided in the first aspect above.

[0113] In a third aspect, please refer to Figure 5 , some embodiments of the present application further provide a schematic structural diagram of an electronic device. The present application provides an electronic device, including: a processor 301 and a memory 302. The processor 301 and the memory 302 are interconnected and communicate with each other through a communication bus 303 and / or other forms of connection mechanisms (not marked). The memory 302 stores computer-readable instructions executable by the processor 301. When the electronic device runs, the processor 301 executes the computer-readable instructions to execute the method in any optional implementation manner of the above embodiments.

[0114] In a fourth aspect, the embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it executes the method in any optional implementation manner of the above embodiments. Among them, the computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, abbreviated as SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, abbreviated as EEPROM), erasable programmable read-only memory (Erasable Programmable Read Only Memory, abbreviated as EPROM), programmable read-only memory (Programmable Red-Only Memory, abbreviated as PROM), read-only memory (Read-OnlyMemory, abbreviated as ROM), magnetic memory, flash memory, magnetic disk or optical disc.

[0115] In summary, the embodiments of the present application provide a real-time geodesic measurement method, device, electronic device, and storage medium for a 3D endoscope. Among them, the real-time geodesic measurement method for the 3D endoscope in the embodiments of the present application performs stereo matching based on the preprocessed left-eye image and right-eye image to obtain a set of matching point clouds, and filters appropriate point clouds from the set of matching point clouds based on the heat propagation algorithm to generate a geodesic on the display screen. It can quickly generate the geodesic without affecting the screen display, and the downsampling process is used to preprocess the left-eye image and right-eye image during the generation process, which can reduce the amount of data processing without affecting the extraction of key features, greatly improving the point cloud analysis efficiency, effectively improving the generation efficiency of the geodesic, and improving the display synchronization of the left-eye image, right-eye image, and geodesic in the display screen.

[0116] In the embodiments provided by the present application, it should be understood that the disclosed device and method can be implemented in other ways. The device embodiments described above are only illustrative. For example, the division of the units is only a logical function division, and there may be other division methods in actual implementation. For another example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces, and the indirect coupling or communication connection of the device or unit may be in an electrical, mechanical or other form.

[0117] In addition, the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0118] Furthermore, in each embodiment of the present application, the various functional modules can be integrated together to form an independent part, or each module can exist alone, or two or more modules can be integrated to form an independent part.

[0119] In this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations.

[0120] The above are only the embodiments of the present application and are not used to limit the protection scope of the present application. For those skilled in the art, the present application can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A real-time geodetic method for a 3D endoscope, applied in a 3D endoscope, characterized in that: The method comprises the following steps: S1, obtaining left-eye images and right-eye images taken by a 3D endoscope; S2, preprocessing the left-eye image and the right-eye image, and then performing stereo matching to obtain a matching point cloud set, wherein the preprocessing includes downsampling processing and dedistortion alignment processing; S3, determining a reference point cloud in the matching point cloud set based on a position of a geodesic center in the left-eye image or the right-eye image; S4, generating geodesic lines on the display screen corresponding to the left eye image and the right eye image based on the matching point cloud set, the reference point cloud, and the target geodesic distance based on a heat propagation algorithm; The preprocessing process of step S2 is: first downsampling processing, and then de-distortion alignment processing; The step of generating geodesic lines on the display screen corresponding to the left-eye image and the right-eye image according to the matching point cloud set, the reference point cloud, and the target geodesic distance based on the heat propagation algorithm comprises: Acquire a target point cloud based on a heat propagation algorithm, wherein the target point cloud is a point cloud in the matching point cloud set whose geodesic distance to the reference point cloud is the target geodesic distance; Obtaining the two-dimensional coordinates of all target point clouds in the left-eye image and the right-eye image; Generating a geodesic on a display screen corresponding to the left-eye image and the right-eye image according to the two-dimensional coordinates; The step of generating a geodesic on a display screen corresponding to the left-eye image and the right-eye image according to the two-dimensional coordinates comprises: Creating blank layers respectively on the display screens corresponding to the left-eye image and the right-eye image; Assigning a value of 1 to the corresponding pixel on the blank layer based on the two-dimensional coordinates; Pixels with a value of 1 in the blank layer are displayed based on a preset color.

2. The real-time geodetic method of 3D endoscope according to claim 1, characterized in that: The de-distortion alignment process comprises the following steps: The left-eye image and the right-eye image after the downsampling process are remapped based on a mapping matrix, where the mapping matrix is ​​obtained based on binocular camera calibration.

3. The real-time geodetic method of 3D endoscope according to claim 1, characterized in that: The downsampling process has a magnification of 2-6 times.

4. The real-time geodetic method of 3D endoscope according to claim 1, characterized in that: The stereo matching comprises the following steps: Perform pixel matching on the preprocessed left-eye image and right-eye image, and obtain the coordinate deviation of each pixel to generate a disparity map; Pixels in the left-eye image and / or the right-eye image after the dedistortion and alignment process are mapped into a three-dimensional space according to the disparity map and the reprojection matrix to obtain a three-dimensional point cloud corresponding to each pixel to form the matching point cloud set.

5. The real-time geodetic method of 3D endoscope according to claim 1, characterized in that: The step of obtaining the target point cloud based on the heat propagation algorithm includes: Calculating the geodesic distances from different point clouds in the matching point cloud set to the reference point cloud based on a heat propagation algorithm; The point cloud whose geodesic distance in the matching point cloud set is equal to the target geodesic distance is selected as the target point cloud.

6. A real-time geodetic device for a 3D endoscope, used in a 3D endoscope, characterized in that: The device comprises: An image acquisition module, used to acquire left-eye images and right-eye images taken by a 3D endoscope; A point cloud reconstruction module, used for preprocessing the left-eye image and the right-eye image, and then performing stereo matching to obtain a matching point cloud set, wherein the preprocessing includes downsampling processing and dedistortion alignment processing; A reference module, configured to determine a reference point cloud in the matching point cloud set based on a position of a geodesic center in the left-eye image or the right-eye image; A geodesic module, for generating a geodesic on a display screen corresponding to the left-eye image and the right-eye image according to the matching point cloud set, the reference point cloud, and the target geodesic distance based on a heat propagation algorithm; The preprocessing process of the point cloud reconstruction module is: first downsampling, then dedistortion and alignment; The step of generating geodesic lines on the display screen corresponding to the left-eye image and the right-eye image according to the matching point cloud set, the reference point cloud, and the target geodesic distance based on the heat propagation algorithm comprises: Acquire a target point cloud based on a heat propagation algorithm, wherein the target point cloud is a point cloud in the matching point cloud set whose geodesic distance to the reference point cloud is the target geodesic distance; Obtaining the two-dimensional coordinates of all target point clouds in the left-eye image and the right-eye image; Generating a geodesic on a display screen corresponding to the left-eye image and the right-eye image according to the two-dimensional coordinates; The step of generating a geodesic on a display screen corresponding to the left-eye image and the right-eye image according to the two-dimensional coordinates comprises: Creating blank layers respectively on the display screens corresponding to the left-eye image and the right-eye image; Assigning a value of 1 to the corresponding pixel on the blank layer based on the two-dimensional coordinates; Pixels with a value of 1 in the blank layer are displayed based on a preset color.

7. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, the steps in the method according to any one of claims 1 to 5 are executed.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are executed.

Images