Real-time geodesic method and device of 3D endoscope, electronic equipment and storage medium
By adopting the pre-processing method of downsampling and dedistortion processing in 3D endoscopes and combining the heat propagation algorithm, the problem of geodesic distance calculation delay in high-definition picture processing is solved, and the rapid generation and real-time display of geodesics are realized. , improves the accuracy and safety of the surgery.
Patent Information
- Application Number
- CN202411430264.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-14
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-10-14
AI Technical Summary
When processing high-definition images in existing 3D endoscopes, there is a large delay in real-time geodesic distance calculation, making it difficult to ensure the real-time performance of geodesic results.
The preprocessing method of downsampling and dedistortion alignment processing is used to preprocess the left and right eye images of the 3D endoscope, and geodesics are generated based on the heat propagation algorithm to reduce data processing and improve point cloud analysis efficiency. .
Without affecting the screen display, geodesics are generated quickly, improving the generation efficiency and display synchronization of geodesics, ensuring real-time performance of geodesic results.
Smart Images

Figure CN118967669A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of endoscopes, and in particular to a real-time geodetic method, device, electronic device and storage medium for a 3D endoscope. Background Art
[0002] In recent years, the popularity of 3D endoscopes in hospitals has increased significantly as advanced medical imaging equipment. This technology relies on binocular cameras to simulate human eyes, creating a three-dimensional surgical field of view, significantly improving surgical accuracy and safety. Compared with two-dimensional endoscopes, 3D endoscopes more accurately display the depth and spatial relationship of tissues, enhancing the doctor's operational control effect.
[0003] In surgical practice, especially for liver and lung lesion resection, it is necessary to use 3D endoscopes to calculate surface distances (geodetic distances) in real time to achieve accurate resection planning and balance thorough resection with reduced bleeding risk. However, the current real-time geodesic distance calculation system faces computational challenges brought by high-definition image processing. Real-time geodesy for high-definition images has the problem of large data analysis, resulting in large delays in geodesic results, making it difficult to ensure real-time performance.
[0004] There is currently no effective technical solution to the above problems. Summary of the invention
[0005] The purpose of the present application is to provide a real-time geodetic method, device, electronic device and storage medium for a 3D endoscope to reduce the delay of geodetic calculation and ensure the real-time nature of the geodetic results.
[0006] In a first aspect, the present application provides a real-time geodesy method for a 3D endoscope, which is applied in a 3D endoscope, and the method comprises the following steps: S1, obtaining left-eye images and right-eye images taken by a 3D endoscope; S2, preprocessing the left-eye image and the right-eye image, and then performing stereo matching to obtain a matching point cloud set, wherein the preprocessing includes downsampling processing and dedistortion alignment processing; S3, determining a reference point cloud in the matching point cloud set based on a position of a geodesic center in the left-eye image or the right-eye image; S4. Generate geodesic lines on display screens corresponding to the left-eye image and the right-eye image according to the matching point cloud set, the reference point cloud, and the target geodesic distance based on a heat propagation algorithm.
[0007] The real-time geodesic method of the 3D endoscope of the present application performs stereo matching based on the preprocessed left-eye image and right-eye image to obtain a matching point cloud set, and screens suitable point clouds in the matching point cloud set based on the heat propagation algorithm to generate geodesics on the display screen. The geodesics can be quickly generated without affecting the screen display, and downsampling processing is used to preprocess the left-eye image and the right-eye image during the generation process. The amount of data processing can be reduced without affecting the extraction of key features, which greatly improves the efficiency of point cloud analysis, effectively improves the efficiency of geodesic generation, and improves the display synchronization of the left-eye image and the right-eye image with the geodesics in the display screen.
[0008] The real-time geodetic method of the 3D endoscope, wherein the de-distortion alignment process comprises the following steps: The left-eye image and the right-eye image after the downsampling process are remapped based on a mapping matrix, where the mapping matrix is obtained based on binocular camera calibration.
[0009] In this example, the remapping process can map pixels in the image from one position to another, and can achieve effects such as image translation, rotation, scaling, and perspective transformation, so that the downsampled left-eye image and right-eye image can be used in pairs to facilitate stereo matching processing.
[0010] The real-time geodetic method of the 3D endoscope, wherein the downsampling processing ratio is 2-6 times.
[0011] In this example, the downsampling ratio is preferably 4 times. The left-eye image and the right-eye image obtained by the downsampling process can achieve stereo matching without affecting the extraction of key features, greatly reducing the amount of data analysis in the stereo matching process, and also reducing the number of point clouds in geodetic calculations.
[0012] The real-time geodetic method of the 3D endoscope, wherein the stereo matching comprises the following steps: Perform pixel matching on the preprocessed left-eye image and right-eye image, and obtain the coordinate deviation of each pixel to generate a disparity map; Pixels in the left-eye image and / or the right-eye image after the dedistortion and alignment process are mapped into a three-dimensional space according to the disparity map and the reprojection matrix to obtain a three-dimensional point cloud corresponding to each pixel to form the matching point cloud set.
[0013] The real-time geodesic method of the 3D endoscope, wherein the step of generating a geodesic line on the display screen corresponding to the left eye image and the right eye image according to the matching point cloud set, the reference point cloud, and the target geodesic distance based on the heat propagation algorithm comprises: Acquire a target point cloud based on a heat propagation algorithm, wherein the target point cloud is a point cloud in the matching point cloud set whose geodesic distance to the reference point cloud is the target geodesic distance; Obtaining the two-dimensional coordinates of all target point clouds in the left-eye image and the right-eye image; A geodesic line is generated on a display screen corresponding to the left-eye image and the right-eye image according to the two-dimensional coordinates.
[0014] The real-time geodetic method of the 3D endoscope, wherein the step of acquiring the target point cloud based on the heat propagation algorithm comprises: Calculating the geodesic distances from different point clouds in the matching point cloud set to the reference point cloud based on a heat propagation algorithm; The point cloud whose geodesic distance in the matching point cloud set is equal to the target geodesic distance is selected as the target point cloud.
[0015] The real-time geodesic method of the 3D endoscope, wherein the step of generating a geodesic line on the display screen corresponding to the left-eye image and the right-eye image according to the two-dimensional coordinates comprises: Creating blank layers respectively on the display screens corresponding to the left-eye image and the right-eye image; Assigning a value of 1 to the corresponding pixel on the blank layer based on the two-dimensional coordinates; Pixels with a value of 1 in the blank layer are displayed based on a preset color.
[0016] In a second aspect, the present application further provides a real-time geodetic device for a 3D endoscope, which is applied in a 3D endoscope, and the device comprises: An image acquisition module, used to acquire left-eye images and right-eye images taken by a 3D endoscope; A point cloud reconstruction module, used for preprocessing the left-eye image and the right-eye image, and then performing stereo matching to obtain a matching point cloud set, wherein the preprocessing includes downsampling processing and dedistortion alignment processing; A reference module, configured to determine a reference point cloud in the matching point cloud set based on a position of a geodesic center in the left-eye image or the right-eye image; The geodesic module is used to generate geodesic lines on the display screen corresponding to the left-eye image and the right-eye image according to the matching point cloud set, the reference point cloud, and the target geodesic distance based on a heat propagation algorithm.
[0017] The real-time geodetic device of the 3D endoscope of the present application performs stereo matching based on the preprocessed left-eye image and right-eye image to obtain a matching point cloud set, and screens suitable point clouds in the matching point cloud set based on the heat propagation algorithm to generate geodesics on the display screen. It can quickly generate geodesics without affecting the screen display, and uses downsampling processing to preprocess the left-eye image and the right-eye image during the generation process, which can reduce the amount of data processing without affecting the extraction of key features, greatly improve the efficiency of point cloud analysis, effectively improve the efficiency of geodesic generation, and improve the display synchronization of the left-eye image and the right-eye image with the geodesic in the display screen.
[0018] In a third aspect, the present application further provides an electronic device, comprising a processor and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps in the method provided in the first aspect are executed.
[0019] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps in the method provided in the first aspect are executed.
[0020] As can be seen from the above, the present application provides a real-time geodetic method, device, electronic device and storage medium for a 3D endoscope, wherein the real-time geodetic method for a 3D endoscope of the present application performs stereo matching based on the preprocessed left-eye image and right-eye image to obtain a matching point cloud set, and screens suitable point clouds in the matching point cloud set based on a heat propagation algorithm to generate geodesics on a display screen, which can complete the rapid generation of geodesics without affecting the screen display, and uses downsampling processing to preprocess the left-eye image and the right-eye image during the generation process, which can reduce the amount of data processing without affecting the extraction of key features, greatly improve the efficiency of point cloud analysis, effectively improve the efficiency of geodesic generation, and improve the display synchronization of the left-eye image and the right-eye image with the geodesic in the display screen. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] Figure 1 A flow chart of a real-time geodesy method for a 3D endoscope provided in an embodiment of the present application.
[0022] Figure 2 A schematic diagram of a display screen corresponding to a left-eye image at a moment in a 3D endoscope when the real-time geodetic method for a 3D endoscope provided in an embodiment of the present application is applied.
[0023] Figure 3 A schematic diagram of a display screen corresponding to a left-eye image at another moment in a 3D endoscope when the real-time geodetic method for a 3D endoscope provided in an embodiment of the present application is applied.
[0024] Figure 4 A schematic diagram of the structure of a real-time geodetic device for a 3D endoscope provided in an embodiment of the present application.
[0025] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.
[0026] Reference numerals: 201, image acquisition module; 202, point cloud reconstruction module; 203, reference module; 204, geodesic module; 301, processor; 302, memory; 303, communication bus. DETAILED DESCRIPTION
[0027] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.
[0028] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.
[0029] First, please refer to Figure 1 Some embodiments of the present application provide a real-time geodesy method for a 3D endoscope, which is applied in a 3D endoscope. The method includes the following steps: S1, obtaining left-eye images and right-eye images taken by a 3D endoscope; S2, preprocessing the left eye image and the right eye image, and then performing stereo matching to obtain a matching point cloud set, the preprocessing including downsampling processing and dedistortion alignment processing; S3, determining a reference point cloud in a matching point cloud set based on a position of a geodesic center in a left-eye image or a right-eye image; S4. Generate geodesic lines on display screens corresponding to the left-eye image and the right-eye image according to the matching point cloud set, the reference point cloud, and the target geodesic distance based on a heat propagation algorithm.
[0030] Specifically, a 3D endoscope is a medical device that integrates high-precision optical and imaging technologies. It uses a dual-camera design inside that can capture and synthesize high-definition three-dimensional images, providing doctors with a more realistic and intuitive surgical field of view. The images taken by the two cameras correspond to the left eye image and right eye image mentioned above, respectively.
[0031] More specifically, in the embodiment of the present application, the two cameras of the 3D endoscope continuously shoot, and the left-eye image and the right-eye image in step S1 are pictures extracted in real time based on the video streams continuously shot by the two cameras, respectively. The real-time geodesic method of the 3D endoscope in the embodiment of the present application obtains geodesics based on real-time analysis of these pictures, so as to output geodesics in real time on the display screen corresponding to the display video stream; wherein, steps S1-S4 are equivalent to the process of frame picture analysis of the video stream obtained by shooting and generating geodesics.
[0032] It should be noted that the real-time geodesy method of the 3D endoscope in the embodiment of the present application can be to perform frame-by-frame analysis on the video stream obtained by the 3D endoscope to generate and update the geodesic line, or it can be to perform frame-skipping analysis on the video stream obtained by the 3D endoscope to generate and update the geodesic line. The former can more effectively ensure the real-time and accuracy of the geodesic line, and the latter can save system resources and reduce the operating load of the 3D endoscope.
[0033] More specifically, under normal circumstances, the left-eye image and the right-eye image in the video stream obtained by the 3D endoscope will be directly displayed on the display screen of the display device of the 3D endoscope. The real-time geodetic method of the 3D endoscope in the embodiment of the present application extracts the left-eye image and the right-eye image while the 3D endoscope is performing display processing, and generates geodesics superimposed on the left-eye image and the right-eye image displayed on the display screen based on steps S2 to S4, wherein the geodesic generation process is mainly a process of analyzing the matching point cloud set established in step S2, that is, the processing process of steps S2 to S4 will not affect the pixel data in the left-eye image and the right-eye image natively displayed on the display screen.
[0034] More specifically, the left-eye image and the right-eye image are planar images with parallax and can be matched to form a three-dimensional visual effect. Therefore, step S2 can perform stereo matching according to the feature pairing relationship of pixels in the preprocessed left-eye image and the right-eye image to reconstruct the point cloud, thereby obtaining a set of matching point clouds, wherein the stereo matching process can be performed using existing stereo matching algorithms for three-dimensional reconstruction, such as SSD (Sum of Squared Differences), SAD (Sum of Absolute Differences), ZNCC (Zero-mean Normalized Cross-Correlation), BM (Block Matching Algorithm), SGBM (Semi-Global Block Matching) and other algorithms, preferably using BM or SGBM algorithms.
[0035] More specifically, the de-distortion alignment processing in the preprocessing can effectively eliminate the pixel position offset caused by distortion of the left-eye image and the right-eye image, effectively improve the matching accuracy and consistency of subsequent stereo matching, and simplify the search range of stereo matching to increase the matching speed.
[0036] More specifically, the downsampling process in the preprocessing can generate two low-resolution images to be analyzed (corresponding to the left eye image and the right eye image). These two low-resolution images to be analyzed are only used for stereo matching to reconstruct the matching point cloud set and are not used for screen display. The downsampling process can effectively reduce the data dimension of subsequent processing and greatly improve the efficiency of subsequent geodesic generation.
[0037] More specifically, the downsampling ratio is set according to system parameters (mainly depending on the resolution of the native left-eye image and right-eye image, and the type of stereo matching algorithm). The downsampling process must be performed without affecting the extraction of key features to reduce the amount of data analysis for subsequent processing, thereby optimizing the allocation and use of computing resources, increasing the generation rate of geodesics, and improving the timeliness of geodesics on the display.
[0038] It should be noted that the video stream obtained by the camera of the 3D endoscope is directly transmitted to the display device to be displayed on the corresponding display screen, and the geodesic lines obtained by the real-time geodesy method of the 3D endoscope in the embodiment of the present application are also displayed on the corresponding display screen; the real-time geodesy method of the 3D endoscope in the embodiment of the present application is based on downsampling processing to reduce the amount of data processing, and the method of obtaining geodesic lines by combining point cloud matching analysis can greatly reduce the time consumption of generating geodesic lines, and can effectively reduce the difference between the transmission time of the video stream and the time consumption of generating geodesic lines, so that the geodesic lines can be displayed almost synchronously on the left eye image and the right eye image corresponding to the video stream.
[0039] More specifically, in an embodiment of the present application, the 3D endoscope outputs the native left-eye image and right-eye image in the video stream based on the dual display screens, and step S4 generates two geodesics corresponding to the left-eye image and the right-eye image respectively, and displays them on the corresponding display screens; in this embodiment, the user of the 3D endoscope can more accurately observe the position of the geodesics, so that the geodesics can be presented as lines distributed on the surface of organ tissue, producing stereoscopic visual characteristics.
[0040] More specifically, the geodesic center is the center of geodesic calculation, which can be an anchor point determined by the user of the 3D endoscope on the display screen, or a center point of the field of view automatically generated based on the interaction of two display screens, or a focus of the vision of the user of the 3D endoscope on the display screen (applicable to the case where the 3D endoscope has a visual tracking function). In the embodiment of the present application, it is preferably an anchor point determined by the user of the 3D endoscope on the display screen to assist the user in marking the geodesic lines required for the resection surgery. The user can determine or change the position of the geodesic center on the display device through a control terminal such as a mouse according to usage requirements to adjust the distribution of geodesic lines.
[0041] More specifically, the target geodetic distance can be the geodetic distance set by the user of the 3D endoscope and requiring geodetic calculation, or it can be a system preset, optional geodetic distance for geodetic calculation in the 3D endoscope. In the embodiment of the present application, it is preferably the set value of the user of the 3D endoscope.
[0042] It should be noted that the geodetic center is a position point determined by the user on the display screen, and step S3 is a process of inferring the position of the position point in the left eye image and the right eye image based on the position coordinates of the position point on the display screen to locate the corresponding point cloud from the matching point cloud set, and defining the point cloud as the reference point cloud.
[0043] More specifically, after determining the reference point cloud, step S4 can screen out a corresponding series of point clouds from the matching point cloud set under the guidance of the target geodesic distance, and these point clouds are continuous. The positions of these point clouds are marked and displayed on the display screen corresponding to the left eye image and the right eye image, so that a geodesic effect can be formed on the display screen.
[0044] More specifically, other implementations may use a vector algorithm instead of a heat propagation algorithm to generate geodesics. The heat propagation algorithm is equivalent to an algorithm for calculating geodesic distances based on the propagation of heat in a vector field, and can more accurately focus on the geodesic distance relationship between different point clouds and the reference point cloud.
[0045] More specifically, the principle of geodesic calculation based on heat propagation algorithm is as follows: based on the heat flow equation, the geodesic distance on the grid surface is solved by vector field and Poisson equation. The algorithm transforms the nonlinear geodesic distance problem into linear Poisson equation solution, which reduces the computational complexity. The specific calculation process is as follows: through time discretization and space discretization, Laplace matrix and area diagonal matrix are constructed, and then the scalar field u and its gradient field ▽u are solved, and the gradient field is normalized and reversed to obtain the geodesic distance gradient field. Finally, the divergence of the gradient field is solved, and the Poisson equation is used to obtain the geodesic distance gradient field of each grid. Geodesic distance of vertices; in the embodiment of the present application, the mesh surface is a surface constructed by matching point cloud sets, which characterizes the actual surface morphology of the organ tissue captured by the corresponding left-eye image and right-eye image. Therefore, the heat propagation algorithm can quickly and conveniently calculate the geodesic distance between different position points (point clouds) on different surfaces and the geodesic center. Then, step S4 uses the target geodesic distance to filter the position points to obtain a continuous, point cloud-based, three-dimensional geodesic line, which can be superimposed and displayed on the left-eye image and the right-eye image in the display screen through projection or coordinate transformation.
[0046] The real-time geodesic method of the 3D endoscope in the embodiment of the present application performs stereo matching based on the preprocessed left-eye image and right-eye image to obtain a matching point cloud set, and screens suitable point clouds in the matching point cloud set based on the heat propagation algorithm to generate geodesics on the display screen. The geodesics can be quickly generated without affecting the screen display, and downsampling is used to preprocess the left-eye image and the right-eye image during the generation process. The amount of data processing can be reduced without affecting the extraction of key features, which greatly improves the efficiency of point cloud analysis, effectively improves the efficiency of geodesic generation, and improves the display synchronization of the left-eye image and the right-eye image with the geodesics in the display screen.
[0047] In some preferred implementations, the preprocessing process of step S2 is: first perform downsampling processing, and then perform de-distortion alignment processing.
[0048] Specifically, the downsampling process can effectively reduce the number of pixels / resolution in the corresponding image. The method of downsampling first and then de-distortion and alignment can reduce the amount of calculation in the de-distortion and alignment process, so as to further improve the efficiency of generating geodesics by the real-time geodesy method of the 3D endoscope in the embodiment of the present application.
[0049] In some preferred embodiments, the downsampling process is performed based on a downsampling interpolation method.
[0050] Specifically, the size (resolution) of the left eye image and the right eye image is the same. If the size of the left eye image is (W, H) and the downsampling ratio is n, where n>1, W is the width, and H is the height, then the size of the left eye image after downsampling is (w=W / n, h=H / n), where the downsampling interpolation method can adopt downsampling methods such as nearest neighbor interpolation, bilinear interpolation, and bicubic interpolation.
[0051] It should be noted that the images displayed on the display screen are left-eye images and right-eye images that are not pre-processed.
[0052] In some preferred embodiments, the de-distortion alignment process comprises the following steps: The downsampled left-eye image and right-eye image are remapped based on a mapping matrix, where the mapping matrix is obtained based on binocular camera calibration.
[0053] Specifically, the binocular camera is a dual camera of a 3D endoscope, which needs to be calibrated before leaving the factory to determine a reference mapping matrix that characterizes the binocular parallax. The reference mapping matrix can directly dedistort and align the left-eye image and the right-eye image; but since the resolution of the left-eye image and the right-eye image after downsampling is different from that of the corresponding native image, the real-time geodetic method of the 3D endoscope in the embodiment of the present application needs to use a mapping matrix suitable for the resolution of the left-eye image and the right-eye image after downsampling for remapping to achieve the effect of dedistortion and alignment. The mapping matrix can be obtained by converting the coordinate relationship of the reference mapping matrix obtained based on the binocular camera calibration in the corresponding image.
[0054] More specifically, the remapping process can map pixels in an image from one position to another, and can achieve effects such as image translation, rotation, scaling, and perspective transformation, so that the downsampled left-eye image and right-eye image can be used in pairs to facilitate stereo matching processing.
[0055] In some preferred embodiments, the downsampling ratio is 2-6 times.
[0056] Specifically, the single-vision image resolution of the current common 3D endoscope is 2K or 4K, and there are a large number of pixels in the image. Therefore, in the embodiment of the present application, the downsampling ratio is preferably 4 times. The left-eye image and the right-eye image obtained by the downsampling process can achieve stereo matching without affecting the extraction of key features, which greatly reduces the amount of data analysis in the stereo matching process and can also reduce the number of point clouds in geodetic calculations.
[0057] It should be noted that the shooting distance of the 3D endoscope is relatively short, and the images actually obtained contain fewer semantic objects. Stereo matching and geodesic calculations mainly focus on the parallax depth of objects. The analysis process mainly considers the pixel positions corresponding to the edge undulation features of organ tissues. The left-eye and right-eye images obtained after 4-fold downsampling can still effectively retain the edge undulation features of organ tissues.
[0058] In some preferred embodiments, stereo matching comprises the following steps: Perform pixel matching on the preprocessed left-eye image and right-eye image, and obtain the coordinate deviation of each pixel to generate a disparity map; The pixels in the left-eye image and / or the right-eye image after the dedistortion and alignment process are mapped into a three-dimensional space according to the disparity map and the reprojection matrix to obtain a three-dimensional point cloud corresponding to each pixel to form a matching point cloud set.
[0059] Specifically, the reprojection matrix realizes the conversion between the world coordinate system and the image pixel coordinate system, which is obtained based on the binocular camera calibration and can be scaled and adjusted based on the aforementioned downsampling ratio. The method of stereo matching of binocular images according to the disparity map and the reprojection matrix to obtain a three-dimensional point cloud is a conventional method, so it will not be elaborated here.
[0060] In some preferred embodiments, the step of generating geodesic lines on the display screen corresponding to the left-eye image and the right-eye image based on the matching point cloud set, the reference point cloud, and the target geodesic distance based on the heat propagation algorithm includes: S41, acquiring a target point cloud based on a heat propagation algorithm, where the target point cloud is a point cloud in the matching point cloud set whose geodesic distance to the reference point cloud is the target geodesic distance; S42, obtaining the two-dimensional coordinates of all target point clouds in the left-eye image and the right-eye image; S43 , generating geodesics on the display screen corresponding to the left-eye image and the right-eye image according to the two-dimensional coordinates.
[0061] Specifically, the real-time geodesic method of the 3D endoscope in the embodiment of the present application utilizes step S41 to select appropriate point clouds from the matching point cloud set, and then projects the corresponding position point marks on the display screen according to the spatial positions of these point clouds to quickly form the geodesic lines required by the user.
[0062] More specifically, the two-dimensional coordinates obtained by the conversion in step S42 correspond to the coordinate space of the native left-eye image and the right-eye image, that is, the conversion process of the target point cloud needs to take into account the resolution difference caused by the aforementioned downsampling processing to obtain the two-dimensional coordinates of the point cloud in the corresponding left-eye image and the right-eye image.
[0063] In some preferred embodiments, the step of acquiring the target point cloud based on the heat propagation algorithm includes: S411, calculating the geodesic distances from different point clouds in the matching point cloud set to the reference point cloud based on a heat propagation algorithm; S412, filter out the point cloud whose geodesic distance in the matching point cloud set is equal to the target geodesic distance as the target point cloud.
[0064] Specifically, based on the foregoing content, it can be known that after establishing a matching point cloud set that can accurately represent the actual surface morphology of the organ tissue captured by the camera, the heat propagation algorithm can quickly and easily calculate the geodesic distance between different point clouds in the matching point cloud set and the reference point cloud; step S412 performs point cloud screening based on the target geodesic distance to obtain all target point clouds.
[0065] More specifically, due to the limitation of the number of point clouds in the matching point cloud set, the target point cloud screened in step S412 may not be able to form a continuous geodesic. In some embodiments, the screened point cloud can be connected to compensate for the selected point cloud required to form the geodesic, so as to obtain a series of point clouds that can form a continuous geodesic as the target point cloud; in some other embodiments, step S412 can also set a distance range based on the target geodesic distance, that is, screen the point cloud whose geodesic distance meets the distance range, so as to ensure that a point cloud presented as a continuous part of a certain width is obtained as the target point cloud.
[0066] In some preferred embodiments, the step of generating a geodesic line on a display screen corresponding to the left-eye image and the right-eye image according to the two-dimensional coordinates includes: S431, creating blank layers on the display screens corresponding to the left-eye image and the right-eye image respectively; S432, assigning a value of 1 to the corresponding pixel on the blank layer based on the two-dimensional coordinates; S433. Display pixels with a value of 1 in the blank layer based on a preset color.
[0067] Specifically, creating a blank layer is to create two new blank layers that match the display screens corresponding to the left-eye image and the right-eye image respectively, wherein the size and position of the blank layer match the size and position of the display screen, and the initial value of all pixels in the blank layer is 0, so that all pixels in the blank layer are in a transparent state.
[0068] More specifically, the real-time geodesic method of the 3D endoscope in the embodiment of the present application is equivalent to selecting the pixels where the geodesic lines are located in the blank layer using step S432, and displaying these pixels as preset colors using step S433 to achieve convenient display of the geodesic lines.
[0069] It should be noted that the user can change the preset color to change the display effect of the geodesic line according to usage requirements, and can also use the real-time geodesic method of the 3D endoscope in the embodiment of the present application to obtain multiple geodesic lines and display them based on different preset colors; the user can also choose to hide or display a blank layer to hide or display the geodesic line according to usage requirements.
[0070] More specifically, the real-time geodesic method of the 3D endoscope in the embodiment of the present application draws and displays geodesic lines in the display screen based on a blank layer superimposed on the left-eye image and the right-eye image, which can avoid the influence of the geodesic lines on the image display and ensure the clarity and smoothness of the picture display.
[0071] More specifically, if Figure 2 and Figure 3 As shown, the display screen preferably displays the geodesic center and the geodesic line at the same time, wherein step S3 determines the reference point cloud corresponding to the geodesic center, and step S42 can simultaneously obtain the two-dimensional coordinates corresponding to the reference point cloud and display them on the display screen based on the same display method as the geodesic line.
[0072] Second, please refer to Figure 4 Some embodiments of the present application also provide a real-time geodetic device for a 3D endoscope, which is applied in a 3D endoscope and includes: An image acquisition module 201 is used to acquire a left-eye image and a right-eye image taken by a 3D endoscope; The point cloud reconstruction module 202 is used to pre-process the left-eye image and the right-eye image and then perform stereo matching to obtain a matching point cloud set, wherein the pre-processing includes down-sampling processing and de-distortion alignment processing; A reference module 203, for determining a reference point cloud in a matching point cloud set based on a position of a geodesic center in a left-eye image or a right-eye image; The geodesic module 204 is used to generate geodesic lines on the display screen corresponding to the left-eye image and the right-eye image based on the heat propagation algorithm according to the matching point cloud set, the reference point cloud, and the target geodesic distance.
[0073] The real-time geodetic device of the 3D endoscope in the embodiment of the present application performs stereo matching based on the preprocessed left-eye image and right-eye image to obtain a matching point cloud set, and screens suitable point clouds in the matching point cloud set based on the heat propagation algorithm to generate geodesics on the display screen. It can quickly generate geodesics without affecting the screen display, and uses downsampling processing to preprocess the left-eye image and the right-eye image during the generation process. It can reduce the amount of data processing without affecting the extraction of key features, greatly improve the efficiency of point cloud analysis, effectively improve the efficiency of geodesic generation, and improve the display synchronization of the left-eye image and the right-eye image with the geodesic in the display screen.
[0074] In some preferred embodiments, the real-time geodetic device for a 3D endoscope according to an embodiment of the present application is used to execute the real-time geodetic method for a 3D endoscope provided in the first aspect above.
[0075] Third, please refer to Figure 5 Some embodiments of the present application also provide a structural diagram of an electronic device. The present application provides an electronic device, including: a processor 301 and a memory 302. The processor 301 and the memory 302 are interconnected and communicate with each other through a communication bus 303 and / or other forms of connection mechanisms (not marked). The memory 302 stores computer-readable instructions executable by the processor 301. When the electronic device is running, the processor 301 executes the computer-readable instructions to execute the method in any optional implementation of the above embodiments.
[0076] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method in any optional implementation of the above embodiment is executed. The computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, disk or optical disk.
[0077] In summary, the embodiments of the present application provide a real-time geodetic method, device, electronic device and storage medium for a 3D endoscope, wherein the real-time geodetic method for a 3D endoscope in the embodiments of the present application performs stereo matching based on the preprocessed left-eye image and right-eye image to obtain a matching point cloud set, and screens suitable point clouds in the matching point cloud set based on a heat propagation algorithm to generate geodesics on a display screen, which can complete the rapid generation of geodesics without affecting the screen display, and uses downsampling processing to preprocess the left-eye image and the right-eye image during the generation process, which can reduce the amount of data processing without affecting the extraction of key features, greatly improve the efficiency of point cloud analysis, effectively improve the efficiency of geodesic generation, and improve the display synchronization of the left-eye image and the right-eye image with the geodesic in the display screen.
[0078] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.
[0079] In addition, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0080] Furthermore, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.
[0081] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any such actual relationship or order between these entities or operations.
[0082] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims
1. A real-time geodetic method for a 3D endoscope, applied in a 3D endoscope, characterized in that: The method comprises the following steps: S1, obtaining left-eye images and right-eye images taken by a 3D endoscope; S2, preprocessing the left-eye image and the right-eye image, and then performing stereo matching to obtain a matching point cloud set, wherein the preprocessing includes downsampling processing and dedistortion alignment processing; S3, determining a reference point cloud in the matching point cloud set based on a position of a geodesic center in the left-eye image or the right-eye image; S4. Generate geodesic lines on display screens corresponding to the left-eye image and the right-eye image according to the matching point cloud set, the reference point cloud, and the target geodesic distance based on a heat propagation algorithm.
2. The real-time geodetic method of 3D endoscope according to claim 1, characterized in that: The de-distortion alignment process comprises the following steps: The left-eye image and the right-eye image after the downsampling process are remapped based on a mapping matrix, where the mapping matrix is obtained based on binocular camera calibration.
3. The real-time geodetic method of 3D endoscope according to claim 1, characterized in that: The downsampling process has a magnification of 2-6 times.
4. The real-time geodetic method of 3D endoscope according to claim 1, characterized in that: The stereo matching comprises the following steps: Perform pixel matching on the preprocessed left-eye image and right-eye image, and obtain the coordinate deviation of each pixel to generate a disparity map; Pixels in the left-eye image and / or the right-eye image after the dedistortion and alignment process are mapped into a three-dimensional space according to the disparity map and the reprojection matrix to obtain a three-dimensional point cloud corresponding to each pixel to form the matching point cloud set.
5. The real-time geodetic method of 3D endoscope according to claim 1, characterized in that: The step of generating geodesic lines on the display screen corresponding to the left-eye image and the right-eye image according to the matching point cloud set, the reference point cloud, and the target geodesic distance based on the heat propagation algorithm comprises: Acquire a target point cloud based on a heat propagation algorithm, wherein the target point cloud is a point cloud in the matching point cloud set whose geodesic distance to the reference point cloud is the target geodesic distance; Obtaining the two-dimensional coordinates of all target point clouds in the left-eye image and the right-eye image; A geodesic line is generated on a display screen corresponding to the left-eye image and the right-eye image according to the two-dimensional coordinates.
6. The real-time geodetic method of 3D endoscope according to claim 5, characterized in that: The step of obtaining the target point cloud based on the heat propagation algorithm includes: Calculating the geodesic distances from different point clouds in the matching point cloud set to the reference point cloud based on a heat propagation algorithm; The point cloud whose geodesic distance in the matching point cloud set is equal to the target geodesic distance is selected as the target point cloud.
7. The real-time geodetic method of 3D endoscope according to claim 5, characterized in that: The step of generating a geodesic on a display screen corresponding to the left-eye image and the right-eye image according to the two-dimensional coordinates comprises: Creating blank layers respectively on the display screens corresponding to the left-eye image and the right-eye image; Assigning a value of 1 to the corresponding pixel on the blank layer based on the two-dimensional coordinates; Pixels with a value of 1 in the blank layer are displayed based on a preset color.
8. A real-time geodetic device for a 3D endoscope, used in a 3D endoscope, characterized in that: The device comprises: An image acquisition module, used to acquire left-eye images and right-eye images taken by a 3D endoscope; A point cloud reconstruction module, used for preprocessing the left-eye image and the right-eye image, and then performing stereo matching to obtain a matching point cloud set, wherein the preprocessing includes downsampling processing and dedistortion alignment processing; A reference module, configured to determine a reference point cloud in the matching point cloud set based on a position of a geodesic center in the left-eye image or the right-eye image; The geodesic module is used to generate geodesic lines on the display screen corresponding to the left-eye image and the right-eye image according to the matching point cloud set, the reference point cloud, and the target geodesic distance based on a heat propagation algorithm.
9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, the steps in the method according to any one of claims 1 to 7 are executed.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are executed.
Citation Information
Patent Citations
Image processing device, endoscopic device, program and image processing method
CN105050473A
Ultrasound imaging device, ultrasound imaging system, ultrasound imaging method and ultrasound imaging program
CN112384149A
Three-dimensional virtual character intelligent skinning method
CN112802161A
Binocular endoscopic image three-dimensional splicing method based on two-dimensional feature points
CN112862687A
Endoscopic image three-dimensional reconstruction method combining SfM and binocular matching
CN112967330A
