Three-dimensional reconstruction method and system based on single-path medical endoscope

Abstract

The invention relates to a three-dimensional reconstruction method and system based on a single-path medical endoscope. The method comprises the steps that single-path image information is acquired and preprocessed; performing three-dimensional reconstruction on the preprocessed single-path image information and obtaining a reconstructed fusion image; and matching the reconstructed fusion image with a corresponding output format and outputting the reconstructed fusion image. According to the three-dimensional reconstruction method based on the single-path medical endoscope, three-dimensional reconstruction can be carried out on the single-path image and the corresponding fused image is generated through cooperation of hardware computing power and a software algorithm, the pixel of a three-dimensional image is improved, and therefore the stereoscopic impression and the ultra-high definition of the three-dimensional image are improved.

Description

Technical Field

[0001] The disclosed embodiments relate to the technical field of medical image processing, and more specifically, to a three-dimensional reconstruction method and system based on a single-channel medical endoscope. Background Art

[0002] The 3D endoscopic camera system is a medical device used for surgery that combines endoscopic technology and three-dimensional imaging technology. Through special cameras and display technology, it converts traditional two-dimensional images into stereoscopic three-dimensional images, helping surgeons obtain a clearer and more detailed perspective when performing surgery.

[0003] Most of the existing 3D endoscope camera systems use a dual-channel imaging solution and dual-channel sensors at a certain distance to present a 3D effect after collecting images.

[0004] In the prior art, the mirror part and the sensor part at the front end of the endoscope are relatively small, resulting in lower pixels of the image transmitted back by the sensor, and ultimately making the clarity of the 3D imaging image poor. Summary of the invention

[0005] One purpose of the embodiments of the present disclosure is to provide a new technical solution for a three-dimensional reconstruction method and system based on a single-channel medical endoscope.

[0006] According to a first aspect of the present disclosure, a three-dimensional reconstruction method based on a single-channel medical endoscope is provided, the method comprising: Obtain single-channel image information and perform preprocessing; Perform three-dimensional reconstruction on the preprocessed single-channel image information and obtain a reconstructed fused image; The reconstructed fused image is matched to a corresponding output format and outputted.

[0007] Optionally, obtaining single-channel image information and performing preprocessing includes: Obtain single-channel image information and corresponding single-channel image format; Matching the single-channel image format with a standard image format in a preset standard image library, wherein the standard image library stores different standard image formats for three-dimensional reconstruction; If the single-channel image format does not match the standard image format, the single-channel image format is converted until the single-channel image format matches the standard image format; The single-channel image information whose single-channel image format matches the standard image format is set as the preprocessed single-channel image information.

[0008] Optionally, three-dimensionally reconstructing the preprocessed single-channel image information and obtaining a reconstructed fused image includes: Generate plane projections of different viewing angles based on the preprocessed single-channel image information through a preset algorithm; Fusion of the plane projections from different perspectives to generate a point cloud image; Rendering a plane projection of a preset viewing angle based on the point cloud image and generating a corresponding depth image; The depth image is set as the reconstructed fused image.

[0009] Optionally, a preset algorithm is used to generate plane projections of different viewing angles based on the preprocessed single-channel image information, including: Inputting the preprocessed single-channel image information into a preset deep learning network and obtaining plane projections of different viewing angles; The error between the plane projection predicted by the deep learning network and the real plane projection is calculated through the loss function; The weights of the deep learning network are adjusted through an optimization algorithm until the error between the plane projection predicted by the deep learning network and the actual plane projection is within a preset error range.

[0010] Optionally, fusing the plane projections of different perspectives to generate a point cloud image includes: Get plane projections and preset 3D point clouds from different perspectives; Obtain camera intrinsic parameters and camera extrinsic parameters through camera calibration; Mapping the planar projections of the different perspectives to a three-dimensional point cloud based on corresponding depth information; The plane projections of different perspectives are fused to generate a point cloud image.

[0011] Optionally, rendering a plane projection of a preset viewing angle based on the point cloud image and generating a corresponding depth image includes: Project the point cloud image onto a plane projection based on the camera intrinsic parameters and camera extrinsic parameters; Calculate depth information and fill the depth map; The pixel depth is optimized based on the nearest point strategy and the corresponding depth image is generated.

[0012] According to a second aspect of the present disclosure, a three-dimensional reconstruction system based on a single-channel medical endoscope is also provided, the system comprising: An image receiving module is used to obtain single-channel image information and perform preprocessing; A three-dimensional reconstruction module is used to perform three-dimensional reconstruction on the pre-processed single-channel image information and obtain a reconstructed fused image; The image output module is used to match the reconstructed fused image to a corresponding output format and output it.

[0013] According to a third aspect of the present disclosure, an electronic device is also provided, including a memory and a processor, wherein the memory is used to store a computer program; and the processor is used to execute the computer program to implement the method according to the first aspect of the present disclosure.

[0014] According to a fourth aspect of the present disclosure, a computer-readable storage medium is further provided, on which a computer program is stored. When the computer program is executed by a processor, the method according to the first aspect of the present disclosure is implemented.

[0015] According to a fifth aspect of the present disclosure, a computer program product is further provided, comprising a computer program, wherein when the computer program is executed by a processor, the method according to the first aspect of the present disclosure is implemented.

[0016] One beneficial effect of the embodiments of the present disclosure is that the three-dimensional reconstruction method based on a single-channel medical endoscope provided by the present invention can, through the coordination of hardware computing power and software algorithms, reconstruct a single-channel image in two dimensions and generate a corresponding fused image, thereby improving the pixels of the three-dimensional image, thereby improving the stereoscopic effect and ultra-high definition of the three-dimensional image.

[0017] Other features and advantages of the embodiments of the present disclosure will become apparent from the following detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the embodiments of the present disclosure.

[0019] Figure 1 is a schematic diagram of a structure of a system to which an embodiment of the present invention can be applied; Figure 2 is a schematic flow chart of a three-dimensional reconstruction method based on a single-channel medical endoscope according to an embodiment; Figure 3 is a schematic flow chart of a three-dimensional reconstruction method based on a single-channel medical endoscope according to another embodiment; Figure 4 is a schematic diagram of a process for embodying three-dimensional image reconstruction according to an embodiment; Figure 5 is a block diagram of a three-dimensional reconstruction device based on a single-channel medical endoscope according to an embodiment; Figure 6 is a schematic diagram of the hardware structure of an electronic device according to an embodiment. DETAILED DESCRIPTION

[0020] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless otherwise specifically stated.

[0021] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0022] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0023] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0024] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0025] <System Embodiment> Figure 1 1 is a schematic diagram of the composition structure of a 3D reconstruction system based on a single-channel medical endoscope to which a 3D reconstruction method based on a single-channel medical endoscope according to an embodiment can be applied. Figure 1 As shown, the three-dimensional reconstruction system based on a single-channel medical endoscope can include an image receiving module, a computing power module and an image output module. The application scenarios of the endoscope include: general surgery, orthopedics, gynecology, otolaryngology, urology, gastroenterology, neurosurgery, etc.

[0026] As used in the embodiments of the present disclosure, the memory is used to store a computer program, which is used to control the processor to operate according to any embodiment of the three-dimensional reconstruction method based on a single-channel medical endoscope. A technician can design a computer program according to the scheme of the embodiments of the present disclosure. How the computer program controls the processor to operate is well known in the art, so it will not be described in detail here.

[0027] <Method Example> Figure 2 It is a flowchart of a three-dimensional reconstruction method based on a single-channel medical endoscope according to an embodiment.

[0028] like Figure 2 and Figure 3 As shown, the three-dimensional reconstruction method based on a single-channel medical endoscope in this embodiment may include the following steps S210 to S230: S210, obtaining single-channel image information and performing preprocessing.

[0029] Among them, perspective transformation is a method of converting a two-dimensional image into another two-dimensional image according to a certain viewpoint, camera position and projection surface. The basic principle of this process is based on the perspective projection model and is transformed through the homography matrix, which includes: Select the corresponding points in the original image, usually manually or automatically select four corner points or other feature points on the original image, which correspond to the corresponding positions in the new viewpoint; calculate the homography matrix, using the known corresponding points, use the least squares method or direct linear transformation (DLT) method to calculate the homography matrix, which describes the transformation relationship from the original perspective to the target perspective; apply the perspective transformation, use the obtained homography matrix to transform the original image to obtain a new image.

[0030] S220, performing three-dimensional reconstruction on the pre-processed single-channel image information and obtaining a reconstructed fused image.

[0031] Among them, a single two-dimensional image cannot provide enough spatial information. The depth estimation technology can be used to infer the depth information corresponding to each pixel in the image. Through depth estimation, the three-dimensional structure of the scene can be inferred from the two-dimensional image, and then the projection from different viewpoints can be calculated.

[0032] S230, matching the reconstructed fused image to a corresponding output format and outputting the image.

[0033] Among them, the interfaces corresponding to different 3D display devices are also different. It is necessary to convert the fused image into an image format that matches the corresponding interface for different 3D display devices, so as to output the fused image to the display device.

[0034] In the above embodiments, the three-dimensional reconstruction method based on a single-channel medical endoscope provided by the present invention can reconstruct a single-channel image in two dimensions and generate a corresponding fused image through the coordination of hardware computing power and software algorithms, thereby improving the pixels of the three-dimensional image and thus improving the stereoscopic effect and ultra-high definition of the three-dimensional image.

[0035] In one embodiment, the preprocessing step specifically includes: Acquire single-channel image information and a corresponding single-channel image format; match the single-channel image format with a standard image format in a preset standard image library; if the single-channel image format does not match the standard image format, convert the single-channel image format until the single-channel image format matches the standard image format; set the single-channel image information whose single-channel image format matches the standard image format as the preprocessed single-channel image information.

[0036] Among them, the standard image library stores different standard image formats for three-dimensional reconstruction; if the single-channel image format matches the standard image format, the single-channel image information whose single-channel image format matches the standard image format is set as the preprocessed single-channel image information.

[0037] In one embodiment, Figure 4 As shown, the three-dimensional reconstruction of the single-channel image information specifically includes: Generate planar projections of different perspectives based on preprocessed single-channel image information through a preset algorithm; fuse the planar projections of different perspectives to generate a point cloud map; render the planar projection of a preset perspective based on the point cloud map and generate a corresponding depth image; set the depth image to reconstruct the fused image.

[0038] Among them, the goal of the deep learning model is to infer the depth of each pixel from a 2D image. Usually, a convolutional neural network (CNN) is used to handle this task. Several common depth estimation models include: Fully Convolutional Network (FCN): Fully convolutional network replaces the fully connected layer in convolutional neural network (CNN) with convolutional layer, which is suitable for pixel-level prediction tasks of images. FCN classifies or regresses each pixel in the image to predict the depth information of the pixel.

[0039] U-Net: U-Net is a network based on an encoder-decoder structure, which extracts features through the encoder part and samples the feature map back to the original image size in the decoder part. In depth estimation, the decoder of U-Net helps to recover spatial information, so it is very effective in fine-grained depth estimation.

[0040] DepthNet: This is a network specifically designed for depth estimation tasks, usually based on an encoder-decoder structure, combined with skip connections to improve accuracy, especially on low-level details.

[0041] Self-supervised learning methods: These methods do not rely on true depth labels, but instead train the network through self-supervision (e.g., by computing the reconstruction error) to infer depth without depth labels.

[0042] In one embodiment, generating a 2D projection of a predetermined viewpoint from a single 2D image specifically includes: The preprocessed single-channel image information is input into a preset deep learning network to obtain plane projections of different perspectives; the error between the plane projection predicted by the deep learning network and the real plane projection is calculated through the loss function; the weight of the deep learning network is adjusted through the optimization algorithm until the error between the plane projection predicted by the deep learning network and the real plane projection is within the preset error range.

[0043] Among them, generating a 2D projection of a predetermined viewpoint from a single 2D image usually relies on several core algorithms in computer vision and graphics, mainly including perspective transformation, camera calibration, image transformation, etc. This process usually involves inferring the geometric information of the 3D scene from the original image and using this information to generate projections from different viewpoints.

[0044] It is worth mentioning that in order to make the image transformation more accurate, the camera can be calibrated first to obtain the camera intrinsic parameters and extrinsic parameters. These parameters describe the geometric characteristics of the camera (such as focal length, principal point position, distortion, etc.) and the relationship between the camera and the world coordinate system; Camera intrinsic parameters: describes the properties of the camera lens, such as focal length, principal point coordinates, pixel ratio, etc.; Camera extrinsic parameters: describes the position and orientation of the camera relative to the world coordinate system.

[0045] Through camera calibration, it is possible to more accurately connect two-dimensional image points with points in 3D space, thereby better performing viewpoint transformation.

[0046] If a single 2D image cannot provide enough spatial information, depth estimation techniques can be used to infer the depth information corresponding to each pixel in the image. Through depth estimation, the algorithm can try to infer the 3D structure of the scene from the 2D image, and then calculate the projection from different viewpoints. Common depth estimation methods include: Monocular depth estimation: Estimate the depth information of each pixel from a single 2D image through a deep learning model; Stereo vision: Use a binocular camera or images from multiple perspectives for depth recovery; Structured light scanning: Help capture depth information by projecting specific light patterns; After obtaining the 3D structure, it can be rotated, translated, and other transformations, and then projected back to a two-dimensional plane to obtain a new viewpoint image.

[0047] In one embodiment, the plane projections from different perspectives are fused to generate a point cloud image, specifically including: Obtain planar projections of different perspectives and preset three-dimensional point clouds; obtain camera intrinsic parameters and camera extrinsic parameters through camera calibration; map planar projections of different perspectives to three-dimensional point clouds based on corresponding depth information; fuse planar projections of different perspectives to generate a point cloud map.

[0048] Among them, the process of fusing the 2D projection image into the 3D point cloud data and generating a point cloud map involves mapping the depth information or other attributes (such as color) of the two-dimensional image to each point of the three-dimensional point cloud.

[0049] If the 2D image contains depth information (such as an RGBD image), the 2D image can be projected into a 3D point cloud by following the steps below: Depth map generation: If the input image is an RGBD image, each pixel value in the depth map represents the depth of the pixel in the scene (the distance from the camera); pixel to 3D point conversion: According to the camera's intrinsic parameters, each pixel in the 2D image is projected into three-dimensional space. For each pixel (u, v), the depth value d can be converted to the coordinates of the 3D point (X, Y, Z) by the following formula: ; ; ; Among them, c x and c y is the principal point coordinate of the camera (part of the camera intrinsics), f x and f y is the focal length of the camera, and d is the depth value of the corresponding pixel in the depth map.

[0050] When converting between 3D point clouds and 2D images, you may need to convert the point cloud data in one coordinate system to another coordinate system. For example, if the 2D image and 3D point cloud data come from different sensors (such as RGB cameras and lidar), they need to be aligned to the same coordinate system through known camera extrinsics (rotation and translation matrices).

[0051] Point cloud coordinate transformation: If the 2D image and the 3D point cloud are in different coordinate systems, a transformation matrix can be used to transform the point cloud from its original coordinate system to the same coordinate system as the 2D image. The transformation matrix usually includes a rotation matrix R and a translation vector T. The conversion formula is: ; Where R is the rotation matrix and T is the translation vector.

[0052] It is worth mentioning that after camera internal and external parameter alignment, depth projection and color mapping, the 2D image can be combined with the 3D point cloud data to generate an enhanced point cloud map. The usual approach is: Assign a color to each point cloud point: If the 2D image contains RGB information, the RGB value of each pixel in the image can be mapped to the corresponding point in the 3D point cloud through the camera's intrinsic and extrinsic parameters; Fusion of depth information: If the depth map complements the point cloud data, the accuracy of the point cloud can be further enhanced or missing parts can be filled.

[0053] After completing the fusion of 2D projection to 3D point cloud data, the point cloud may need to go through some post-processing steps, such as: Noise reduction: Point cloud data may be affected by noise, and filtering algorithms (such as statistical filtering, voxel grid filtering, etc.) can be applied to smooth and clean the point cloud; Point cloud simplification: If the point cloud data is too dense, algorithms such as the voxel grid method (Voxel Grid) can be used to simplify the point cloud and reduce the number of points; Point cloud reconstruction: If a sparse point cloud is generated during the fusion process, reconstruction algorithms (such as surface reconstruction algorithms, interpolation algorithms, etc.) can be used to generate a more complete 3D model.

[0054] In one embodiment, rendering a plane projection of a preset viewing angle based on the point cloud image and generating a corresponding depth image specifically includes: The point cloud image is projected onto a plane projection based on the camera intrinsic parameters and camera extrinsic parameters; the depth information is calculated and the depth map is filled; the pixel depth is optimized based on the nearest point strategy and the corresponding depth image is generated.

[0055] Among them, during the rendering process, multiple 3D points may be projected onto the same 2D pixel; at this time, the occlusion relationship needs to be handled, that is, points far away from the camera will not affect the depth value of close points; the commonly used strategy is the nearest point strategy, that is, selecting the point closest to the camera as the depth of the pixel.

[0056] Closest point processing: For each pixel, if multiple 3D points are projected to the pixel location, select the closest point (the one with the smallest z c value) as the depth of the pixel.

[0057] It is worth mentioning that the depth information of each 3D point is mapped to the 2D image to obtain a depth map; the depth map is a grayscale image, and the grayscale value is proportional to the depth of each pixel, and is usually normalized; Depth map normalization: The grayscale value of the depth map usually needs to be normalized to a specific range (such as 0 to 255) for display or subsequent processing. The normalization formula can be: ; Among them, d is the depth value, maxd is the maximum depth value, mind is the minimum depth value, and d' is the normalized depth value.

[0058] <Equipment Example 1> Figure 5 FIG. 1 is a schematic block diagram of a three-dimensional reconstruction system based on a single-channel medical endoscope according to an embodiment. Figure 5 As shown, the three-dimensional reconstruction system 500 based on a single-channel medical endoscope may include: an image receiving module 510, a three-dimensional reconstruction module 520 and an image output module 530, specifically including: An image receiving module is used to obtain single-channel image information and perform preprocessing; A three-dimensional reconstruction module is used to perform three-dimensional reconstruction on the pre-processed single-channel image information and obtain a reconstructed fused image; The image output module is used to match the reconstructed fused image to a corresponding output format and output it.

[0059] In one embodiment, the image receiving module is also used to obtain single-channel image information and a corresponding single-channel image format; match the single-channel image format with a standard image format in a preset standard image library, wherein the standard image library stores different standard image formats for three-dimensional reconstruction; if the single-channel image format does not match the standard image format, convert the single-channel image format until the single-channel image format matches the standard image format; and set the single-channel image information whose single-channel image format matches the standard image format as the preprocessed single-channel image information.

[0060] In one embodiment, the three-dimensional reconstruction module is also used to generate planar projections of different perspectives based on preprocessed single-channel image information through a preset algorithm; fuse the planar projections of different perspectives to generate a point cloud map; render the planar projection of a preset perspective based on the point cloud map and generate a corresponding depth image; and set the depth image as a reconstructed fused image.

[0061] In one embodiment, the three-dimensional reconstruction module is also used to input the preprocessed single-channel image information into a preset deep learning network and obtain plane projections of different perspectives; calculate the error between the plane projection predicted by the deep learning network and the real plane projection through a loss function; adjust the weight of the deep learning network through an optimization algorithm until the error between the plane projection predicted by the deep learning network and the real plane projection is within a preset error range.

[0062] In one embodiment, the three-dimensional reconstruction module is also used to obtain planar projections of different perspectives and a preset three-dimensional point cloud; obtain camera intrinsic parameters and camera extrinsic parameters through camera calibration; map the planar projections of different perspectives to the three-dimensional point cloud based on the corresponding depth information; and fuse the planar projections of different perspectives to generate a point cloud map.

[0063] In one embodiment, the three-dimensional reconstruction module is also used to project the point cloud image onto a plane projection based on camera intrinsic parameters and camera extrinsic parameters; calculate depth information and fill the depth map; optimize pixel depth based on the nearest point strategy and generate a corresponding depth image.

[0064] <Equipment Example 2> Figure 6 is a schematic diagram of the hardware structure of an electronic device according to another embodiment.

[0065] like Figure 6As shown, the electronic device 600 includes a processor 610 and a memory 620, wherein the memory 620 is used to store an executable computer program, and the processor 610 is used to execute a method such as any of the above method embodiments under the control of the computer program.

[0066] Each module of the above-mentioned three-dimensional reconstruction system 500 based on a single-channel medical endoscope can be implemented by the processor 610 in this embodiment executing a computer program stored in the memory 620, or can be implemented by other structures, which is not limited here.

[0067] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0068] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. More specific examples (a non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the above. The computer-readable storage medium used herein is not to be interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0069] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0070] The computer program instructions for performing the operation of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect through the Internet). In some embodiments, by using the state information of the computer-readable program instructions to personalize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit may execute the computer-readable program instructions, thereby implementing various aspects of the present invention.

[0071] Various aspects of the present invention are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each box of the flowchart and / or block diagram and the combination of the boxes in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0072] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0073] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0074] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a part of a module, a program segment or an instruction, and a part of the module, a program segment or an instruction contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that it is equivalent to implement it by hardware, implement it by software, and implement it by combining software and hardware.

[0075] Embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the marketplace, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims

1. A three-dimensional reconstruction method based on a single-channel medical endoscope, characterized in that: The method comprises: Obtain single-channel image information and perform preprocessing; Perform three-dimensional reconstruction on the preprocessed single-channel image information and obtain a reconstructed fused image; The reconstructed fused image is matched to a corresponding output format and outputted.

2. The method according to claim 1, characterized in that: The step of obtaining single-channel image information and performing preprocessing includes: Obtain single-channel image information and corresponding single-channel image format; Matching the single-channel image format with a standard image format in a preset standard image library, wherein the standard image library stores different standard image formats for three-dimensional reconstruction; If the single-channel image format does not match the standard image format, the single-channel image format is converted until the single-channel image format matches the standard image format; The single-channel image information whose single-channel image format matches the standard image format is set as the preprocessed single-channel image information.

3. The method according to claim 1, characterized in that The three-dimensional reconstruction of the pre-processed single-channel image information and obtaining the reconstructed fused image includes: Generate plane projections of different viewing angles based on the preprocessed single-channel image information through a preset algorithm; Fusion of the plane projections from different perspectives to generate a point cloud image; Rendering a plane projection of a preset perspective based on the point cloud image and generating a corresponding depth image; The depth image is set as the reconstructed fused image.

4. The method according to claim 3, characterized in that: The method of generating plane projections of different viewing angles based on the preprocessed single-channel image information by using a preset algorithm includes: Inputting the preprocessed single-channel image information into a preset deep learning network and obtaining plane projections of different viewing angles; The error between the plane projection predicted by the deep learning network and the actual plane projection is calculated through the loss function; The weights of the deep learning network are adjusted through an optimization algorithm until the error between the plane projection predicted by the deep learning network and the actual plane projection is within a preset error range.

5. The method according to claim 3, characterized in that: The step of fusing the plane projections from different perspectives to generate a point cloud image includes: Get plane projections and preset 3D point clouds from different perspectives; Obtain camera intrinsic parameters and camera extrinsic parameters through camera calibration; Mapping the planar projections of the different perspectives to a three-dimensional point cloud based on corresponding depth information; The plane projections of different perspectives are fused to generate a point cloud image.

6. The method according to claim 3, characterized in that: The step of rendering a plane projection of a preset viewing angle based on the point cloud image and generating a corresponding depth image includes: Project the point cloud image onto a plane projection based on the camera intrinsic parameters and camera extrinsic parameters; Calculate depth information and fill the depth map; The pixel depth is optimized based on the nearest point strategy and the corresponding depth image is generated.

7. A three-dimensional reconstruction system based on a single-channel medical endoscope, characterized in that: The system comprises: An image receiving module is used to obtain single-channel image information and perform preprocessing; A three-dimensional reconstruction module is used to perform three-dimensional reconstruction on the pre-processed single-channel image information and obtain a reconstructed fused image; The image output module is used to match the reconstructed fused image to a corresponding output format and output it.

8. An electronic device, characterized in that: The invention comprises a memory and a processor, wherein the memory is used to store a computer program; and the processor is used to execute the computer program to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 6 when being executed by a processor.

Images