Three-dimensional imaging method and device based on endoscope and electronic equipment

Abstract

The invention provides a three-dimensional imaging method and device based on an endoscope and electronic equipment. The method comprises the steps that multiple sub-aperture images obtained after the same light path passes through multiple micro lenses located on the same plane are obtained; classifying the corresponding sub-aperture images into a left sub-image set and a right sub-image set according to the positions of the micro-lenses; generating a corresponding first depth image and a corresponding second depth image based on the left sub-image set and the right sub-image set respectively; and generating a target three-dimensional image according to the first depth image and the second depth image. According to the invention, a clear target three-dimensional image can be obtained based on micro-lens simulation binocular imaging.

Description

Technical Field

[0001] The present invention relates to the field of medical devices, and in particular to an endoscope-based three-dimensional imaging method, device and electronic equipment. Background Art

[0002] Traditional medical endoscopes typically utilize a single optical path and a single sensor. Image sensing of the light signal from this single optical path only yields a two-dimensional image, providing only a two-dimensional image of diseased tissue, without providing three-dimensional information. Diagnosis and treatment based solely on two-dimensional images can easily lead to misdiagnosis or missed diagnoses. While some three-dimensional endoscopes have been introduced, these utilize at least two objective lens systems to capture binocular or multi-lens image information and generate a three-dimensional image. These optical components are complex, bulky, and expensive. Summary of the Invention

[0003] In view of the above problems existing in the prior art, the present invention proposes a three-dimensional imaging method, device and electronic equipment based on an endoscope, which mainly solves the problems of complex structure and high cost of the existing endoscope optical components.

[0004] In order to achieve the above-mentioned and other purposes, the technical solutions adopted by the present invention are as follows.

[0005] The present application provides an endoscope-based three-dimensional imaging method, which includes: acquiring multiple sub-aperture images obtained after the same optical path passes through multiple microlenses in the same plane; classifying the corresponding sub-aperture images into a left sub-image set and a right sub-image set according to the positions of the microlenses; generating corresponding first depth images and second depth images based on the left sub-image set and the right sub-image set, respectively; and generating a target three-dimensional image based on the first depth image and the second depth image.

[0006] In one embodiment of the present application, the step of generating a target three-dimensional image based on the first depth image and the second depth image includes: refocusing the left sub-image set and the right sub-image set respectively to generate a left sub-image image sequence and a right sub-image image sequence with different focal planes; determining a first weight of the corresponding pixel point in the left sub-image image sequence based on the first depth image, and determining a second weight of the corresponding pixel point in the right sub-image image sequence based on the second depth image; wherein the first weight and the second weight are used to characterize the clarity of the pixel point in the corresponding image; performing image fusion on the left sub-image image sequence according to the first weight to obtain a left sub-image panoramic depth image, and performing image fusion on the right sub-image image sequence according to the second weight to obtain a right sub-image panoramic depth image; and generating a target three-dimensional image based on the left sub-image panoramic depth image and the right sub-image panoramic depth image.

[0007] In one embodiment of the present application, before classifying the corresponding sub-images into the left sub-image set and the right sub-image set according to the positions of the micro-lenses, the following steps are further included: obtaining the position of each of the micro-lenses; dividing the area where each of the micro-lenses is located into a left area and a right area according to the position of the micro-lens, and classifying the sub-image sets based on the left area and the right area to obtain the left sub-image set and the right sub-image set.

[0008] In one embodiment of the present application, the steps of generating corresponding first depth images and second depth images based on the left sub-image set and the right sub-image set respectively include: matching feature points of any two sub-aperture images in the left sub-image set to obtain a first matching point pair; determining a first depth value of the corresponding feature point based on the disparity of the first matching point pair in the corresponding sub-aperture image; fusing the first depth values of each feature point to obtain the first depth image; and matching feature points of any two sub-aperture images in the right sub-image set to obtain a second matching point pair; determining a second depth value of the corresponding feature point based on the disparity of the second matching point pair in the corresponding sub-aperture image; and fusing the second depth values of each feature point to obtain the second depth image.

[0009] In one embodiment of the present application, the steps of refocusing the left sub-image set and the right sub-image set respectively to generate a left sub-image image sequence and a right sub-image image sequence with different focal planes include: translating each sub-aperture image in the left sub-image set to a window of the same viewing angle based on the frequency domain or the spatial domain to obtain the left sub-image image sequence; and translating each sub-aperture image in the right sub-image set to a window of the same viewing angle based on the frequency domain or the spatial domain to obtain the right sub-image image sequence.

[0010] In one embodiment of the present application, the steps of determining the first weight of the corresponding pixel point in the left sub-image image sequence based on the first depth image, and determining the second weight of the corresponding pixel point in the right sub-image image sequence based on the second depth image include: scoring the clarity of the pixel points in each sub-aperture image in the left sub-image image sequence based on the first depth image; generating the weight of the corresponding pixel point based on the scoring score as the first weight; and scoring the clarity of the pixel points in each sub-aperture image in the right sub-image image sequence based on the second depth image; generating the weight of the corresponding pixel point based on the scoring score as the second weight.

[0011] In one embodiment of the present application, the steps of performing image fusion of the left sub-image sequence according to the first weight to obtain a left sub-image panoramic depth image, and performing image fusion of the right sub-image sequence according to the second weight to obtain a right sub-image panoramic depth image include: comparing the first weights of the pixels at the same position in the left sub-image sequence to use the sub-aperture image with the highest first weight value as the first target sub-image, thereby obtaining multiple first target sub-images; fusing the pixels with the highest clarity in each of the target sub-images to obtain the left sub-image panoramic depth image; and comparing the first weights of the pixels at the same position in the right sub-image sequence to use the sub-aperture image with the highest first weight value as the second target sub-image, thereby obtaining multiple second target sub-images; fusing the pixels with the highest clarity in each of the second target sub-images to obtain the right sub-image panoramic depth image.

[0012] In one embodiment of the present application, the step of generating a target three-dimensional image based on the left sub-image panoramic depth image and the right sub-image panoramic depth image includes: using the left sub-image panoramic depth image and the right sub-image panoramic depth image as binocular images; calculating the depth information of each pixel point based on the disparity of the binocular images, and generating the target three-dimensional image based on the depth information.

[0013] The present application also provides a three-dimensional imaging device, which includes: an optical lens having a main light path arranged therein; a microlens array arranged on the main light path; and an imaging unit for executing the steps of the endoscope-based three-dimensional imaging method.

[0014] The present application also provides an electronic device, which includes: one or more processors; a storage device for storing one or more programs, and when the one or more programs are executed by the one or more processors, the electronic device implements the steps of the endoscope-based three-dimensional imaging method.

[0015] As described above, the present application provides an endoscope-based three-dimensional imaging method, device, and electronic device, which have the following beneficial effects.

[0016] This application uses multiple microlenses in the same plane to generate sub-aperture images with different viewing angles. This simulates binocular imaging through the microlenses, splitting the microlens area into two left and right viewing zones. Depth calculations are performed for each viewing zone, resulting in depth images of the two viewing zones, which can then be used to generate corresponding 3D images of the target. This allows doctors to refer to the target 3D images for accurate diagnosis and treatment. Binocular imaging based on multiple microlenses meets real-time and accuracy requirements, has a simple structure, and can be implemented without significant modifications to the original endoscope, saving costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Schematic diagram of the process of an endoscope-based three-dimensional imaging method in one embodiment of the present application.

[0018] Figure 2 Schematic diagram of the optical structure of an optical lens in one embodiment of the present application.

[0019] Figure 3 FIG. 1 is a schematic structural diagram of a microlens array in an embodiment of the present application.

[0020] Figure 4 FIG. 1 is a schematic diagram of the structure of a three-dimensional imaging device in one embodiment of the present application.

[0021] Figure 5 FIG. 1 is a schematic diagram of the structure of a three-dimensional imaging device in another embodiment of the present application.

[0022] Figure 6 Schematic diagram of the structure of an electronic device in one embodiment of the present application. DETAILED DESCRIPTION

[0023] The following describes the embodiments of the present invention through specific examples. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments. The details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other unless they conflict.

[0024] It should be noted that the illustrations provided in the following embodiments are merely schematic illustrations of the basic concept of the present invention. Therefore, the illustrations only show components related to the present invention and are not drawn according to the number, shape, and size of components in actual implementation. In actual implementation, the type, quantity, and proportion of each component may be changed arbitrarily, and the component layout may also be more complex.

[0025] The inventors have discovered that:

[0026] Traditional two-dimensional endoscopes use a single optical path and single sensor design (such as white light CMOS (Complementary Metal-Oxide Semiconductor) or CCD (Charge-Coupled Device)). They can only provide two-dimensional images of the surface of lesions and cannot obtain depth information. This makes it difficult for doctors to determine key parameters such as the three-dimensional location and invasion depth of the lesion, which can easily lead to misdiagnosis or missed diagnosis.

[0027] Although some three-dimensional endoscopes have been launched, these three-dimensional endoscopes are equipped with at least two objective lens groups to obtain binocular or multi-eye image information and then generate three-dimensional images. Their optical components are complex in structure, bulky in size, and relatively expensive.

[0028] In addition, the traditional depth calculation based on dual cameras simulating human eye parallax requires strict calibration of the dual optical paths. In the miniaturized design of endoscopes, matching errors are easily caused by mechanical deformation, and the imaging resolution is limited by the sensor size, making it difficult to achieve both high resolution and depth accuracy.

[0029] In view of the problems existing in the above-mentioned related technologies, the present application proposes a three-dimensional imaging method, device and electronic equipment based on an endoscope. The technical solution of the present application is described in detail below with reference to specific embodiments.

[0030] See also Figure 1 , Figure 1 The following is a flow chart of an endoscope-based three-dimensional imaging method according to an embodiment of the present application. The three-dimensional imaging method provided in the embodiment of the present application includes the following steps:

[0031] Step S100 , obtaining a plurality of sub-aperture images obtained after the same optical path passes through a plurality of micro-lenses located in the same plane.

[0032] In one embodiment, the optical structure of the optical lens is as follows: Figure 2As shown, the optical lens includes a first lens 1, a second lens 3, a third lens 4, a fourth lens 5, a fifth lens 6, a sixth lens 7, a seventh lens 8, an eighth lens 9 and a ninth lens 10, which are arranged on an axis from the object side to the image side of the main optical path, wherein the first lens 1 is a flat protective lens, and the first lens 1 cooperates with the optical lens housing 12 to prevent other lenses from contacting the external environment, and the optical lens housing 12 is a straight rod structure or a lens barrel structure; the second lens 3 is a lens with negative optical power, whose object side is concave and the image side is concave or flat; an aperture 2 can also be set between the first lens 1 and the second lens 3, and the aperture 2 can adjust the width of the main optical path (that is, the width of the optical path from the first lens 1 to the second lens). The third lens 4 is a lens with positive focal power, whose object side is convex or flat, and whose image side is convex; the fourth lens 5 is a lens with positive focal power, whose object side is flat, and whose image side is convex; the fifth lens 6 is a lens with positive focal power, whose object side is convex, and whose image side is flat; the sixth lens 7 is a lens with positive focal power, whose object side is convex, and whose image side is concave; the seventh lens 8 is a lens with negative focal power, whose object side is convex, and whose image side is concave. The sixth lens 7 and the seventh lens 8 can be bonded together to form a cemented lens. By bonding lenses of different materials and refractive indices, aberrations can be eliminated and image quality can be improved. At the same time, the cemented lens can reduce the number of interfaces between air and glass, thereby reducing reflection loss. In addition, the cemented lens can compensate for the difference in curvature radius of the cemented surfaces, thereby reducing the requirements for processing accuracy and simplifying the processing process. The eighth lens element 9 is a beam splitter. Part of the main optical path passes through the beam splitter and enters the ninth lens element 10. Another part of the light is reflected by the eighth lens element 9, forming a split optical path. For example, this split optical path forms a 90-degree angle with the main optical path, but other angles can be used as needed and are not limited here. The ninth lens element 10 is a protective flat glass. A second image sensor 11 (such as a CMOS, CCD, etc.) can be located on the image side of the ninth lens element 10 to convert the optical signal from the main optical path into an electrical signal, which is then transmitted to a processor to generate a corresponding two-dimensional image. The reflectivity and transmittance of the eighth lens element 9 can be set and adjusted according to actual application requirements and are not limited here. The second lens element 3 uses negative power to diverge the incident parallel light beam, shifting the main plane backward. This achieves a longer working distance than the actual focal length, even with a shorter physical focal length. The intermediate lenses use positive power to focus the diverged light. The combined negative and positive power distribution significantly reduces the incident angle of off-axis light beams, achieving a wider field of view. The seventh lens 8 uses a negative optical power lens to further adjust the optical path, adapt to the image sensor, and avoid edge image quality degradation.

[0033] A micro lens array 13 may be provided on the light splitting path. The structure of the micro lens array 13 is as follows: Figure 3As shown. The microlens array is obtained by arranging a plurality of microlenses in the same plane according to a certain pattern. The object side of each microlens faces the spectroscope to receive the light transmitted by the spectroscopic optical path, and the image side is provided with a tenth lens 14. The tenth lens 14 is a protective flat glass. A first image sensor 15 is provided on the side of the tenth lens 14 away from the microlens array 13, which is used to convert the light focused by each microlens into a corresponding electrical signal, thereby generating a corresponding sub-aperture image. Each microlens has a positive optical focal length. The specific number of microlenses in the microlens array 13 and the size of each microlens can be set and adjusted according to actual application requirements, and are not limited here. Of course, the arrangement of the microlenses in this microlens column 13 can be a symmetrical structure or other asymmetric arrangements can be adopted as needed, and are not limited here. By setting a spectroscope to divide the light path into two, this solution can be implemented by only making minor changes to the original endoscope structure, which can effectively save design and manufacturing costs.

[0034] In one embodiment, the beam splitter may not be used, and the microlens array 13 may be directly set on the main light path. This method provides a relatively simple output image, but it can also achieve the technical solution of the embodiment of the present application.

[0035] Step S110 : classifying the corresponding sub-aperture images into a left sub-image set and a right sub-image set according to the positions of the micro-lenses.

[0036] In one embodiment, before categorizing the corresponding sub-aperture images into the left sub-image set and the right sub-image set based on the position of the microlenses, the following steps are further included: obtaining the position of each microlens; dividing the region where each microlens resides into a left region and a right region based on the position of the microlens, and categorizing the sub-image sets based on the left and right regions to obtain corresponding left and right sub-image sets. Specifically, after the number and arrangement of microlenses in the microlens array 13 are determined, the relative positions of each microlens are also determined. A coordinate system can be established based on the plane where the microlens array 13 resides, and the position information of each microlens in this coordinate system can be recorded. The microlens array can be directly divided into left and right regions, and the number of microlenses in the left and right regions can be the same or different. The sub-aperture images generated by the microlenses in the left region are classified into the left sub-image set, and the sub-aperture images generated by the microlenses in the right region are classified into the right sub-image set. By dividing the left and right regions, binocular imaging is simulated to obtain a three-dimensional image of the target.

[0037] In one embodiment, a method of extracting corresponding sub-aperture images for different microlens positions can be expressed as follows:

[0038]

[0039] Among them, L[x, y, u, v] is a four-dimensional square function, (x, y) is the coordinate of the main lens plane; (u, v) is the coordinate of the micro lens plane, I s,t (x, y) is the extracted sub-aperture image; Δ is the viewing angle step, and i and j are the center coordinates of the corresponding microlens in the microlens array.

[0040] Step S120 : Generate a corresponding first depth image and a corresponding second depth image based on the left sub-image set and the right sub-image set respectively.

[0041] In one embodiment, the steps of generating corresponding first depth images and second depth images based on the left sub-image set and the right sub-image set respectively include:

[0042] Matching feature points of any two sub-aperture images in the left sub-image set to obtain a first matching point pair; determining a first depth value of a corresponding feature point based on the disparity of the first matching point pair in the corresponding sub-aperture image; fusing the first depth values of the feature points to obtain a first depth image; and

[0043] Matching is performed based on the feature points of any two sub-aperture images in the right sub-image set to obtain a second matching point pair; determining a second depth value of the corresponding feature point based on the disparity of the second matching point pair in the corresponding sub-aperture image; and fusing the second depth values of the feature points to obtain a second depth image.

[0044] In one embodiment, the disparity between the matching point pairs may be calculated using a disparity matching cost function, which may be expressed as:

[0045]

[0046] Where d is the disparity, C(x, y, d) is the matching cost, and I 0，0 is the reference subgraph, I s，t is the target subgraph, and W is the matching window.

[0047] After obtaining the parallax, the corresponding depth value can be calculated through the depth solution. The specific depth solution formula can be expressed as:

[0048]

[0049] in, is the virtual baseline length, p is the microlens pitch, and f0 is the microlens focal length.

[0050] It should be pointed out that the right sub-image set can calculate the corresponding depth value in the same way as the left sub-image set, and then fuse the depth value of each feature point in the left sub-image set and the depth value of each feature point in the right sub-image set to obtain the first depth image and the second depth image.

[0051] Step S130: Generate a target three-dimensional image according to the first depth image and the second depth image.

[0052] In one embodiment, generating a target three-dimensional image according to the first depth image and the second depth image includes the following steps:

[0053] Step S131 : refocusing the left sub-image set and the right sub-image set respectively to generate a left sub-image image sequence and a right sub-image image sequence with different focal planes.

[0054] In one embodiment, the left and right sub-image sets may be refocused based on the spatial domain or the frequency domain, respectively. The spatial domain digital refocusing algorithm may be expressed as follows:

[0055]

[0056] Among them, E (α·F) is the refocused image, α is the relative depth, and F is the distance between the pupil plane and the focal plane. The frequency domain digital refocusing algorithm can be expressed as follows:

[0057] P α [L F ](x,y)=E (αF) (x,y)

[0058]

[0059] Among them, P α [L F ] is the refocused image, B α is the shear matrix, F4 is the four-dimensional Fourier transform, and F-2 is the inverse transform of the two-dimensional Fourier transform.

[0060] The left sub-image refocused image obtained based on the above algorithm contains a sequence of left sub-image images with different focal planes, that is, any two images in the left sub-image image sequence have different focal planes; the right sub-image refocused image contains a sequence of right sub-image images with different focal planes, that is, any two images in the right sub-image image sequence also have different focal planes.

[0061] Step S132: Determine a first weight of the corresponding pixel point in the left sub-image sequence based on the first depth image, and determine a second weight of the corresponding pixel point in the right sub-image sequence based on the second depth image; wherein the first weight and the second weight are used to represent the clarity of the pixel point in the corresponding image.

[0062] In one embodiment, because each image in the left and right sub-image sequences has a different focal plane, i.e., different focused areas, the sharp areas in each image also differ. The sharpness of each pixel in the image is calculated, and pixels at the same location in each image in the sequence are scored based on the sharpness. The pixels are then ranked based on the scores, and the weights of the pixels in the different images are determined based on the scores, with higher scores indicating greater weights. This method can determine the weight distribution of each pixel in the left and right sub-image sequences within the corresponding image sequence.

[0063] Step S133 , performing image fusion on the left sub-image sequence according to the first weight to obtain a left sub-image full depth image, and performing image fusion on the right sub-image sequence according to the second weight to obtain a right sub-image full depth image.

[0064] In one embodiment, the steps of performing image fusion on the left sub-image sequence according to the first weight to obtain a left sub-image panoramic depth image, and performing image fusion on the right sub-image sequence according to the second weight to obtain a right sub-image panoramic depth image include: comparing the first weights of the pixels at the same position in the left sub-image sequence to use the sub-aperture image with the highest first weight value as the first target sub-image, thereby obtaining multiple first target sub-images; fusing the pixels with the highest clarity in each first target sub-image to obtain the left sub-image panoramic depth image; and comparing the second weights of the pixels at the same position in the right sub-image sequence to use the sub-aperture image with the highest second weight value as the second target sub-image, thereby obtaining multiple second target sub-images; and fusing the pixels with the highest clarity in each second target sub-image to obtain the right sub-image panoramic depth image.

[0065] Taking the image fusion of the left sub-image sequence as an example, based on the comparison of the first weight values of the same pixel point of each image in the left sub-image sequence, the pixel point with the highest first weight value is selected from each image in the left sub-image sequence, and the image where the pixel point is located is used as the first target sub-image. In this way, multiple first target sub-images containing the clearest pixels can be screened out from the left sub-image sequence; the clearest pixels in each first target sub-image can be further mapped to the same two-dimensional matrix to obtain the corresponding left sub-image full-depth image, which contains the clearest pixels of each image in the left sub-image sequence. In this way, even areas that cannot be seen clearly on a single focal plane can be clearly displayed in the fused image, and a high-definition, large depth of field image can be obtained. The same method can also be used to obtain the right sub-image full-depth image corresponding to the right sub-image sequence, which will not be repeated here.

[0066] Step S134 : generating a target three-dimensional image according to the left sub-image full-depth image and the right sub-image full-depth image.

[0067] The obtained left and right sub-image panoramic depth images are used as the output binocular image. Based on the binocular image, depth information such as the distance to the target object (such as lesions) in the image can be determined, which facilitates the construction of a three-dimensional image based on the binocular image, thereby assisting doctors in diagnosis and treatment. At the same time, combined with the two-dimensional image of the main light path, it provides more comprehensive and reliable diagnostic auxiliary information.

[0068] Based on the technical solutions of the above embodiments of the present application, each lens in the optical imaging system is placed in a lens barrel, and the lens barrel is connected to a handle. The microlens array and the processor for processing each sub-aperture image to obtain a target three-dimensional image can be placed in the handle. Only the handle structure needs to be adjusted, and no changes need to be made to the lens barrel. The new handle can be adapted to the original lens barrel structure, greatly reducing the hardware complexity and cost; the depth solution, focus and fusion processes are respectively executed based on the left and right sub-image sets, and the target three-dimensional image finally outputted not only has a large depth of field characteristic (covering multi-focal plane tissues), but also provides accurate parallax information, solving the pain point of limited depth of field of traditional 3D endoscopes.

[0069] See also Figure 4 , Figure 4 This is a schematic diagram of the architecture of a three-dimensional imaging device in one embodiment of the present application. The three-dimensional imaging device provided in this embodiment of the present application includes:

[0070] an optical lens 40, in which a main light path is provided;

[0071] A microlens array 13 is provided on the optical path having the main optical path;

[0072] The imaging unit 41 is configured to execute the steps of the aforementioned endoscope-based three-dimensional imaging method.

[0073] The optical lens 40 may include the aforementioned first to seventh lenses and the ninth lens. The microlens array 13 may be disposed at the end of the optical lens 40. The optical lens 40 may be a lens barrel structure. The imaging unit 41 may be disposed in the endoscope camera handle.

[0074] See also Figure 5 As an optional embodiment, the three-dimensional imaging device further includes a light splitting unit 42, which is disposed at the end of the optical lens 40 to split the main light path into two light paths for transmission, one of which is an imaging light path and the other is a light splitting light path;

[0075] A microlens array 13 is provided on the light splitting path;

[0076] The imaging unit 41 is used to receive the imaging light path to obtain a two-dimensional image; it is also used to receive the light path on the light-emitting side of each microlens in the microlens array 13 to obtain multiple sub-aperture images, and generate a target three-dimensional image corresponding to the two-dimensional image based on the sub-aperture images.

[0077] In one embodiment, the light splitting unit 42 is an eighth lens 9 , which can be disposed between the seventh lens 8 and the ninth lens 10 .

[0078] The imaging unit 41 includes a first image sensor 15, a second image sensor 11 and a processor. The second image sensor 11 is used to receive the imaging light path to obtain a two-dimensional image. The first image sensor 15 is used to receive the light path on the light-emitting side of each microlens in the microlens array 13 to obtain multiple sub-aperture images. The processor is used to generate a target three-dimensional image corresponding to the two-dimensional image based on the sub-aperture image.

[0079] The specific execution process of the imaging unit 41 has been described in detail in the aforementioned method embodiment and will not be repeated here.

[0080] See also Figure 6 , Figure 6 A schematic diagram of the structure of a computer system suitable for implementing an electronic device according to an embodiment of the present application is shown. Figure 6 The computer system 600 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0081] like Figure 6 As shown, computer system 600 includes a central processing unit 601, which can perform various appropriate actions and processes according to programs stored in read-only memory 602 or programs loaded from storage unit 608 into random access memory 603, such as executing the methods described in the above embodiments. Random access memory 603 also stores various programs and data required for system operation. Central processing unit 601, read-only memory 602, and random access memory 603 are connected to each other via bus 604. Input / output interface 605 is also connected to bus 604.

[0082] The following components are connected to the input / output interface 605: an input section 606 including a keyboard, a mouse, and the like; an output section 607 including devices such as a CRT (Cathode Ray Tube), an LCD (Liquid Crystal Display), and a speaker; a storage section 608 including devices such as a hard disk; and a communication section 609 including a network interface card such as a LAN (Local Area Network) card or a modem. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output interface 605 as needed. Removable media 611, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 610 as needed, so that computer programs read therefrom can be installed into the storage section 608 as needed.

[0083] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 609 and / or installed from a removable medium 611. When the computer program is executed by the central processing unit 601, the various functions defined in the system of the present application are performed.

[0084] The computer-readable medium shown in the embodiments of the present application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. The computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable Read Only Memory), flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), optical storage device, magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable computer program. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. A computer program embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0085] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions and operations of the devices, methods and computer program products according to various embodiments of the present application. Among them, each box in the flowchart or block diagram can represent a module, program segment, or part of the code, and the above-mentioned module, program segment, or part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0086] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. In some cases, the names of these units do not constitute limitations on the units themselves.

[0087] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the present invention. Anyone skilled in the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by one of ordinary skill in the art without departing from the spirit and technical principles disclosed herein are intended to be covered by the claims of the present invention.

Claims

1. A three-dimensional imaging method based on endoscope, characterized in that: The method comprises: Acquire multiple sub-aperture images obtained after the same optical path passes through multiple microlenses in the same plane; Classifying the corresponding sub-aperture images into a left sub-image set and a right sub-image set according to the position of the micro-lens; Generating a corresponding first depth image and a second depth image based on the left subimage set and the right subimage set respectively; A target three-dimensional image is generated according to the first depth image and the second depth image.

2. The endoscope-based three-dimensional imaging method according to claim 1, characterized in that: The step of generating a target three-dimensional image according to the first depth image and the second depth image comprises: Refocusing the left sub-image set and the right sub-image set respectively to generate a left sub-image image sequence and a right sub-image image sequence with different focal planes; Determining a first weight of a corresponding pixel point in the left sub-image sequence based on the first depth image, and determining a second weight of the corresponding pixel point in the right sub-image sequence based on the second depth image; wherein the first weight and the second weight are used to represent the clarity of the pixel point in the corresponding image; Performing image fusion on the left sub-image sequence according to the first weight to obtain a left sub-image panoramic depth image, and performing image fusion on the right sub-image sequence according to the second weight to obtain a right sub-image panoramic depth image; A target three-dimensional image is generated according to the left sub-image panoramic depth image and the right sub-image panoramic depth image.

3. The endoscope-based three-dimensional imaging method according to claim 1, characterized in that: Before classifying the corresponding sub-images into the left sub-image set and the right sub-image set according to the positions of the micro-lenses, the following steps are also included: Obtaining the position of each of the microlenses; The area where each microlens is located is divided into a left area and a right area according to the position of the microlens, and the sub-image sets are classified based on the left area and the right area to obtain the left sub-image set and the right sub-image set.

4. The endoscope-based three-dimensional imaging method according to claim 1, characterized in that: The step of generating corresponding first depth images and second depth images based on the left sub-image set and the right sub-image set respectively comprises: Matching feature points of any two sub-aperture images in the left sub-image set to obtain a first matching point pair; determining a first depth value of a corresponding feature point based on the disparity of the matching point pair in the corresponding sub-aperture image; fusing the first depth values of the feature points to obtain the first depth image; and Matching is performed based on feature points of any two sub-aperture images in the right sub-image set to obtain a second matching point pair; determining a second depth value of the corresponding feature point based on the disparity of the second matching point pair in the corresponding sub-aperture image; and fusing the second depth values of the feature points to obtain a second depth image.

5. The endoscope-based three-dimensional imaging method according to claim 2, characterized in that: The step of refocusing the left sub-image set and the right sub-image set respectively to generate a left sub-image image sequence and a right sub-image image sequence with different focal planes comprises: Shifting each sub-aperture image in the left sub-image set to a window with the same viewing angle based on the frequency domain or the spatial domain to obtain the left sub-image image sequence; and Each sub-aperture image in the right sub-image set is translated to a window of the same viewing angle based on the frequency domain or the spatial domain to obtain the right sub-image image sequence.

6. The endoscope-based three-dimensional imaging method according to claim 2, characterized in that: The step of determining a first weight of a corresponding pixel point in the left sub-image sequence according to the first depth image, and determining a second weight of the corresponding pixel point in the right sub-image sequence according to the second depth image comprises: Scoring the clarity of pixels in each sub-aperture image in the left sub-image sequence according to the first depth image; generating a weight of the corresponding pixel based on the scoring score as the first weight; and The clarity of the pixels in each sub-aperture image in the right sub-image sequence is scored according to the second depth image; and the weight of the corresponding pixel is generated based on the scoring score as the second weight.

7. The endoscope-based three-dimensional imaging method according to claim 1, characterized in that: The steps of fusing the left sub-image sequence according to the first weight to obtain a left sub-image panoramic depth image, and fusing the right sub-image sequence according to the second weight to obtain a right sub-image panoramic depth image include: Comparing the first weights of pixels at the same position in the left sub-image sequence to select the sub-aperture image with the highest first weight value as the first target sub-image, thereby obtaining a plurality of the first target sub-images; fusing the pixels with the highest definition in each of the first target sub-images to obtain the left sub-image full depth image; and The second weights of the pixels at the same position in the right sub-image sequence are compared to take the sub-aperture image with the highest second weight value as the second target sub-image, thereby obtaining a plurality of the second target sub-images; the pixels with the highest clarity in each of the second target sub-images are fused to obtain the right sub-image full depth image.

8. The endoscope-based three-dimensional imaging method according to claim 1, characterized in that: The step of generating a target three-dimensional image according to the left sub-image panoramic depth image and the right sub-image panoramic depth image comprises: Taking the left sub-image full-depth image and the right sub-image full-depth image as binocular images; Depth information of each pixel is calculated according to the disparity of the binocular image, and the target three-dimensional image is generated according to the depth information.

9. A three-dimensional imaging device, characterized in that: The three-dimensional imaging device comprises: an optical lens having a main light path disposed therein; a microlens array, which is arranged on the main light path; An imaging unit, configured to execute the steps of the endoscope-based three-dimensional imaging method according to any one of claims 1 to 8.

10. An electronic device, characterized in that: The electronic device comprises: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the electronic device to implement the steps of the endoscope-based three-dimensional imaging method as described in any one of claims 1 to 8.

Images