A microscopic imaging device and method with depth reconstruction effect

Abstract

A microscopic imaging device and method with depth reconstruction effect, which acquires images of an object to be measured through an image acquisition component to obtain a first image and a second image of the object to be measured; according to the inverse filters corresponding to multiple preset object distances of a first shooting component and a second shooting component, and the transformation matrix between the first shooting component and the second shooting component at each object distance, the offset loss amount of any pixel point at the corresponding object distance is obtained; then, according to the offset loss amounts of any pixel point corresponding to all object distances, the depth value of this pixel point is obtained, thereby completing depth reconstruction to obtain a super-depth-of-field image and a depth map. By adding a specially designed phase mask, an optical super-depth-of-field effect is achieved; and since the depth reconstruction process does not require multiple shootings or the use of a displacement platform, a super-depth-of-field image and a depth image can be obtained only through one shooting of a dual camera, which is simpler, more efficient and easier to implement.

Description

Technical Field

[0001] The present invention relates to the field of microscopic three-dimensional imaging, and particularly to a microscopic imaging device and method with depth reconstruction effect. Background Art

[0002] With the development of various fields such as mold design, precision machining, product inspection, and industrial measurement, the traditional two-dimensional image sensor technology can no longer meet the needs of production scenarios, and the demand for three-dimensional measurement technology is growing rapidly. Compared with the two-dimensional image sensor technology, the three-dimensional measurement technology has an additional dimension of depth information. Moreover, since depth itself is less sensitive to factors such as posture and illumination, the three-dimensional measurement technology has greater advantages in object recognition and measurement tasks.

[0003] In the field of microscopic three-dimensional imaging, due to the extremely small depth of field of the objective lens, it is often necessary to rely on a motion structure and image synthesis to obtain the effect of super-depth of field and the depth information of the object surface. The current mainstream three-dimensional reconstruction schemes for microscopes are interferometers or multi-focal plane focus changes. In the reconstruction, a displacement platform needs to be used to move along the depth of field direction, and multiple shots at different positions are used to synthesize the target image. However, the use of the displacement platform and the sampling at different times will introduce systematic perturbation errors, resulting in errors in the obtained measurement results. Summary of the Invention

[0004] The main technical problem to be solved by the present invention is how to achieve depth reconstruction without using a displacement platform and multiple samplings.

[0005] According to a first aspect, in one embodiment, a microscopic imaging device with depth reconstruction effect is provided, including a microscopic objective lens, a tube lens, a light source, a lifting table, and a plurality of beam splitting devices, and further including:

[0006] A phase mask, which is located between the microscopic objective lens and a beam splitting device;

[0007] An image acquisition component, which at least includes a first shooting component and a second shooting component; the first shooting component and the second shooting component each have a corresponding inverse filter at a plurality of preset object distances, and at each object distance, there is also a corresponding transformation matrix between the first shooting component and the second shooting component;

[0008] And a processor, configured to:

[0009] Control the lifting table to move in the vertical direction according to a preset moving step size;

[0010] Obtain the inverse filters corresponding to the first shooting component and the second shooting component at a plurality of preset object distances, and the transformation matrix between the first shooting component and the second shooting component at each object distance;

[0011] Control the first imaging component and the second imaging component to simultaneously acquire images of the object to be measured, obtaining a first image and a second image;

[0012] For any object distance: Filter the first image according to the inverse filter corresponding to the first imaging component at this object distance to obtain a third image corresponding to this object distance; Filter the second image according to the inverse filter corresponding to the second imaging component at this object distance to obtain a fourth image corresponding to this object distance; And perform a rigid transformation on the fourth image according to the transformation matrix between the first imaging component and the second imaging component at this object distance to obtain a fifth image corresponding to this object distance; According to the pixel values corresponding to any pixel point in the obtained third image and fifth image, obtain the offset loss amount of the any pixel point corresponding to this object distance;

[0013] Obtain the depth value of the any pixel point according to the offset loss amounts of the any pixel point corresponding to all object distances, and the depth values of all pixel points constitute the depth map corresponding to the object to be measured, completing depth reconstruction.

[0014] According to a second aspect, an embodiment provides a microscopic imaging method with a depth reconstruction effect, including:

[0015] Acquire images of the object to be measured through an image acquisition component, where the image acquisition component includes at least a first imaging component and a second imaging component, and the first imaging component and the second imaging component simultaneously acquire images to obtain a first image and a second image of the object to be measured; Wherein, the first imaging component and the second imaging component each have a corresponding inverse filter at multiple preset object distances, and at each object distance, there is also a corresponding transformation matrix between the first imaging component and the second imaging component;

[0016] For any object distance: Filter the first image according to the inverse filter corresponding to the first imaging component at this object distance to obtain a third image corresponding to this object distance; Filter the second image according to the inverse filter corresponding to the second imaging component at this object distance to obtain a fourth image corresponding to this object distance; And perform a rigid transformation on the fourth image according to the transformation matrix between the first imaging component and the second imaging component at this object distance to obtain a fifth image corresponding to this object distance; According to the pixel values corresponding to any pixel point in the obtained third image and fifth image, obtain the offset loss amount of the any pixel point corresponding to this object distance;

[0017] Obtain the depth value of the any pixel point according to the offset loss amounts of the any pixel point corresponding to all object distances, and the depth values of all pixel points constitute the depth map corresponding to the object to be measured.

[0018] A microscopic imaging device and method with depth reconstruction effect according to the above embodiments, by adding a specially designed phase mask in the optical system, makes the point spread function after imaging remain unchanged in shape within a certain range, greatly expanding the depth of field of the optical system, achieving the super-depth-of-field effect optically, and improving the phenomenon of extremely small depth of field caused by expanding the aperture in ordinary microscopes; during the depth reconstruction process, first, based on the inverse filters corresponding to different shooting components and the transformation matrices between different shooting components obtained during the calibration process, an implicit correspondence relationship between the offset differences between different object distances and the point spread functions corresponding to different shooting components is established, thereby recording the changes in the corresponding offsets of the point spread functions caused by the object distance changes for different shooting components; then, during the actual use process, according to the offset loss amounts corresponding to the object to be measured at different object distances, the true depth of the surface of the object to be measured is deduced inversely, thereby completing the depth reconstruction and obtaining the corresponding super-depth-of-field image; since the depth reconstruction process no longer requires multiple shootings or the use of a displacement platform, the super-depth-of-field image and the depth image can be obtained only by one shooting of the dual cameras, which is more simple and efficient; and by filtering the image with the inverse filters corresponding to different object distances and restoring it to the original image, the triangular ghosting generated by any point passing through the phase mask can be eliminated, thereby improving the problem of imaging artifacts existing during the reconstruction with traditional phase masks; in addition, since this application does not require special auxiliary lighting and other optical devices except the phase mask are standard products, this application is also easier to implement. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a processing flow chart of a processor in a microscopic imaging device with depth reconstruction effect;

[0020] Figure 2 It is a method flow chart of a microscopic imaging method with depth reconstruction effect;

[0021] Figure 3 It is a structural schematic diagram of a microscopic imaging device with depth reconstruction effect;

[0022] Figure 4 It is a shape schematic diagram of the PSF function after inserting the phase mask;

[0023] Figure 5 It is a schematic diagram of the offset change conditions corresponding to different shooting components at different object distances;

[0024] Figure 6 It is the image corresponding to a partial area of the standard object in the first standard image;

[0025] Figure 7 It is a schematic diagram of the depth reconstruction result;

[0026] Figure 8 The diffusion pattern formed by a certain pinhole as the standard object.

[0027] Reference numerals: 11 - microscope objective; 12 - tube lens; 13 - light source; 14 - lifting table; 15 - beam splitter; 16 - phase mask; 17 - image acquisition component. Specific embodiments

[0028] The present invention will be further described in detail below in conjunction with the accompanying drawings through specific embodiments. Similar elements in different embodiments are labeled with related similar reference numerals. In the following embodiments, many detailed descriptions are provided to enable a better understanding of the present application. However, those skilled in the art can easily recognize that some of the features can be omitted in different situations, or can be replaced by other elements, materials, or methods. In some cases, some operations related to the present application are not shown or described in the specification to avoid overwhelming the core part of the present application with excessive descriptions. For those skilled in the art, it is not necessary to describe these related operations in detail, and they can fully understand the related operations based on the descriptions in the specification and the general technical knowledge in the art.

[0029] In addition, the features, operations, or characteristics described in the specification can be combined in any suitable manner to form various embodiments. At the same time, the steps or actions in the method description can also be reordered or adjusted in an obvious manner by those skilled in the art. Therefore, the various sequences in the specification and the drawings are only for clearly describing a certain embodiment and do not mean that they are the necessary sequences, unless it is stated that a certain sequence must be followed.

[0030] The serial numbers assigned to the components herein, such as "first", "second", etc., are only used to distinguish the described objects and do not have any sequential or technical meaning. The "connection" and "coupling" mentioned in the present application, unless otherwise specified, both include direct and indirect connection (coupling).

[0031] The phase mask belongs to the wavefront coding technology. By modulating the pupil phase distribution function, it can extend the depth of field of the optical system and has the effect of greatly expanding the depth of field of the optical system. Its point spread function has almost exactly the same shape, size, and blur distribution at different object distances, although there is a certain offset.

[0032] Embodiments of the present invention aim to use a specially designed optical element (i.e., a phase mask) to make the shapes of the modulation transfer function and the point spread function of an optical system insensitive to the object distance. While greatly expanding the depth of field of the optical system, based on the imaging characteristics of a microscopic imaging system, two cameras can obtain the depth information of the object surface after one shot, so as to obtain results with super-depth-of-field and depth reconstruction effects without using a displacement platform and multiple samplings.

[0033] Some embodiments provide a microscopic imaging device with a depth reconstruction effect. For its structural schematic diagram, please refer to Figure 3 , which includes a microscopic objective lens 11, a tube lens 12, a light source 13, a lifting platform 14, and multiple beam splitting devices 15. It also includes a phase mask 16, an image acquisition component 17, and a processor. In the microscopic imaging device of this embodiment, the light generated by the incoherent light source 13 irradiates the object to be measured. After passing through the microscopic objective lens 11, it is modulated by the phase mask 16, and then after passing through the beam splitting devices 15, it is imaged by two identical image acquisition components 17 with different image distances. The following is a specific description:

[0034] The phase mask 16 in this embodiment is a bicubic phase mask. It is located between the microscopic objective lens 11 and a beam splitting device 15. By placing a phase mask 16 with a special surface shape at the pupil of the microscopic objective lens 11, the light phase distribution function at the pupil is modulated, and the shapes of the modulation transfer function and the point spread function of the optical system are made insensitive to the object distance. At the same time, the depth of field of the optical system is greatly expanded using wavefront coding technology. The bicubic constraint satisfied by the phase distribution function at the pupil after optical optimization is:

[0035] φ ( u , v ) = 2 πα ( u 3 + v 3 )

[0036] Where, φ ( u , v ) represents the phase distribution function, u , v are the position coordinates on the pupil, α is the gain coefficient for controlling the surface bending strength, which is optimized by optical software (such as Zemax);

[0037] The image acquisition component 17 in this embodiment includes at least a first shooting component and a second shooting component; among them, the first shooting component and the second shooting component each have a corresponding inverse filter at multiple preset object distances, and at each object distance, there is also a corresponding transformation matrix between the first shooting component and the second shooting component, which is used to convert the images simultaneously acquired by the first shooting component and the second shooting component into the same coordinate system;

[0038] It should be noted that the optical system after adding the phase mask will have the following characteristics: on the one hand, the modulation transfer function (i.e., the MTF function) of the optical system imaging will not drop to 0, so deconvolution can be used to perform the inverse transformation of image degradation, that is, to restore the output image of the optical system to the original input image; on the other hand, the shape of the light intensity distribution corresponding to the point spread function (PSF) of the optical system will not change within a large object distance range (ultra-depth of field range), and its shape is a triangular diffraction fringe offset from the symmetry center; within this ultra-depth of field range, although the shape of the light intensity distribution will not change, the position of the imaging spot of a single point light source after passing through the phase mask will shift, that is, the center position of the spot will shift, and the shift amount will change with the change of the working distance (or object distance), where the function formed by the shift amount and the working distance δ ( w ) has a curve shape approximately like a parabola, and the change of the shift amount corresponding to different shooting components at different object distances is as shown in Figure 5 shown, where δ represents the shift amount, and w represents the working distance / object distance;

[0039] According to the imaging characteristics of the bicubic phase mask, if two shooting components are fixedly placed at different image distances so that the difference in the back focal length is kept fixed, then the difference between the shift amounts corresponding to the two shooting components at the same object distance and the function Δ formed by the corresponding working distance δ ( w ) is a monotonic continuous function, as shown in Figure 5 shown. At this time, if the corresponding relationship between the shift amount difference and the working distance can be obtained in advance, the corresponding working distance can be restored according to the calculated shift amount difference during real-time shooting;

[0040] In this embodiment, the process of obtaining the corresponding relationship between the shift amount difference and the working distance is called the calibration process. In this calibration process, the inverse filters corresponding to the first shooting component and the second shooting component at multiple preset object distances are obtained, as well as the transformation matrix between the first shooting component and the second shooting component at each object distance;

[0041] During this calibration process, the processor controls the lifting platform 14 to move in the vertical direction according to a preset moving step. After each movement of the lifting platform 14, the image acquisition component 17 is controlled to acquire images of a preset standard object, so as to obtain images corresponding to the standard object at multiple preset object distances. The standard object is placed on the lifting platform and is an opaque flat plate with equally spaced ultra-fine pinholes. A backlight source is set for this opaque flat plate to assist illumination to illuminate the ultra-fine pinholes on the opaque flat plate. Then the specific calibration process of this embodiment is as follows:

[0042] First, an opaque flat plate with equally spaced ultra-fine pinholes is used as the preset standard object, which is placed on a lifting platform 14 equipped with a stepper motor, and a backlight source is set so that the pinholes on the opaque flat plate are illuminated. The lifting direction of the lifting platform 14 is perpendicular to the optical axis of the objective lens. Then, starting from the position where the nominal working distance is 0, the lifting platform 14 is controlled to move in the vertical direction according to the preset moving step. Exemplarily, the preset moving step can be set to 1 μm , corresponding to each upward movement of the lifting platform 14 by 1 μm ; After each movement of the lifting platform 14, the image acquisition component 17 is used to acquire images of the preset standard object until it exceeds the range of the working distance, and the two images acquired by the first shooting component and the second shooting component of the standard object at the same time after each movement and the corresponding position of the lifting platform are recorded, so as to obtain two images corresponding to the standard object at different object distances. For any object distance: the two images obtained after the first shooting component and the second shooting component acquire images of the standard object at this object distance are respectively called the first standard image and the second standard image;

[0043] In this embodiment, after the pinholes on the opaque flat plate are illuminated by the backlight source, the light rays originally concentrated at one point will form an extended light spot on the image plane. As Figure 4 shown, the light intensity distribution of a point after passing through the phase mask shows a triangular trailing shadow. At this time, the light intensity distribution of this light spot is the PSF function. In this embodiment, through an optical software (such as Zemax), the corresponding PSF functions are respectively calculated according to the diffusion patterns formed by the pinholes on the opaque flat plate in the first standard image and the second standard image. The image corresponding to a partial area of the standard object in the first standard image is as Figure 6 shown, Figure 8 is Figure 6 a partial enlarged view of, which shows the diffusion pattern formed by a certain pinhole of the standard object;

[0044] Since the modulation transfer function of the system has no zeros and the shape of the PSF function is insensitive to the field position, the inverse transform can be solved by Fourier transform, and then the inverse filters corresponding to the obtained PSF functions can be constructed respectively by Wiener filtering. That is, for any object distance, the inverse filter corresponding to the first imaging component at this object distance is obtained according to the PSF function corresponding to the first standard image; the inverse filter corresponding to the second imaging component at this object distance is obtained according to the PSF function corresponding to the second standard image; thus, the inverse filters corresponding to the first imaging component and the second imaging component at this object distance are obtained.

[0045] Then, the first standard image and the second standard image are respectively filtered by the obtained inverse filters to restore the degraded images containing point spread function noise to clear original images. Among them, the first standard image is filtered by the inverse filter corresponding to the first imaging component at this object distance to obtain a third standard image; the second standard image is filtered by the inverse filter corresponding to the second imaging component at this object distance to obtain a fourth standard image; thus, the first standard image and the second standard image are restored to the original images containing the ultra-fine pinhole array on the standard object. That is, at this time, the third standard image and the fourth standard image contain a dot array with equally spaced distribution, and each dot corresponds to an ultra-fine pinhole on the standard object.

[0046] The center point coordinates of each dot in the third standard image and the fourth standard image are respectively obtained. For example, the center point coordinates of each dot in the third standard image and the fourth standard image are obtained by a dot extraction algorithm. Since each ultra-fine pinhole on the standard object has a corresponding dot in the third standard image and the fourth standard image, the transformation matrix can be obtained by linear least squares according to the center point coordinates of the corresponding dots in the third standard image and the fourth standard image, that is:

[0047]

[0048] Among them, cam 1 represents the first imaging component; cam 2 represents the second imaging component; is the matrix composed of the center point coordinates of all dots in the third standard image corresponding to the first imaging component when the object distance is w ; is the matrix composed of the center point coordinates of all dots in the fourth standard image corresponding to the second imaging component when the object distance is w ; T w is the transformation matrix between the first imaging component and the second imaging component when the object distance is w . This transformation matrix is a rigid body transformation matrix including rotation and translation.

[0049] The center point coordinates of the corresponding dot in the images captured by the dual cameras can also be used to obtain the change in the offset value caused by the object distance transformation. Therefore, during the process of converting the images corresponding to the first shooting component and the second shooting component into the same coordinate system, the transformation matrix will implicitly include the offset difference between different shooting components, that is, the transformation matrices corresponding to different object distances can implicitly establish the corresponding relationship between different object distances and the corresponding offset differences.

[0050] In this embodiment, the processor is further configured to perform depth reconstruction to obtain a depth map and a super-depth-of-field image. Among them, the processing flow of the processor for depth reconstruction is as Figure 1 shown, including:

[0051] Step S100: Obtain the inverse filters corresponding to the first shooting component and the second shooting component at multiple preset object distances, and at each object distance, obtain the transformation matrix between the first shooting component and the second shooting component.

[0052] Obtain the inverse filters corresponding to the first shooting component and the second shooting component at multiple preset object distances obtained during the calibration process, and at each object distance, obtain the transformation matrix between the first shooting component and the second shooting component.

[0053] Step S110: Control the first shooting component and the second shooting component to simultaneously perform image acquisition on the object to be measured, and obtain a first image and a second image.

[0054] It is recorded that the object to be measured is placed on a plane with an object distance of m where m is unknown. That is, in this embodiment, it is only necessary to ensure that the object to be measured is fixed within the field of view of the shooting component, and there is no need to know the specific object distance at which the object to be measured is placed.

[0055] Step S120: Obtain the offset loss amount corresponding to any pixel point at each object distance according to the obtained first image and second image.

[0056] Since the function Δ δ ( w ) formed by the offset difference and the corresponding working distance is a monotonically continuous function, for any pixel point p within the imaging range of the shooting component, there should be a unique object distance z satisfying:

[0057]

[0058] where I cam1 is the image captured by the first shooting component, corresponding to the first image in this embodiment; I cam2is the image captured by the second shooting component, corresponding to the second image in this embodiment; represents that when the object distance is z the inverse filter corresponding to the first shooting component; represents that when the object distance is z the inverse filter corresponding to the second shooting component; T z represents that when the object distance is z the transformation matrix between the first shooting component and the second shooting component; "|| ||" represents the L2 norm; "·" represents the dot product; "*" represents the convolution operation;

[0059] At this time, z is the depth value corresponding to this pixel point p In this embodiment, by traversing multiple preset object distances stored in advance, image transformation is performed based on the inverse filters and transformation matrices corresponding to different object distances. At this time, the difference between the image transformation results is the difference caused by the offset between the point spread functions of the corresponding points of different shooting components. In this embodiment, the degree of difference between the corresponding pixel points in the image transformation results is called the offset loss amount of this pixel point. Then, the depth value of this pixel point is obtained by minimizing the degree of difference corresponding to this pixel point, that is, the object distance that minimizes the offset loss amount of this pixel point is used as the depth value of this pixel point;

[0060] Then for any object distance: the first image is filtered according to the inverse filter corresponding to the first shooting component at this object distance to obtain the third image corresponding to this object distance; the second image is filtered according to the inverse filter corresponding to the second shooting component at this object distance to obtain the fourth image corresponding to this object distance;

[0061] Since in this embodiment, the transformation matrix is to convert the image after the filtering process corresponding to the second shooting component to the coordinate system where the image after the filtering process corresponding to the first shooting component is located. Therefore, here, the fourth image is rigidly transformed according to the transformation matrix between the first shooting component and the second shooting component at this object distance to obtain the fifth image corresponding to this object distance;

[0062] At this time, the fifth image and the third image are in the same coordinate system, and the difference between them is the difference caused by the offset between the point spread functions of the corresponding points of different shooting components. According to the pixel values corresponding to any pixel point in the obtained third image and fifth image, the degree of difference between this pixel point in the third image and the fifth image is evaluated, and the obtained degree of difference is the offset loss amount of this pixel point at this object distance;

[0063] For example, for any pixel, obtain the grayscale values of the pixel on different channels in the third image and the fifth image respectively, calculate the norm according to the difference between the corresponding grayscale values of the pixel on different channels, such as calculating the L2 norm, and use the obtained norm as the offset loss amount corresponding to the pixel at the object distance.

[0064] Step S130: Obtain the depth value of any pixel according to the offset loss amounts corresponding to all object distances of the pixel. The depth values of all pixels constitute the depth map corresponding to the object to be measured, and depth reconstruction is completed.

[0065] For any pixel, obtain the minimum value among the offset loss amounts corresponding to the pixel at all preset object distances, use the object distance corresponding to the obtained minimum value as the depth value of the pixel, and the depth values of all pixels constitute the depth map corresponding to the object to be measured, thereby completing depth reconstruction; the result obtained after depth reconstruction of a block object in this embodiment is as Figure 7 shown.

[0066] In addition, this embodiment can also obtain a super-depth-of-field image. Among them, for any pixel in the super-depth-of-field image, obtain the third image and the fifth image corresponding to the object distance with the depth value of the pixel, and in the obtained third image and fifth image, calculate the average value of the corresponding grayscale values of the pixel on each channel, and use the obtained average value as the grayscale value of the pixel on the corresponding channel in the super-depth-of-field image.

[0067] In this embodiment, a specially designed phase mask is added to the optical system, so that the point spread function after imaging remains unchanged in shape within a certain range, greatly expanding the depth of field of the optical system, achieving the super-depth-of-field effect optically, and improving the phenomenon of extremely small depth of field caused by enlarging the aperture in a common microscope; during the depth reconstruction process, first, based on the inverse filters corresponding to different imaging components obtained during the calibration process and the transformation matrices between different imaging components, the corresponding relationship between the offset differences of different object distances and the point spread functions corresponding to different imaging components is implicitly established, thereby recording the change in the offset of the point spread function corresponding to different imaging components caused by the change in object distance; then, during the actual use process, according to the offset loss corresponding to the object under test at different object distances, the true depth of the surface of the object under test is inversely deduced to complete the depth reconstruction and obtain the corresponding super-depth-of-field image; since this embodiment does not require multiple shootings or the use of a displacement platform during the depth reconstruction process, and only one shooting with a dual camera can obtain a super-depth-of-field image and a depth image, it is thus simpler and more efficient; and by filtering the image with the inverse filters corresponding to different object distances and restoring it to the original image, the triangular ghosting generated by any point passing through the phase mask can be eliminated, that is, by considering the imaging offsets corresponding to all object distances, the problem of imaging artifacts existing during the reconstruction with a traditional phase mask can be improved; in addition, this embodiment does not require special auxiliary illumination, and all other optical devices except the phase mask are standard products, so the entire solution is also easier to implement.

[0068] Please refer to Figure 2 , in some embodiments, a microscopic imaging method with a depth reconstruction effect is provided, specifically including:

[0069] Step S200: Image the object under test through an image acquisition component to obtain a first image and a second image of the object under test.

[0070] In this embodiment, the image acquisition component includes at least a first imaging component and a second imaging component, and the first imaging component and the second imaging component perform image acquisition simultaneously to obtain a first image and a second image of the object under test; wherein, the first imaging component and the second imaging component each have their corresponding inverse filters at multiple preset object distances, and at each object distance, there is also a corresponding transformation matrix between the first imaging component and the second imaging component, and this transformation matrix is used to convert the images corresponding to the first imaging component and the second imaging component into the same coordinate system.

[0071] Step S210: According to the first image and the second image of the object under test, obtain the offset loss corresponding to any pixel point at each object distance.

[0072] For any object distance: Filter the first image according to the inverse filter corresponding to the object distance of the first imaging component to obtain a third image corresponding to the object distance; filter the second image according to the inverse filter corresponding to the object distance of the second imaging component to obtain a fourth image corresponding to the object distance; and perform a rigid transformation on the fourth image according to the transformation matrix between the first imaging component and the second imaging component at the object distance to obtain a fifth image corresponding to the object distance; obtain the offset loss amount corresponding to any pixel at the object distance according to the pixel values corresponding to the obtained third image and the fifth image for any pixel.

[0073] Step S220: Obtain the depth value of the pixel according to the offset loss amounts corresponding to the pixel at all object distances, and the depth values of all pixels form a depth map corresponding to the object to be measured.

[0074] It should be noted that the method steps in this embodiment correspond to the processing steps of the processor in the above embodiment, and the specific implementation manners have been specifically described in the above embodiment, so they will not be repeated here.

[0075] In this embodiment, first, the image acquisition component is used to acquire images of the object to be measured placed at an unknown position to obtain a first image and a second image; then, according to the inverse filters corresponding to different imaging components at different object distances, the corresponding images are filtered to eliminate the point spread function noise contained in the images and restore them to clear original images, thereby obtaining third images and fourth images corresponding to different object distances; then, image transformation is performed according to the transformation matrices of different imaging components at the corresponding object distances to obtain fifth images corresponding to different object distances, so as to convert the images corresponding to different imaging components into the same coordinate system, that is, convert the fourth image into the coordinate system corresponding to the third image. At this time, the difference between the fifth image and the third image is the difference caused by the offset of the point spread function. Therefore, the offset loss amount corresponding to the pixel at different object distances is obtained according to the difference degree between the third image and the fifth image for the same pixel, and the depth map corresponding to the pixel is obtained by minimizing the offset loss amount, so as to complete depth reconstruction and obtain the corresponding super-depth-of-field image. Since this embodiment does not require multiple shootings or the use of a displacement platform, the super-depth-of-field image and the depth image can be obtained only by one shooting of the dual cameras, which is simpler and more efficient.

[0076] Those skilled in the art can understand that all or part of the functions of the above-mentioned embodiments can be implemented in a hardware manner or in a computer program manner. When all or part of the functions in the above-mentioned embodiments are implemented in a computer program manner, the program can be stored in a computer-readable storage medium, and the storage medium can include: read-only memory, random access memory, magnetic disk, optical disk, hard disk, etc. The above functions can be realized by a computer executing this program. For example, store the program in the memory of the device. When the processor executes the program in the memory, the above-mentioned all or part of the functions can be realized. In addition, when all or part of the functions in the above-mentioned embodiments are implemented in a computer program manner, the program can also be stored in a storage medium such as a server, another computer, magnetic disk, optical disk, flash drive or mobile hard disk, and saved to the memory of the local device by downloading or copying, or the system of the local device can be updated. When the processor executes the program in the memory, all or part of the functions in the above-mentioned embodiments can be realized.

[0077] The above uses specific examples to elaborate on the present invention, which is only used to help understand the present invention and is not intended to limit the present invention. For those skilled in the art of the present invention, according to the idea of the present invention, several simple deductions, deformations or substitutions can also be made.

Claims

1. A microscopic imaging device with a depth reconstruction effect, comprising a microscope objective lens, a tube lens, a light source, a lifting platform and a plurality of light splitting devices, characterized in that: Also includes: A phase mask, wherein the phase mask is located between a microscope objective and a light splitting device; An image acquisition component, the image acquisition component at least comprising a first shooting component and a second shooting component; the first shooting component and the second shooting component have corresponding inverse filters at multiple preset object distances, and at each object distance, there is also a corresponding transformation matrix between the first shooting component and the second shooting component; and a processor for: Control the lifting platform to move in the vertical direction according to the preset moving step length; Acquire inverse filters corresponding to the first photographing component and the second photographing component at a plurality of preset object distances, and a transformation matrix between the first photographing component and the second photographing component at each object distance; Controlling the first photographing component and the second photographing component to simultaneously capture images of the object to be measured to obtain a first image and a second image; For any object distance: filtering the first image according to the inverse filter corresponding to the object distance by the first photographing component to obtain a third image corresponding to the object distance; filtering the second image according to the inverse filter corresponding to the object distance by the second photographing component to obtain a fourth image corresponding to the object distance; and performing a rigid body transformation on the fourth image according to the transformation matrix between the first photographing component and the second photographing component at the object distance to obtain a fifth image corresponding to the object distance; obtaining the offset loss amount corresponding to the arbitrary pixel point at the object distance according to the pixel value corresponding to the arbitrary pixel point in the obtained third image and the fifth image; The depth value of any pixel is obtained according to the offset loss corresponding to all object distances, and the depth values ​​of all pixels constitute a depth map corresponding to the object to be measured, thereby completing depth reconstruction.

2. The microscopic imaging device according to claim 1, characterized in that: The phase mask comprises a bicubic phase mask.

3. The microscopic imaging device according to claim 1, characterized in that: The processor is also used to control the image acquisition component to acquire images of a preset standard object after each movement of the lifting platform, so as to obtain images corresponding to the standard object at multiple preset object distances, wherein the standard object is placed on the lifting platform.

4. The microscopic imaging device according to claim 3, characterized in that: The pre-set standard object comprises: an opaque flat plate with ultra-fine pinholes distributed at equal intervals, and the opaque flat plate is provided with a backlight surface light source for auxiliary lighting.

5. The microscopic imaging device according to claim 1, characterized in that: The processor obtains the inverse filters corresponding to the first shooting component and the second shooting component at multiple preset object distances, including: The first photographing component and the second photographing component respectively collect images of a preset standard object at a plurality of preset object distances; For any object distance: obtain the inverse filter corresponding to the first photographing component at the object distance based on the image captured by the first photographing component; obtain the inverse filter corresponding to the second photographing component at the object distance based on the image captured by the second photographing component.

6. The microscopic imaging device according to claim 1, characterized in that: The processor obtains the transformation matrix between the first shooting component and the second shooting component at each object distance, including: For any object distance: at the object distance, the images captured by the first photographing component and the second photographing component on the preset standard object are respectively referred to as the first standard image and the second standard image; The first standard image is filtered by the first shooting component using an inverse filter corresponding to the object distance to obtain a third standard image; the second standard image is filtered by the second shooting component using an inverse filter corresponding to the object distance to obtain a fourth standard image; wherein the third standard image and the fourth standard image contain arrays of dots distributed at equal intervals, and each dot corresponds to an ultrafine pinhole on the standard object; The center point coordinates of each dot in the third standard image and the fourth standard image are respectively obtained; a transformation matrix is ​​obtained according to the center point coordinates of corresponding dots in the third standard image and the fourth standard image, and the transformation matrix is ​​a rigid body transformation matrix including rotation and translation.

7. The microscopic imaging device according to claim 1, characterized in that: The processor obtains the offset loss amount corresponding to any pixel point at the object distance according to the pixel value corresponding to any pixel point in the obtained third image and the fifth image, including: For any pixel point, the grayscale value of the pixel point in different channels in the third image and the fifth image is obtained respectively, and the norm is calculated according to the difference between the corresponding grayscale values ​​of the pixel point in different channels, and the obtained norm is used as the offset loss corresponding to the pixel point at the object distance.

8. The microscopic imaging device according to claim 1, characterized in that: The processor obtains the depth value of any pixel point based on the offset loss amount corresponding to any pixel point at all object distances, including: for any pixel point, obtaining the minimum value of the offset loss amount corresponding to the pixel point at all object distances, and using the object distance corresponding to the obtained minimum value as the depth value of the pixel point.

9. The microscopic imaging device according to claim 1, characterized in that: The processor further comprises: A super depth of field image is obtained, wherein, for any pixel point in the super depth of field image, a third image and a fifth image corresponding to the object distance are obtained when the depth value of the pixel point is used as the object distance, and an average value of the grayscale value corresponding to the pixel point in each channel in the obtained third image and fifth image is calculated, and the obtained average value is used as the grayscale value of the pixel point in the corresponding channel in the super depth of field image.

10. A microscopic imaging method with depth reconstruction effect, characterized in that: include: The image acquisition component is used to acquire an image of the object to be measured, and the image acquisition component at least includes a first shooting component and a second shooting component, and the first shooting component and the second shooting component simultaneously acquire images to obtain a first image and a second image of the object to be measured; wherein the first shooting component and the second shooting component have corresponding inverse filters at multiple preset object distances, and at each object distance, there is also a corresponding transformation matrix between the first shooting component and the second shooting component; For any object distance: filtering the first image according to the inverse filter corresponding to the object distance by the first photographing component to obtain a third image corresponding to the object distance; filtering the second image according to the inverse filter corresponding to the object distance by the second photographing component to obtain a fourth image corresponding to the object distance; and performing a rigid body transformation on the fourth image according to the transformation matrix between the first photographing component and the second photographing component at the object distance to obtain a fifth image corresponding to the object distance; obtaining the offset loss amount corresponding to the arbitrary pixel point at the object distance according to the pixel value corresponding to the arbitrary pixel point in the obtained third image and the fifth image; A depth value of any pixel is obtained according to the offset loss corresponding to all object distances, and the depth values ​​of all pixels constitute a depth map corresponding to the object to be measured.

11. The microscopic imaging method according to claim 10, characterized in that: The first photographing component and the second photographing component each have a corresponding inverse filter at a plurality of preset object distances, including: The first photographing component and the second photographing component respectively collect images of a preset standard object at a plurality of preset object distances; For any object distance: obtain the inverse filter corresponding to the first photographing component at the object distance based on the image captured by the first photographing component; obtain the inverse filter corresponding to the second photographing component at the object distance based on the image captured by the second photographing component.

12. The microscopic imaging method according to claim 11, characterized in that: The pre-set standard object includes: an opaque plane plate with ultra-fine pinholes distributed at equal intervals.

13. The microscopic imaging method according to claim 10, characterized in that: At each object distance, there is also a corresponding transformation matrix between the first shooting component and the second shooting component, including: For any object distance: at the object distance, the images captured by the first photographing component and the second photographing component on the preset standard object are respectively referred to as the first standard image and the second standard image; The first standard image is filtered by the first shooting component using an inverse filter corresponding to the object distance to obtain a third standard image; the second standard image is filtered by the second shooting component using an inverse filter corresponding to the object distance to obtain a fourth standard image; wherein the third standard image and the fourth standard image contain arrays of dots distributed at equal intervals, and each dot corresponds to an ultrafine pinhole on the standard object; The center point coordinates of each dot in the third standard image and the fourth standard image are respectively obtained; a transformation matrix is ​​obtained according to the center point coordinates of corresponding dots in the third standard image and the fourth standard image, and the transformation matrix is ​​a rigid body transformation matrix including rotation and translation.

14. The microscopic imaging method according to claim 10, characterized in that: The step of obtaining the offset loss amount corresponding to any pixel point at the object distance according to the pixel value corresponding to any pixel point in the obtained third image and the fifth image comprises: For any pixel point, the grayscale value of the pixel point in different channels in the third image and the fifth image is obtained respectively, and the norm is calculated according to the difference between the corresponding grayscale values ​​of the pixel point in different channels, and the obtained norm is used as the offset loss corresponding to the pixel point at the object distance.

15. The microscopic imaging method according to claim 10, characterized in that: The method of obtaining the depth value of any pixel point according to the offset loss amount corresponding to any pixel point at all object distances includes: for any pixel point, obtaining the minimum value of the offset loss amount corresponding to the pixel point at all object distances, and using the object distance corresponding to the obtained minimum value as the depth value of the pixel point.

16. The microscopic imaging method according to claim 10, characterized in that: Also includes: A super depth of field image is obtained, wherein, for any pixel point in the super depth of field image, a third image and a fifth image corresponding to the object distance are obtained when the depth value of the pixel point is used as the object distance, and an average value of the grayscale value corresponding to the pixel point in each channel in the obtained third image and fifth image is calculated, and the obtained average value is used as the grayscale value of the pixel point in the corresponding channel in the super depth of field image.

17. A computer-readable storage medium, characterized in that: A computer program is stored on the medium, and the computer program can be executed by a processor to implement the method according to any one of claims 10 to 16.

Images