Meta Imaging Passive Scanning System and Method

Abstract

This application relates to the field of imaging technology, and particularly to a meta-imaging passive scanning system and method. The method includes: obtaining a scanned image of a target object by using passive scanning methods such as natural jitter during shooting, jitter of the image itself, or movement of the sample; estimating the relative movement of the sample to obtain a relatively accurate sampling position; and combining a motion correction algorithm based on optical flow estimation for each view to correct local dynamic structure regions and extract high-resolution target object image signals. Thus, the problems in the related art such as complex optical paths and difficult operations, difficulties in synchronizing the scanning system and the imaging system, difficulties in integrating and miniaturizing the entire imaging system, and high costs that cannot well meet market demands are solved. It has the advantages of small size, high integration, low cost, the ability to achieve multi-dimensional high-resolution imaging, and broad applications.

Description

Technical Field

[0001] This application relates to the field of imaging technologies, and particularly to a meta-imaging passive scanning system and method. Background Art

[0002] Currently, with the continuous progress of the processing technology of image sensors, it enables us to obtain pictures with tens of millions of pixels in a single exposure. However, the subsequent problem is that the amount of data contained in the image is getting larger and larger, and its acquisition dimension is still limited to two dimensions only. In real scenarios, high-resolution and complex high-dimensional information is often required. How to quickly, accurately, and real-time collect target objects through two-dimensional sensors has always been a current research hotspot.

[0003] In related technologies, the most intuitive method is to use a two-dimensional sensor to scan dimension by dimension and surface by surface, or to use a microlens array combined with a three-dimensional deconvolution algorithm to extract three-dimensional signals.

[0004] However, current optical scanning systems have the following disadvantages: (1) The optical path is complex, the operation is difficult, and the synchronization between the scanning system and the imaging system is difficult; (2) The volume is large, and it is difficult to integrate and miniaturize the entire imaging system. Usually, an optical test bench is required for assistance; (3) The cost is high. The cost of a mechanical scanning galvanometer is tens of thousands of yuan, which cannot well meet the market demand. Summary of the Invention

[0005] This application provides a meta-imaging passive scanning system and method to solve problems in related technologies such as complex optical path and difficult operation, difficult synchronization between the scanning system and the imaging system, difficult integration and miniaturization of the entire imaging system, and high cost that cannot well meet the market demand. It has the advantages of small volume, high integration, low cost, and can achieve multi-dimensional and multi-scale phase space imaging, with broad applications, etc.

[0006] The first aspect of the embodiments of this application provides a meta-imaging passive scanning method, including the following steps:

[0007] Obtain a scanned image of a target object by using the passive scanning function of a meta-imaging system, where the passive scanning function is to perform scanning by using natural jitter during the shooting process, or jitter of the image itself, or movement of the sample;

[0008] Estimate the relative movement of the target object based on the scanned image to obtain the target sampling position of the target object;

[0009] Based on a preset correction algorithm, obtain a high-resolution image of the target object by performing scattered interpolation with dense sampling and precise equivalent scanning of each microlens at the target sampling position.

[0010] Optionally, after obtaining the scanned image of the target object, the meta-imaging system further includes:

[0011] Advancing the high-resolution imaging speed of the target object with a preset propulsion strategy until the camera frame rate condition is met.

[0012] Optionally, the meta-imaging system includes a photosensitive chip and a microlens array.

[0013] Optionally, the preset algorithm is a motion correction algorithm based on optical flow estimation for each view.

[0014] An embodiment of the second aspect of the present application provides a meta-imaging passive scanning system, including:

[0015] An acquisition module for obtaining a scanned image of a target object by using the passive scanning function of the meta-imaging system, where the passive scanning function is to perform scanning by using natural jitter during shooting, or jitter of the image itself, or movement of the sample

[0016] An acquisition module for estimating the relative motion of the target object based on the scanned image to obtain the target sampling position of the target object; a generation module for obtaining a high-resolution image of the target object by performing dense sampling of scattered interpolation and precise equivalent scanning of each microlens based on a preset correction algorithm.

[0017] Optionally, after obtaining the three-dimensional sampling image of the target object, the generation module further includes:

[0018] Advancing the high-resolution imaging speed of the target object with a preset propulsion strategy until the camera frame rate condition is met.

[0019] Optionally, the meta-imaging system includes a photosensitive chip and a microlens array.

[0020] Optionally, the preset correction algorithm is a motion correction algorithm based on optical flow estimation for each view.

[0021] An embodiment of the third aspect of the present application provides an electronic device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor executes the program to implement the meta-imaging passive scanning method as described in the above embodiments.

[0022] An embodiment of the fourth aspect of the present application provides a computer-readable storage medium, on which a computer program is stored, and the program is executed by a processor to implement the meta-imaging passive scanning method as described in the above embodiments.

[0023] Therefore, the meta-imaging passive scanning method of the embodiments of the present application has the following advantages.

[0024] (1) Simple operation. The scanning system and the imaging system in the embodiments of the present application have excellent synchronization. By applying passive mechanical scanning systems such as natural jitters during shooting, jitters of the image itself, or movements of the sample to the actual system, the problem of out-of-sync between the scanning system and the imaging system is avoided, and high-resolution images can be accurately realized by subsequent algorithms.

[0025] (2) Small volume. The embodiments of the present application miniaturize and integrate the entire imaging system. The system design is miniaturized and can be integrated on various small instruments, such as used on mobile phones. Compared with the original system, the space occupied by the entire system hardly changes.

[0026] (3) Low price. The embodiments of the present application can meet market demands. Compared with the prices of galvanometers and micro-galvanometers which are often tens of thousands of yuan, the cost of this system is low, and it has fast response and good performance, making it suitable for mass production and market demands.

[0027] Additional aspects and advantages of the present application will be partially given in the following description, partially become obvious from the following description, or be understood through the practice of the present application. Description of the Drawings

[0028] The above and / or additional aspects and advantages of the present application will become obvious and easy to understand from the following description of the embodiments in conjunction with the drawings, where:

[0029] Figure 1 is a schematic flowchart of the meta-imaging passive scanning method provided according to an embodiment of the present application;

[0030] Figure 2 is a schematic diagram of a microlens provided according to an embodiment of the present application;

[0031] Figure 3 is a schematic diagram of the passive scanning principle of a meta-imaging system provided according to an embodiment of the present application;

[0032] Figure 4 is a schematic block diagram of a meta-imaging passive scanning system provided according to an embodiment of the present application;

[0033] Figure 5 is a schematic diagram of an electronic device provided according to an embodiment of the present application. Detailed Embodiments

[0034] The embodiments of the present application will be described in detail below. Examples of the embodiments are shown in the drawings, where the same or similar reference numerals denote the same or similar elements or elements with the same or similar functions throughout. The embodiments described below with reference to the drawings are exemplary and are intended to explain the present application, and should not be construed as a limitation of the present application.

[0035] The meta-imaging passive scanning system and method according to the embodiments of the present application will be described below with reference to the accompanying drawings. In view of the problems in the related art mentioned in the above background art, such as complex optical paths and difficult operations, difficulties in the synchronization of the scanning system and the imaging system, difficulties in the integration and miniaturization of the entire imaging system, and high costs that cannot well meet the market demand, the present application provides a meta-imaging passive scanning method. In this method, natural jitter during shooting, or jitter of the image itself, or movement of the sample is used for passive scanning imaging. The relative movement of the sample is estimated to obtain relatively accurate sampling positions, and based on the motion correction algorithm of the optical flow estimation for each view, high-resolution images are obtained through scattered interpolation of dense sampling and precise equivalent scanning positions of each microlens.

[0036] Specifically, Figure 1 FIG. is a schematic flow chart of a meta-imaging passive scanning method provided by an embodiment of the present application.

[0037] As Figure 1 shown, the meta-imaging passive scanning method includes the following steps:

[0038] In step S101, a scanned image of the target object is obtained by using the passive scanning function of the meta-imaging system, where the passive scanning function is to perform scanning by using natural jitter during shooting, or jitter of the image itself, or movement of the sample.

[0039] Optionally, in some embodiments, the meta-imaging system includes a photosensitive chip and a microlens array.

[0040] Specifically, natural jitter during shooting, or jitter of the image itself, or movement of the sample is used for passive scanning imaging. For example, a vibration force is applied to the scanning galvanometer system and / or the optical image stabilization system by hand, so that the passive scanning device performs scanning based on the vibration action, and the optical lens maps the spectral information of the target object to the image plane.

[0041] In step S102, based on the scanned image, the relative movement of the target object is estimated to obtain the target sampling position of the target object.

[0042] It can be understood that during the actual imaging process, passive scanning (natural jitter, or jitter of the image itself, or movement of the sample) will cause the generation of artifacts during imaging, so rigid motion correction is required. However, the movement of the target sample may be uneven, and residual artifacts will still be generated after correction. Therefore, it is necessary to estimate the relative movement of the sample to obtain relatively accurate sampling positions.

[0043] Specifically, in the embodiments of the present application, the change of the image can be reflected by the optical flow information. Since the optical flow contains the information of the target movement, it can be used by the observer to determine the movement of the target. The three-dimensional structure and movement of the sample are restored from the optical flow to obtain a relatively accurate sampling position.

[0044] Specifically, as Figure 3 shown, by using the natural jitter during the shooting process, or the jitter of the image itself, or the passive scanning imaging of the movement of the sample, the relative movement of the sample is estimated to obtain a relatively accurate sampling position, and based on the motion correction algorithm estimated by the optical flow of each view, a high-resolution image is obtained through the scattered interpolation of dense sampling and the accurate equivalent scanning position of each microlens.

[0045] In step S103, based on a preset correction algorithm, a high-resolution image of the target object is obtained through the scattered interpolation of dense sampling and the accurate equivalent scanning target sampling position of each microlens.

[0046] Optionally, in some embodiments, after obtaining the sampling image of the target object, it further includes: advancing the high-resolution imaging speed of the target object with a preset propulsion strategy until the camera frame rate condition is met.

[0047] Optionally, in some embodiments, the preset algorithm is a motion correction algorithm based on the optical flow estimation of each view.

[0048] Specifically, a sliding window of the scanning period is used to realign multiple low-resolution images into a high-resolution image at the same time sampling rate at the central time point. The optical flow map from other resolution frames to the central resolution frame is estimated to calculate the accurate coordinates of all low-resolution sampling points in the high-resolution grid at the central time point, and then a high-resolution image is obtained through the scattered interpolation of dense sampling and the accurate equivalent scanning position of each microlens..

[0049] According to the meta-imaging passive scanning method proposed by the embodiments of the present application, by using the natural jitter during the shooting process, or the jitter of the image itself, or the passive scanning imaging of the movement of the sample, the relative movement of the sample is estimated to obtain a relatively accurate sampling position, and based on the motion correction algorithm estimated by the optical flow of each view, a high-resolution image is obtained through the scattered interpolation of dense sampling and the accurate equivalent scanning position of each microlens. Thus, the problems in the related art such as complex optical path and difficult operation, difficulty in synchronizing the scanning system and the imaging system, difficulty in integrating and miniaturizing the entire imaging system, high cost, and inability to well meet the market demand are solved. It has the advantages of small volume, high integration, low cost, and can realize multi-dimensional and multi-scale phase space imaging, and has broad applications.

[0050] Secondly, a meta-imaging passive scanning system proposed according to the embodiments of the present application is described with reference to the accompanying drawings.

[0051] Figure 4 It is a block diagram of the meta-imaging passive scanning system according to an embodiment of the present application.

[0052] As Figure 4 shown, the meta-imaging passive scanning system 10 includes: an acquisition module 100, a collection module 200, and a generation module 300.

[0053] Among them, the acquisition module 100 is used to acquire the scanned image obtained by the passive scanning device scanning the target object;

[0054] The collection module 200 is used to collect the scanned image information of the target object;

[0055] The generation module 300 is used for the estimation of the relative motion of the sample, obtaining a relatively accurate sampling position, based on the motion correction algorithm of the optical flow estimation of each view, and obtaining a high-resolution image through the scattered interpolation of dense sampling and the accurate equivalent scanning position of each microlens.

[0056] Optionally, in some embodiments, after obtaining the sampled image of the target object, the generation module 300 further includes:

[0057] Advancing the three-dimensional imaging speed of the three-dimensional sampled image of the target object with a preset advancing strategy until the camera frame rate condition is met.

[0058] Optionally, in some embodiments, before obtaining the scanned image obtained by the passive scanning device scanning the target object, the acquisition module 100 further includes:

[0059] Passively scanning and imaging by using the natural jitter during shooting, or the jitter of the image itself, or the movement of the sample.

[0060] Optionally, in some embodiments, the passive scanning device includes a scanning galvanometer system.

[0061] Optionally, in some embodiments, the preset algorithm is a motion correction algorithm based on the optical flow estimation of each view.

[0062] Optionally, in some embodiments, the image acquisition unit is a CMOS sensor.

[0063] It should be noted that the foregoing explanation of the embodiments of the meta-imaging passive scanning method also applies to the meta-imaging passive scanning system of this embodiment, and will not be elaborated here.

[0064] The meta-imaging passive scanning system proposed according to the embodiments of the present application uses the natural jitter during the shooting process, or the jitter of the image itself, or the movement of the sample for passive scanning imaging, estimates the relative movement of the sample, obtains relatively accurate sampling positions, and based on the motion correction algorithm estimated by the optical flow of each view, obtains high-resolution images through the scattered interpolation of dense sampling and the precise equivalent scanning positions of each microlens. Thus, it solves the problems in the related art such as complex optical paths and difficult operations, difficult synchronization between the scanning system and the imaging system, difficult integration and miniaturization of the entire imaging system, high cost, and inability to well meet market demands. It has the advantages of small volume, high integration, low cost, and can realize multi-dimensional and multi-scale phase space imaging, and has broad applications.

[0065] Figure 5 The following is a schematic structural diagram of an electronic device provided by an embodiment of the present application. The electronic device may include:

[0066] A memory 501, a processor 502, and a computer program stored on the memory 501 and executable on the processor 502.

[0067] When the processor 502 executes the program, it implements the meta-imaging passive scanning method provided in the above embodiment.

[0068] Further, the electronic device further includes:

[0069] A communication interface 503 for communication between the memory 501 and the processor 502.

[0070] The memory 501 is used to store a computer program executable on the processor 502.

[0071] The memory 501 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory.

[0072] If the memory 501, the processor 502, and the communication interface 503 are implemented independently, the communication interface 503, the memory 501, and the processor 502 can be interconnected through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5It is represented only by a thick line, but it does not mean that there is only one bus or one type of bus.

[0073] Optionally, in a specific implementation, if the memory 501, the processor 502, and the communication interface 503 are integrated on a single chip, the memory 501, the processor 502, and the communication interface 503 can communicate with each other through an internal interface.

[0074] The processor 502 may be a central processing unit (CPU for short), or an application specific integrated circuit (ASIC for short), or one or more integrated circuits configured to implement the embodiments of the present application.

[0075] The embodiments of the present application also provide a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the above-mentioned meta-imaging passive scanning method is implemented.

[0076] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, without conflict, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples.

[0077] In addition, the terms "first" and "second" are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "N" is at least two, such as two, three, etc., unless otherwise specifically defined.

[0078] Any process or method description represented in a flowchart or otherwise described herein can be understood to represent a module, segment, or portion of code including one or more executable instructions for implementing custom logic functions or processes. The scope of the preferred embodiments of this application includes additional implementations where functions may be executed in a manner that is not shown or discussed, including in a substantially simultaneous manner or in a reverse order according to the functions involved, which should be understood by those skilled in the technical field to which the embodiments of this application pertain.

[0079] Logic and / or steps represented in a flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or in conjunction with these instruction execution systems, apparatuses, or devices. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection portion (electronic device) having one or more wirings, a portable computer diskette (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpretation, or otherwise processing as appropriate, and then storing it in a computer memory.

[0080] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application-specific integrated circuits having suitable combinational logic gate circuits, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0081] Those of ordinary skill in the art can understand that all or part of the steps carried out in the methods of the above embodiments can be completed by instructing relevant hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiments.

[0082] In addition, each functional unit in various embodiments of the present application can be integrated into a processing module, or each unit can exist physically alone, or two or more units can be integrated into one module. The above integrated module can be implemented in the form of hardware or in the form of a software functional module. When the above integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0083] The above-mentioned storage medium can be a read-only memory, a magnetic disk, an optical disc, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present application. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

Claims

1. A meta-imaging passive scanning method, characterized in that, wherein, the method comprises the following steps: Obtain a scanned image of the target object by using the passive scanning function of the meta-imaging system, wherein the passive scanning function is to perform scanning by using the natural jitter during the shooting process, or the jitter of the image itself, or the movement of the sample; the meta-imaging system includes a photosensitive chip and a microlens array; Based on the scanned image, estimate the relative movement of the target object to obtain the target sampling position of the target object, wherein obtaining the target sampling position of the target object includes: recovering the three-dimensional structure and movement of the target object from the optical flow, and the optical flow contains the movement information of the target object; Based on a preset correction algorithm, calculate the coordinate positions of all low-resolution sampling points in the high-resolution grid at the central time point of the target sampling position, and based on the coordinate positions of all low-resolution sampling points, perform scattered interpolation by dense sampling and accurately equivalent scanning of each microlens on the target sampling position to obtain a high-resolution image of the target object, wherein the preset correction algorithm is a motion correction algorithm based on optical flow estimation for each view.

2. The method according to claim 1, characterized in that, after obtaining the high-resolution image of the target object, it further includes: Advance the imaging speed of the high-resolution image of the target object with a preset propulsion strategy until the camera frame rate condition is met.

3. A meta-imaging passive scanning system, characterized in that, the meta-imaging system has a passive scanning function, wherein the system includes: An acquisition module for obtaining a scanned image of the target object by using the passive scanning function of the meta-imaging system, wherein the passive scanning function is to perform scanning by using the natural jitter during the shooting process, or the jitter of the image itself, or the movement of the sample; the meta-imaging system includes a photosensitive chip and a microlens array; A collection module for estimating the relative movement of the target object based on the scanned image to obtain the target sampling position of the target object, wherein obtaining the target sampling position of the target object includes: recovering the three-dimensional structure and movement of the target object from the optical flow, and the optical flow contains the movement information of the target object; A generation module for calculating the coordinate positions of all low-resolution sampling points in the high-resolution grid at the central time point of the target sampling position based on a preset correction algorithm, and based on the coordinate positions of all low-resolution sampling points, performing scattered interpolation by dense sampling and accurately equivalent scanning of each microlens on the target sampling position to obtain a high-resolution image of the target object, wherein the preset correction algorithm is a motion correction algorithm based on optical flow estimation for each view.

4. The system according to claim 3, characterized in that, after obtaining the high-resolution image of the target object, the generation module further includes: Advance the high-resolution imaging speed of the target object with a preset propulsion strategy until the camera frame rate condition is met.

5. An electronic device, characterized in that, comprises: A memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to implement the meta-imaging passive scanning method according to any one of claims 1-2.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that the program is executed by a processor to implement the meta-imaging passive scanning method according to any one of claims 1-2.

Images