Calculation ghost imaging reconstruction method and device under low sampling rate, equipment and medium

Abstract

The invention relates to a calculation ghost imaging reconstruction method, device and equipment under a low sampling rate and a medium, and the method comprises the steps: randomly initializing to generate illumination speckles, loading the illumination speckles into a spatial light modulator to control a light field irradiated by laser, irradiating the spatial light modulator by the laser to determine a modulated light beam, irradiating the modulated light beam to a target object, and after the modulated light beam passes through one or more media, capturing a light intensity value after the modulated light beam passes through the target object by a barrel detector; inputting the light intensity value into a second visual converter model, training by adopting a self-supervision method to determine an optimized illumination speckle, and re-loading the optimized illumination speckle to the spatial light modulator to re-modulate the light field irradiated by the laser until the improved visual converter neural network reaches a convergence state; and inputting the optimized light intensity value of the illumination speckle into a second visual converter model, and determining a reconstructed image of the target object. The calculation ghost imaging reconstruction picture quality under the low sampling rate can be obviously improved.

Description

Technical Field

[0001] The present application relates to the field of computer graphics, and in particular to a computational ghost imaging reconstruction method at a low sampling rate, a corresponding device, an electronic device, and a computer-readable storage medium. Background Art

[0002] Ghost imaging is an imaging method that uses a single photodetector to obtain information about a target scene. Instead of using a detector array with a large number of pixels in traditional imaging, it uses a bucket detector with no spatial resolution to capture a two-dimensional image. Computational ghost imaging preloads a series of speckles on the spatial light modulator. When the spatial light modulator is illuminated by a laser, the modulated light is reflected to the object. The light passing through the object is detected by the bucket detector. The intensity value is correlated with the speckle to reconstruct the target image. Computational ghost imaging has higher detection efficiency and sensitivity and has attracted much attention in recent years.

[0003] Currently, due to its ability to solve various challenging problems in different fields, deep learning has also been recently applied to ghost imaging, and the reconstruction quality of ghost imaging at low sampling rates has been improved by iteratively optimizing network parameters; however, at extremely low sampling rates, the reconstruction quality of ghost imaging methods is not high, and the reconstruction quality of ghost imaging methods based on deep learning is not ideal.

[0004] To sum up, in order to adapt to the problems in the prior art such as the low reconstruction quality of ghost imaging methods at extremely low sampling rates and the unsatisfactory reconstruction quality of ghost imaging methods based on deep learning, the applicant has made corresponding explorations in order to solve these problems. Summary of the invention

[0005] The purpose of the present application is to solve the above-mentioned problems and to provide a method for computational ghost imaging reconstruction at a low sampling rate, a corresponding device, an electronic device and a computer-readable storage medium.

[0006] In order to meet the various objectives of this application, this application adopts the following technical solutions:

[0007] A computational ghost imaging reconstruction method at a low sampling rate is proposed to meet one of the purposes of this application, comprising:

[0008] In response to a computational ghost imaging reconstruction instruction at a low sampling rate, randomly initialize and generate an illumination speckle and load it into a spatial light modulator to control a light field irradiated by a laser, irradiate the spatial light modulator with a laser to determine a modulated light beam, irradiate the modulated light beam to a target object and after passing through one or more media, a bucket detector captures a light intensity value after passing through the target object;

[0009] Adding an encoder, a bottleneck layer, and a decoder to the preset first visual converter model, integrating the coordinate attention mechanism into each visual converter block of the first visual converter model, and integrating multiple scales of deep convolution into the feedforward network of the first visual converter model to construct a second visual converter model;

[0010] Inputting the light intensity value into the second visual converter model and training it using a self-supervised method to determine an optimized illumination speckle, and reloading the optimized illumination speckle into the spatial light modulator to remodulate the light field illuminated by the laser until the second visual converter model reaches a convergence state;

[0011] The light intensity value corresponding to the optimized illumination speckle is input into the second visual converter model that has been trained to a convergence state to determine the reconstructed image of the target object, so as to complete the computational ghost imaging reconstruction at a low sampling rate.

[0012] Optionally, the light intensity value corresponding to the optimized illumination speckle is input into a second visual converter model that has been trained to a convergent state to determine a reconstructed image of the target object to complete the step of computational ghost imaging reconstruction at a low sampling rate, including:

[0013] Process the light intensity value after passing through the target object and convert it into a two-dimensional feature map;

[0014] The encoder is used to reduce the spatial resolution of spatial features and capture information at different scales and levels;

[0015] The bottleneck layer is used to integrate the abstract features from the high-level of the encoder;

[0016] The decoder restores the low-resolution feature map to the resolution of the input image and fuses it with the corresponding features in the encoder through jump connections to determine the reconstructed image of the target object.

[0017] Optionally, the light intensity value corresponding to the optimized illumination speckle is input into a second visual converter model that has been trained to a convergent state to determine a reconstructed image of the target object to complete the step of computational ghost imaging reconstruction at a low sampling rate, including:

[0018] The coordinate attention mechanism is integrated into each visual converter block, and the coordinate attention mechanism is used to pool the two-dimensional feature map in the X direction and the Y direction respectively, and the generated feature map is transformed and concatenated;

[0019] The images are segmented along the spatial dimension, and 1×1 convolution is used to increase the dimension, and the weighted feature map is obtained by combining the activation function.

[0020] Optionally, the light intensity value corresponding to the optimized illumination speckle is input into a second visual converter model that has been trained to a convergent state to determine a reconstructed image of the target object to complete the step of computational ghost imaging reconstruction at a low sampling rate, including:

[0021] The deep convolutions of multiple scales are fused into the feedforward network. Four deep convolutions of different scales are inserted between two linear layers in the feedforward network to learn features of different scales. Each deep convolution processes a quarter of the channel and reshapes the features into a one-dimensional format through point-wise convolution.

[0022] Optionally, the step of inputting the light intensity value into the second visual converter model, training it using a self-supervised method to determine an optimized illumination speckle, and reloading the optimized illumination speckle into the spatial light modulator to remodulate the light field illuminated by the laser until the second visual converter model reaches a convergence state comprises:

[0023] In each iteration of the second visual converter model training process, the laser irradiates the illumination speckle so that the bucket detector obtains the light intensity value after passing the target object;

[0024] The light intensity value after passing through the target object is input into the second visual converter model, and the reconstructed image is obtained through the mapping matrix, encoder, bottleneck layer and decoder;

[0025] The preset loss function is used to calculate the mean square error between the reconstructed image and the original image, and the network parameters and illumination speckle are optimized by back-propagation. The computer reloads the optimized illumination speckle to the spatial light modulator and repeats the above steps until the preset number of iterations is reached to complete the training of the second visual converter model.

[0026] Optionally, the expression of the illumination speckle is:

[0027] The expression for obtaining the light intensity value after passing the target object for the i-th time is:

[0028]

[0029] Among them, D(x,y) represents the target object, Indicates the medium, I i is the intensity value of light propagation, B i The light intensity after passing through the target object.

[0030] Optionally, the first visual converter model is a basic model of the visual converter model, and the second visual converter model is an improved visual converter model.

[0031] A computational ghost imaging reconstruction device at a low sampling rate is provided to meet another purpose of the present application, comprising:

[0032] A light intensity value acquisition module is configured to respond to a computational ghost imaging reconstruction instruction at a low sampling rate, randomly initialize and generate an illumination speckle and load it into a spatial light modulator to control a light field irradiated by a laser, irradiate the spatial light modulator with a laser to determine a modulated light beam, irradiate the modulated light beam to a target object and after passing through one or more media, a light intensity value after passing through the target object is captured by a bucket detector;

[0033] a second visual converter building module, configured to add an encoder, a bottleneck layer, and a decoder to a preset first visual converter model, fuse a coordinate attention mechanism into each visual converter block of the first visual converter model, and fuse multiple scales of deep convolution into a feedforward network of the first visual converter model to build a second visual converter model;

[0034] a second visual converter training module, configured to input the light intensity value into the second visual converter model, and perform training using a self-supervised method to determine an optimized illumination speckle, and reload the optimized illumination speckle into the spatial light modulator to re-modulate the light field illuminated by the laser until the second visual converter model reaches a convergence state;

[0035] The ghost imaging reconstruction module is configured to input the light intensity value corresponding to the optimized illumination speckle into a second visual converter model that has been trained to a convergence state, determine a reconstructed image of the target object, and complete computational ghost imaging reconstruction at a low sampling rate.

[0036] An electronic device provided to meet another purpose of the present application includes a central processing unit and a memory, wherein the central processing unit is used to call and run a computer program stored in the memory to execute the steps of the computational ghost imaging reconstruction method at a low sampling rate described in the present application.

[0037] A computer-readable storage medium is provided to meet another purpose of the present application, which stores a computer program implemented according to the computational ghost imaging reconstruction method under the low sampling rate in the form of computer-readable instructions. When the computer program is called and executed by a computer, the steps included in the corresponding method are executed.

[0038] Compared with the prior art, the present application aims at the problems that the reconstruction quality of the ghost imaging method is not high at an extremely low sampling rate, and the reconstruction quality of the ghost imaging method based on deep learning is not ideal. The present application includes but is not limited to the following beneficial effects:

[0039] First, this application can enhance channel attention. According to the strong correlation between ghost imaging measurements, the coordinate attention mechanism is incorporated into the visual converter block, which effectively improves the imaging quality.

[0040] Secondly, the present application inserts deep convolutions of different scales between two linear layers in a feedforward network, performs feature fusion at multiple scales, and realizes the effective transmission of information and features between multiple feature layers;

[0041] Third, the computational ghost imaging reconstruction method based on the improved visual converter of the present application is trained by simultaneously optimizing the optimization strategy of the illumination speckle and the visual converter model. The improved visual converter combines local and global information and multi-scale feature fusion, which effectively improves the reconstruction efficiency and reconstruction quality of computational ghost imaging images under extremely low sampling rates. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0043] Figure 1 An exemplary network architecture used in the computational ghost imaging reconstruction method at low sampling rate of this application;

[0044] Figure 2 is an exemplary network architecture for computing ghost imaging reconstruction in the embodiments of the present application;

[0045] Figure 3 It is a flowchart of a computational ghost imaging reconstruction method based on an improved visual converter in an embodiment of the present application;

[0046] Figure 4 is a block diagram of the improved visual converter;

[0047] Figure 5 is a block diagram of each visual converter block in the improved visual converter;

[0048] Figure 6 This is a block diagram of the feed-forward network in the Vision Transformer block.

[0049] Figure 7 This is a principle block diagram of a computational ghost imaging reconstruction device at a low sampling rate in an embodiment of the present application;

[0050] Figure 8 It is a schematic diagram of the structure of the computer device in the embodiment of the present application. DETAILED DESCRIPTION

[0051] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.

[0052] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0053] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0054] It will be understood by those skilled in the art that the terms "client", "terminal" and "terminal device" as used herein include both devices with wireless signal receivers, which are devices with only wireless signal receivers without transmission capability, and devices with receiving and transmitting hardware, which are devices with receiving and transmitting hardware capable of two-way communication over a two-way communication link. Such devices may include: cellular or other communication devices such as personal computers, tablet computers, which have single-line displays or multi-line displays or cellular or other communication devices without multi-line displays; PCS (Personal Communications Services) which may combine voice, data processing, fax and / or data communication capabilities; PDA (Personal Digital Assistant) which may include a radio frequency receiver, pager, Internet / intranet access, web browser, notepad, calendar and / or GPS (Global Positioning System) receiver; conventional laptop and / or palmtop computers or other devices which have and / or include a radio frequency receiver. The "client", "terminal" and "terminal device" used herein may be portable, transportable, installed in a vehicle (air, sea and / or land), or adapted and / or configured to run locally, and / or in a distributed form, at any other location on the earth and / or in space. The "client", "terminal" and "terminal device" used herein may also be a communication terminal, an Internet terminal, a music / video playing terminal, such as a PDA, a MID (Mobile Internet Device) and / or a mobile phone with a music / video playing function, or may be a smart TV, a set-top box and other devices.

[0055] The hardware referred to by the names such as "server", "client", and "service node" in this application is essentially an electronic device with the equivalent capabilities of a personal computer. It is a hardware device with the necessary components revealed by the von Neumann principle, such as a central processing unit (including an arithmetic unit and a controller), a memory, an input device, and an output device. The computer program is stored in its memory, and the central processing unit calls the program stored in the external memory into the internal memory for execution, executes the instructions in the program, and interacts with the input and output devices to complete specific functions.

[0056] It should be pointed out that the concept of "server" referred to in this application can also be extended to the case of server clusters. According to the network deployment principle understood by those skilled in the art, the servers should be logically divided. In physical space, these servers can be independent of each other but can be called through interfaces, or integrated into a physical computer or a set of computer clusters. Those skilled in the art should understand this flexibility, and should not use it to restrict the implementation of the network deployment method of this application.

[0057] Unless expressly specified, one or more technical features of the present application can be deployed on a server for implementation and accessed by a client through a remote call to obtain an online service interface provided by the server, or can be directly deployed and run on a client for access.

[0058] The neural network models referenced or may be referenced in this application, unless expressly specified, can be deployed on a remote server and remotely called on the client, or can be deployed and directly called on a client with sufficient device capabilities. In some embodiments, when it runs on the client, its corresponding intelligence can be obtained through transfer learning to reduce the requirements for the client's hardware operating resources and avoid excessive occupation of the client's hardware operating resources.

[0059] Unless explicitly specified, the various data involved in this application can be stored remotely on a server or on a local terminal device, as long as it is suitable for being called by the technical solution of this application.

[0060] Those skilled in the art should be aware that, although the various methods of the present application are described based on the same concept and thus present commonality with each other, unless otherwise specified, these methods can be independently executed. Similarly, for each embodiment disclosed in the present application, they are all proposed based on the same inventive concept, therefore, concepts with the same expression, and concepts that are appropriately changed for convenience despite different expressions, should be understood as equivalent.

[0061] Unless the mutually exclusive relationship between the embodiments to be disclosed in this application is explicitly stated, the relevant technical features involved in each embodiment can be cross-combined to flexibly construct a new embodiment, as long as such combination does not deviate from the creative spirit of this application and can meet the needs of the prior art or solve certain deficiencies in the prior art. Those skilled in the art should be aware of this flexibility.

[0062] See also Figures 1 to 6 In one embodiment, the method for computational ghost imaging reconstruction at a low sampling rate of the present application includes:

[0063] Step S10, in response to a computational ghost imaging reconstruction instruction at a low sampling rate, randomly initialize and generate an illumination speckle and load it into a spatial light modulator to control the light field irradiated by the laser, irradiate the spatial light modulator with the laser to determine a modulated light beam, irradiate the modulated light beam to a target object and after passing through one or more media, the light intensity value after passing through the target object is captured by a bucket detector;

[0064] The computing terminal device can respond to the computational ghost imaging reconstruction instruction at a low sampling rate, randomly initialize and generate an illumination speckle and load it into a spatial light modulator to control the light field irradiated by the laser, the laser irradiates the spatial light modulator to determine a modulated light beam, and after the modulated light beam is irradiated to a target object and passes through one or more media, the light intensity value after passing through the target object is captured by a bucket detector;

[0065] Specifically, see Figure 2 , the illumination speckle is a binary illumination speckle image, and a random binary speckle image can be generated using a computer program. The value of each pixel is 0 or 1, which can be achieved by a random number generation function in a programming language (such as Python). The generated binary image is saved in an appropriate image format (such as PNG or TI FF) so that it can be loaded into a spatial light modulator; the laser source includes a laser diode and a solid-state laser. The laser beam passes through a spatial light modulator (SLM), and the spatial light modulator (SLM) modulates the laser beam according to the loaded binary speckle image, thereby generating a modulated light field, and this modulation can be a modulation of light intensity or a modulation of light phase;

[0066] Further, the modulated light beam is irradiated onto the target object. After passing through the target object, the light beam may pass through one or more media, such as lenses, optical fibers or other optical components, which affect the propagation and characteristics of the light, and finally the light intensity is measured by the bucket detector.

[0067] In some embodiments, ghost imaging is an imaging method based on speckle imaging technology. Unlike traditional imaging techniques, ghost imaging does not rely on direct optical images, but reconstructs images by statistically analyzing the speckle light field. In ghost imaging, light from a light source passes through a random scattering medium (such as fog or frosted glass) to form a light field with specific speckle patterns. These speckle patterns vary depending on the imaged object.

[0068] In some embodiments, the illumination speckle is a specific implementation of a speckle pattern, which is generally used to describe the speckle effect caused by the coherence of the light source and the lighting conditions. Illumination speckle is a speckle pattern formed under specific lighting conditions (such as using a coherent light source), and they are generally used to describe the speckle effect produced when light is irradiated onto an object, which affects the clarity and quality of the image.

[0069] In some embodiments, a spatial light modulator (SLM) is an optical device used to modulate the spatial distribution of light. It can control the intensity, phase or polarization state of light, and common types include liquid crystal spatial light modulator (LC-SLM) and micromirror array (MEMS-SLM).

[0070] In some embodiments, the expression of the illumination speckle is:

[0071] The expression for obtaining the light intensity value after passing the target object for the i-th time is:

[0072]

[0073] Among them, D(x,y) represents the target object, Indicates the medium, I i is the intensity value of light propagation, B i The light intensity after passing through the target object.

[0074] Step S20, adding an encoder, a bottleneck layer, and a decoder to the preset first visual converter model, integrating the coordinate attention mechanism into each visual converter block of the first visual converter model, and integrating the deep convolutions of multiple scales into the feedforward network of the first visual converter model to construct a second visual converter model;

[0075] After the light intensity value after passing the target object is captured by the bucket detector, an encoder, a bottleneck layer, and a decoder are added to the preset first visual converter model, a coordinate attention mechanism is integrated into each visual converter block of the first visual converter model, and multiple scales of deep convolution are integrated into the feedforward network of the first visual converter model to construct a second visual converter model;

[0076] Specifically, the first visual converter model is the base model of the visual converter model, and the second visual converter model is the improved visual converter model. The base model of the visual converter model is a deep learning model for computer vision tasks. It is based on the Transformer architecture, which was originally used for natural language processing. The main features of the visual converter include: image block partitioning: dividing the image into patches of fixed size, each patch is flattened into a one-dimensional vector; linear embedding: converting the flattened image patch into a high-dimensional feature vector through a linear layer; position encoding: adding position encoding to represent the position of each image patch and retaining spatial information; self-attention mechanism: capturing the long-distance dependency between image patches through the transformer's self-attention mechanism; classification head: the processed features are output through the classification head to the final result, such as image classification or other visual task prediction.

[0077] Furthermore, the second visual converter model adds an encoder, a bottleneck layer, and a decoder to the first visual converter model, fuses the coordinate attention mechanism into each visual converter block of the first visual converter model, and fuses deep convolutions of multiple scales into the feedforward network of the first visual converter model to construct the second visual converter model.

[0078] More specifically, see Figure 4 , add an encoder, a bottleneck layer, and a decoder structure to the preset first visual converter model, process the light intensity value after passing the target object, and convert it into a two-dimensional feature map. The encoder reduces the spatial resolution of the spatial features and captures information of different scales and levels. The bottleneck layer integrates the high-level abstract features from the encoder. Finally, the decoder restores the low-resolution feature map to the resolution of the input image; it is fused with the corresponding features in the encoder through jump connections to finally reconstruct the picture.

[0079] The coordinate attention mechanism is incorporated into each visual transformer block. Specifically, Figure 5 As shown in the figure, the coordinate attention mechanism pools the feature map in the X and Y directions respectively, transforms the generated feature map, and then performs a splicing operation. Then it is segmented along the spatial dimension, and a 1×1 convolution is used to increase the dimension, and the weighted feature map is obtained by combining the activation function.

[0080] The deep convolution of multiple scales is integrated into the feed-forward network. Specifically, Figure 6As shown in the figure, four deep convolutions of different scales (1×1, 3×3, 5×5 and 7×7) are inserted between two linear layers in the feedforward network to learn features of different scales, and each convolution processes a quarter of the channel. Finally, the features are reshaped into a one-dimensional format through point-wise convolution. Since there is a strong correlation between each ghost image measurement value, this method can simultaneously utilize local and global information, activate more pixels, and thus better restore high-frequency information in the image.

[0081] Step S30, inputting the light intensity value into the second visual converter model, and training it using a self-supervised method to determine an optimized illumination speckle, and reloading the optimized illumination speckle into the spatial light modulator to re-modulate the light field illuminated by the laser until the second visual converter model reaches a convergence state;

[0082] After constructing the second visual converter model, the light intensity value is input into the second visual converter model, and a self-supervised method is used for training to determine an optimized illumination speckle, and the optimized illumination speckle is reloaded into the spatial light modulator to re-modulate the light field illuminated by the laser until the second visual converter model reaches a convergence state;

[0083] For further information, see Figure 3 , inputting the light intensity value into the second visual converter model, and training it by a self-supervised method to determine an optimized illumination speckle, and reloading the optimized illumination speckle into the spatial light modulator to remodulate the light field illuminated by the laser until the second visual converter model reaches a convergence state, comprising:

[0084] Step S301, in each iteration of the second visual converter model training process, the laser irradiates the illumination speckle so that the bucket detector obtains the light intensity value after passing the target object;

[0085] Step S302: the light intensity value after passing through the target object is input into the second visual converter model, and a reconstructed image is obtained through a mapping matrix, an encoder, a bottleneck layer and a decoder;

[0086] Step S303: use a preset loss function to calculate the mean square error between the reconstructed image and the original image, perform back propagation on the network to optimize the network parameters and illumination speckle, and reload the optimized illumination speckle to the spatial light modulator on the computer. Repeat the above steps until the preset number of iterations is reached, and the training of the second visual converter model is completed.

[0087] Step S40: input the light intensity value corresponding to the optimized illumination speckle into the second visual converter model that has been trained to a convergence state, determine the reconstructed image of the target object, and complete the computational ghost imaging reconstruction at a low sampling rate.

[0088] After the second visual converter model is trained to a convergent state, the light intensity value corresponding to the optimized illumination speckle is input into the second visual converter model that has been trained to a convergent state to determine a reconstructed image of the target object, so as to complete computational ghost imaging reconstruction at a low sampling rate.

[0089] Furthermore, the light intensity value corresponding to the optimized illumination speckle is input into the second visual converter model that has been trained to a convergent state to determine the reconstructed image of the target object, so as to complete the step of computational ghost imaging reconstruction at a low sampling rate, including:

[0090] Step S401, processing the light intensity value after passing through the target object and converting it into a two-dimensional feature map;

[0091] Step S402: using the encoder to reduce the spatial resolution of spatial features to capture information of different scales and levels;

[0092] Step S403, using the bottleneck layer to integrate high-level abstract features from the encoder;

[0093] Step S404: The decoder restores the low-resolution feature map to the resolution of the input image, and fuses it with the corresponding features in the encoder through a jump connection to determine a reconstructed image of the target object.

[0094] Furthermore, the light intensity value corresponding to the optimized illumination speckle is input into the second visual converter model that has been trained to a convergent state to determine the reconstructed image of the target object, so as to complete the step of computational ghost imaging reconstruction at a low sampling rate, including:

[0095] Step S4001: Integrate the coordinate attention mechanism into each visual converter block, use the coordinate attention mechanism to pool the two-dimensional feature map in the X direction and the Y direction respectively, transform the generated feature map, and perform a splicing operation;

[0096] Step S4002: Segment along the spatial dimension, use 1×1 convolution to perform dimensionality increase operation, and combine the activation function to obtain a weighted feature map.

[0097] Furthermore, the light intensity value corresponding to the optimized illumination speckle is input into the second visual converter model that has been trained to a convergent state to determine the reconstructed image of the target object, so as to complete the step of computational ghost imaging reconstruction at a low sampling rate, including:

[0098] The deep convolutions of multiple scales are fused into the feedforward network. Four deep convolutions of different scales are inserted between two linear layers in the feedforward network to learn features of different scales. Each deep convolution processes a quarter of the channel and reshapes the features into a one-dimensional format through point-wise convolution.

[0099] It can be seen from the above embodiments that, compared with the prior art, the present application aims at the problems in the prior art that the reconstruction quality of the ghost imaging method is not high at an extremely low sampling rate, and the reconstruction quality of the ghost imaging method based on deep learning is not ideal. The present application includes but is not limited to the following beneficial effects:

[0100] First, this application can enhance channel attention. According to the strong correlation between ghost imaging measurements, the coordinate attention mechanism is incorporated into the visual converter block, which effectively improves the imaging quality.

[0101] Secondly, the present application inserts deep convolutions of different scales between two linear layers in a feedforward network, performs feature fusion at multiple scales, and realizes the effective transmission of information and features between multiple feature layers;

[0102] Third, the computational ghost imaging reconstruction method based on the improved visual converter of the present application is trained by simultaneously optimizing the optimization strategy of the illumination speckle and the visual converter model. The improved visual converter combines local and global information and multi-scale feature fusion, which effectively improves the reconstruction efficiency and reconstruction quality of computational ghost imaging images under extremely low sampling rates.

[0103] See also Figure 7, a computational ghost imaging reconstruction device at a low sampling rate is provided to meet one of the purposes of the present application, including a light intensity value acquisition module 1100, a second visual converter construction module 1200, a second visual converter training module 1300 and a ghost imaging reconstruction module 1400. Among them, the light intensity value acquisition module 1100 is configured to respond to the computational ghost imaging reconstruction instruction at a low sampling rate, randomly initialize and generate illumination speckle and load it into the spatial light modulator to control the light field irradiated by the laser, the laser irradiates the spatial light modulator to determine the modulated light beam, and the modulated light beam is irradiated to the target object and passes through one or more media, and the light intensity value after passing the target object is captured by the bucket detector; the second visual converter construction module 1200 is configured to add an encoder, a bottleneck layer and a decoder to the preset first visual converter model, fuse the coordinate attention mechanism into each visual converter block of the first visual converter model, and fuse multiple scales of deep convolution into the first A feedforward network of a visual converter model is used to construct a second visual converter model; a second visual converter training module 1300 is configured to input the light intensity value into the second visual converter model, and train it using a self-supervised method to determine an optimized illumination speckle, and reload the optimized illumination speckle into the spatial light modulator to remodulate the light field irradiated by the laser until the second visual converter model reaches a convergence state; a ghost imaging reconstruction module 1400 is configured to input the light intensity value corresponding to the optimized illumination speckle into the second visual converter model that has been trained to a convergence state, determine a reconstructed image of the target object, and complete computational ghost imaging reconstruction at a low sampling rate.

[0104] Based on any embodiment of this application, please refer to Figure 8 Another embodiment of the present application further provides an electronic device, which can be implemented by a computer device, such as Figure 8 As shown, a schematic diagram of the internal structure of a computer device. The computer device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. Among them, the computer-readable storage medium of the computer device stores an operating system, a database, and computer-readable instructions. The database may store a control information sequence. When the computer-readable instructions are executed by the processor, the processor can implement a method for computational ghost imaging reconstruction at a low sampling rate. The processor of the computer device is used to provide computing and control capabilities to support the operation of the entire computer device. The memory of the computer device may store computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor can execute the method for computational ghost imaging reconstruction at a low sampling rate of the present application. The network interface of the computer device is used to connect and communicate with a terminal. Those skilled in the art will understand that Figure 8The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0105] In this embodiment, the processor is used to execute Figure 7 The memory stores the program code and various data required to execute the above modules or submodules. The network interface is used to transmit data between user terminals or servers. The memory in this embodiment stores the program code and data required to execute all modules / submodules in the computational ghost imaging reconstruction device at a low sampling rate of this application, and the server can call the program code and data of the server to execute the functions of all submodules.

[0106] The present application also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the method for computational ghost imaging reconstruction at a low sampling rate described in any embodiment of the present application.

[0107] The present application also provides a computer program product, including a computer program / instruction, which, when executed by one or more processors, implements the steps of the method for computational ghost imaging reconstruction at a low sampling rate described in any embodiment of the present application.

[0108] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments of the present application can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a computer-readable storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0109] The above description is only a partial implementation method of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

[0110] In summary, the computational ghost imaging reconstruction method based on the improved visual converter of the present application is trained by simultaneously optimizing the optimization strategy of the illumination speckle and the visual converter model. The improved visual converter combines local and global information and multi-scale feature fusion, which effectively improves the reconstruction efficiency and reconstruction quality of computational ghost imaging images under extremely low sampling rates.

Claims

1. A method for computational ghost imaging reconstruction at low sampling rate, characterized in that: include: In response to a computational ghost imaging reconstruction instruction at a low sampling rate, randomly initialize and generate an illumination speckle and load it into a spatial light modulator to control a light field irradiated by a laser, irradiate the spatial light modulator with a laser to determine a modulated light beam, irradiate the modulated light beam to a target object and after passing through one or more media, a bucket detector captures a light intensity value after passing through the target object; Adding an encoder, a bottleneck layer, and a decoder to the preset first visual converter model, integrating the coordinate attention mechanism into each visual converter block of the first visual converter model, and integrating multiple scales of deep convolution into the feedforward network of the first visual converter model to construct a second visual converter model; Inputting the light intensity value into the second visual converter model and training it using a self-supervised method to determine an optimized illumination speckle, and reloading the optimized illumination speckle into the spatial light modulator to remodulate the light field illuminated by the laser until the second visual converter model reaches a convergence state; The light intensity value corresponding to the optimized illumination speckle is input into the second visual converter model that has been trained to a convergence state to determine the reconstructed image of the target object, so as to complete the computational ghost imaging reconstruction at a low sampling rate.

2. The method for computational ghost imaging reconstruction at a low sampling rate according to claim 1, characterized in that: Inputting the light intensity value corresponding to the optimized illumination speckle into the second visual converter model that has been trained to a convergent state to determine the reconstructed image of the target object to complete the step of computational ghost imaging reconstruction at a low sampling rate, including: Process the light intensity value after passing through the target object and convert it into a two-dimensional feature map; The encoder is used to reduce the spatial resolution of spatial features and capture information at different scales and levels; The bottleneck layer is used to integrate the abstract features from the high-level of the encoder; The decoder restores the low-resolution feature map to the resolution of the input image and fuses it with the corresponding features in the encoder through jump connections to determine the reconstructed image of the target object.

3. The method for computational ghost imaging reconstruction at a low sampling rate according to claim 2, characterized in that: Inputting the light intensity value corresponding to the optimized illumination speckle into the second visual converter model that has been trained to a convergent state to determine the reconstructed image of the target object to complete the step of computational ghost imaging reconstruction at a low sampling rate, including: The coordinate attention mechanism is integrated into each visual converter block, and the coordinate attention mechanism is used to pool the two-dimensional feature map in the X direction and the Y direction respectively, and the generated feature map is transformed and concatenated; The images are segmented along the spatial dimension, and 1×1 convolution is used to increase the dimension, and the weighted feature map is obtained by combining the activation function.

4. The method for computational ghost imaging reconstruction at a low sampling rate according to claim 3, characterized in that: Inputting the light intensity value corresponding to the optimized illumination speckle into the second visual converter model that has been trained to a convergent state to determine the reconstructed image of the target object to complete the step of computational ghost imaging reconstruction at a low sampling rate, including: The deep convolutions of multiple scales are fused into the feedforward network. Four deep convolutions of different scales are inserted between two linear layers in the feedforward network to learn features of different scales. Each deep convolution processes a quarter of the channel and reshapes the features into a one-dimensional format through point-wise convolution.

5. The method for computational ghost imaging reconstruction at a low sampling rate according to claim 1, characterized in that: The step of inputting the light intensity value into the second visual converter model and training it by a self-supervised method to determine an optimized illumination speckle, and reloading the optimized illumination speckle into the spatial light modulator to remodulate the light field illuminated by the laser until the second visual converter model reaches a convergence state comprises: In each iteration of the second visual converter model training process, the laser irradiates the illumination speckle so that the bucket detector obtains the light intensity value after passing the target object; The light intensity value after passing through the target object is input into the second visual converter model, and the reconstructed image is obtained through the mapping matrix, encoder, bottleneck layer and decoder; The preset loss function is used to calculate the mean square error between the reconstructed image and the original image, and the network parameters and illumination speckle are optimized by back-propagation. The computer reloads the optimized illumination speckle to the spatial light modulator and repeats the above steps until the preset number of iterations is reached to complete the training of the second visual converter model.

6. The method for computational ghost imaging reconstruction at low sampling rate according to claim 1, characterized in that: The expression of the illumination speckle is: The expression for obtaining the light intensity value after passing the target object for the i-th time is: Among them, D(x,y) represents the target object, Indicates the medium, I i is the intensity value of light propagation, B i The light intensity after passing through the target object.

7. The method for computational ghost imaging reconstruction at a low sampling rate according to any one of claims 1 to 6, characterized in that: The first visual converter model is a basic model of the visual converter model, and the second visual converter model is an improved visual converter model.

8. A computational ghost imaging reconstruction device at a low sampling rate, characterized in that: include: A light intensity value acquisition module is configured to respond to a computational ghost imaging reconstruction instruction at a low sampling rate, randomly initialize and generate an illumination speckle and load it into a spatial light modulator to control a light field irradiated by a laser, irradiate the spatial light modulator with a laser to determine a modulated light beam, irradiate the modulated light beam to a target object and after passing through one or more media, a light intensity value after passing through the target object is captured by a bucket detector; a second visual converter building module, configured to add an encoder, a bottleneck layer, and a decoder to a preset first visual converter model, fuse a coordinate attention mechanism into each visual converter block of the first visual converter model, and fuse multiple scales of deep convolution into a feedforward network of the first visual converter model to build a second visual converter model; a second visual converter training module, configured to input the light intensity value into the second visual converter model, and perform training using a self-supervised method to determine an optimized illumination speckle, and reload the optimized illumination speckle into the spatial light modulator to re-modulate the light field illuminated by the laser until the second visual converter model reaches a convergence state; The ghost imaging reconstruction module is configured to input the light intensity value corresponding to the optimized illumination speckle into a second visual converter model that has been trained to a convergence state, determine a reconstructed image of the target object, and complete computational ghost imaging reconstruction at a low sampling rate.

9. An electronic device, comprising a central processing unit and a memory, characterized in that: The central processing unit is used to call and run the computer program stored in the memory to execute the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: It stores a computer program implemented according to the method described in any one of claims 1 to 7 in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the corresponding method are executed.

Images