Lightweight and compact programmable optoelectronic hybrid computing target detection remote sensing system

Abstract

A lightweight, programmable, optoelectronic hybrid computing target detection remote sensing system comprises: an optical system that focuses the incoherent light field of a remote sensing image onto the focal plane of a detector; a first detector that converts the incoherent light field into a digital remote sensing image; a memory that stores the digital remote sensing image; an image segmentation and driving circuit that constructs a sliding window of the digital remote sensing image and sequentially selects regions of interest (ROIs) for input to a spatial light modulator; a coherent light source and illumination system that generates a uniformly distributed coherent light field and illuminates the surface of the spatial light modulator; a spatial light modulator that modulates the coherent light field to generate a coherent light field for the RIO; a programmable optical computing device that performs target detection or classification operations on the coherent light field of the remote sensing image based on the type of target to be classified; a second detector that acquires the computation results from the programmable optical computing device; and an inference circuit that determines the location of the target region based on the acquired information and image segmentation information.

Description

Technical Field

[0001] This invention relates to the field of on-orbit image processing for remote sensing satellites, and particularly to a lightweight, programmable, optoelectronic hybrid computing target detection remote sensing system. Background Technology

[0002] With the development of optoelectronic imaging technology, the resolution of remote sensing images is increasing daily, placing immense pressure on satellites for on-orbit image processing and data downlink. Constrained by the size, weight, and power consumption limitations of satellites during launch and operation, high-performance, high-power electronic chips such as CPUs, GPUs, and FPGAs cannot be used for on-board processing. Optical computing, utilizing the physical effects of light interference and diffraction, can process complex problems within electronic chips in parallel at the speed of light, offering dual advantages in computing power and energy consumption. It has the potential to resolve the contradiction between the demand for on-orbit processing of massive amounts of remote sensing image data and the design requirements of remote sensing payloads in terms of power consumption, size, and weight.

[0003] The invention patent "Optical Logic Element and Logic Operation Method for Optoelectronic Digital Logic Operation" (Tsinghua University, CN202111198459.3) proposes an optical neural network computing device based on the principle of diffraction. However, the optical diffraction neural network proposed in this invention is limited by device technology and scale, resulting in limited complexity of the logic operations it can handle, making it unsuitable for large-scale remote sensing satellite image processing problems. Furthermore, the structure of the optical diffraction array described in this patent is determined by the digital logic operation function of a pre-defined optical diffraction neural network, meaning it can only handle specific problems and lacks the ability to update and adaptively handle new problems based on changes in the task. Summary of the Invention

[0004] The technical problem solved by this invention is to overcome the shortcomings of the prior art and provide a lightweight, programmable optoelectronic hybrid computing target detection remote sensing system.

[0005] The solution provided by this invention is: a lightweight, programmable, optoelectronic hybrid computing target detection remote sensing system, comprising:

[0006] An optical system is used to focus the incoherent light field of a remotely sensed image onto the focal plane of the detector;

[0007] The first detector is used to convert the incoherent light field of the remote sensing image into a digital remote sensing image;

[0008] Memory, used to store digital remote sensing images;

[0009] Image segmentation and driving circuitry is used to construct a sliding window for digital remote sensing images, sequentially selecting regions of interest and inputting them into a spatial light modulator.

[0010] Coherent light source and illumination system for generating a uniformly distributed coherent light field and illuminating the surface of a spatial light modulator;

[0011] Spatial light modulators are used to modulate coherent light fields to generate coherent light fields for remote sensing images of regions of interest.

[0012] Programmable optical computing devices are used to perform target detection or classification operations on the coherent light field of remote sensing images according to the type of target to be classified.

[0013] The second detector is used to collect the computation results of the programmable optical computing device;

[0014] The inference circuit is used to determine the location of the target area based on the information acquired by the second detector and the image segmentation information.

[0015] Preferably, the first detector includes, but is not limited to, converting remotely sensed incident light fields in the visible, infrared, or ultraviolet spectral bands into digital remote sensing images.

[0016] Preferably, the image segmentation and driving circuit segments the original digital remote sensing image with M×N pixels into k m×n region of interest images, and converts the region of interest images into a data stream in spatial light modulator read format according to the time sequence, with adjacent region of interest images overlapping by several pixels.

[0017] Preferably, the number of modulation units in the spatial light modulator is P. r ×P c Then the value of k is [M / P] r ]×[N / P c [] indicates rounding up, and the number of overlapping pixels does not exceed the diagonal length of the target to be detected as a percentage of the number of pixels in the image.

[0018] Preferably, the step of converting the segmented region of interest image into a data stream in a spatial light modulator readout format according to the time sequence includes the following steps:

[0019] (1) Perform image segmentation according to the spatial order from left to right and from top to bottom;

[0020] (2) The segmented regions of interest are sequentially labeled with numbers 1, 2, ..., k;

[0021] (3) At time t1, the image data with sequence number 1 is copied to the read memory address of the spatial light modulator; at time t2, the image data with sequence number 2 is copied to the read memory address of the spatial light modulator; ..., t k At any given time, the image data with sequence number k is copied to the read memory address of the spatial light modulator.

[0022] Preferably, the spatial light modulator modulates the coherent light field generated by the coherent light source with a single intensity, a single phase, or both intensity and phase simultaneously to generate a temporally segmented image coherent light field.

[0023] Preferably, the programmable optical computing device includes two Fourier lenses, a programmable liquid crystal on silicon (LCD), and a nonlinear crystal. The input region of interest (ROI) image, the first Fourier lens, the programmable LCD, the second Fourier lens, the nonlinear crystal, and the output target detection result image are arranged sequentially and centered. The ROI image, the programmable LCD, and the first Fourier lens are spaced one lens focal length apart, as are the programmable LCD, the nonlinear crystal, and the second Fourier lens. The ROI image is focused by the first Fourier lens, forming a spectral distribution image on the surface of the programmable LCD. The programmable LCD loads a neural network convolution kernel pattern for target detection or classification in a specific remote sensing scene, performs convolution calculations on the spectral image, restores the frequency domain convolution image through the second Fourier lens, and performs nonlinear operations through the nonlinear crystal to obtain the result calculated on the ROI image, forming the target detection result image.

[0024] Preferably, for different remote sensing scenarios, the convolution kernel pattern loaded by the programmable silicon-based liquid crystal can be changed to realize the neural network structure and function for different types of target detection or classification.

[0025] The beneficial effects of this invention compared with the prior art are as follows: Compared with the prior art, this invention can significantly improve the on-orbit image processing and target detection capabilities and speed of remote sensing imaging systems, while meeting the requirements of miniaturization, lightweighting and low power consumption of remote sensing payloads. It can be used for multispectral collaborative target detection, further improving target recognition capabilities and intelligence accuracy, realizing intelligent remote sensing payload design, and serving the construction of a space-based cloud-edge-device intelligent computing system. Attached Figure Description

[0026] Figure 1 This is a block diagram illustrating the structural composition of a lightweight, programmable, optoelectronic hybrid computing target detection remote sensing system proposed in this invention.

[0027] Figure 2 This is a flowchart of the optoelectronic hybrid computing process of a lightweight, programmable optoelectronic hybrid computing target detection remote sensing system proposed in this invention.

[0028] Figure 3 This is a schematic diagram of the optoelectronic hybrid computing process of a lightweight programmable optoelectronic hybrid computing target detection remote sensing system proposed in this invention;

[0029] Figure 4 This refers to the optical remote sensing image acquisition and segmentation process based on existing remote sensing imaging payload systems.

[0030] Figure 5After segmenting optical remote sensing images, the spatial domain image sequence is converted into a temporal domain image sequence through electrical computation, and then loaded into a remote sensing coherent image light field generating device to generate a coherent optical remote sensing image temporal sequence.

[0031] Figure 6 The coherent optical remote sensing image sequence is input to the optical computing device in a temporal sequence, and the device sequentially detects whether there is a target in the segmented image region. The location of the target region is output based on the optical computing results.

[0032] Figure 7 This is a schematic diagram of a specific embodiment of the present invention, wherein 10-existing remote sensing imaging payload system, including: 101-natural light field of remote sensing image, 102-optical system, 103-first detector, 104-memory; 20-remote sensing image segmentation, spatiotemporal conversion and coherent light field generation, including: 201-image segmentation and driving circuit, 202-digital micromirror array (DMD), 203-coherent light source, 204-converging lens, 205-collimating lens; 30-optical computing unit assembly, including: 301-aperture stop, 302-optical computing device, 303-second detector, 304-inference circuit;

[0033] Figure 8 This is a schematic diagram of an optical computing device according to a specific embodiment of the present invention, wherein 401-region of interest image, 402-Fourier lens, 403-programmable liquid crystal on silicon (LCOS), 404-Fourier lens, 405-nonlinear lithium niobate crystal, and 406-target detection result image.

[0034] Figure 9 This is a schematic diagram of a four-class geometric target according to a specific embodiment of the present invention;

[0035] Figure 10 This is a geometric target detection result diagram of a specific embodiment of the present invention, showing triangles, circles, squares, and hexagons detected from left to right. Detailed Implementation

[0036] The present invention will be further described below with reference to the embodiments.

[0037] This invention aims to resolve the contradiction between the need for on-orbit processing of massive amounts of remote sensing image data and the design requirements of remote sensing payloads in terms of power consumption, size, and weight. Specifically, it addresses the following technical problems:

[0038] ① It is compatible with existing remote sensing imaging payload systems and generates a coherent light field for remote sensing images that can be processed using optical computing methods.

[0039] ② Compatible with 4f optical computing systems, enabling existing remote sensing payloads to have high-speed, low-power target detection capabilities;

[0040] ③ Enable intelligent optical computing remote sensing payloads to have on-board programmability and algorithm update capabilities;

[0041] ④ Applicable to broadband remote sensing payloads, optical remote sensing imaging systems in the visible, infrared or any spectral band can use optical computing methods for on-orbit target detection;

[0042] ⑤ To achieve the target detection function of optoelectronic hybrid computing while meeting the design requirements of miniaturization and lightweighting of the payload.

[0043] like Figure 1 As shown, this invention proposes a lightweight, programmable, optoelectronic hybrid computing target detection remote sensing system, comprising: an optical system for focusing an incoherent light field from a remote sensing image onto the focal plane of a detector; a first detector for converting the incoherent light field of the remote sensing image into a digital remote sensing image; a memory for storing the digital remote sensing image output by the detector; an image segmentation and driving circuit for constructing a sliding window of the digital remote sensing image and sequentially selecting the region of interest (ROI) image and inputting it into a spatial light modulator; a coherent light source and illumination system for generating a uniformly distributed coherent light field and illuminating the surface of the spatial light modulator; a spatial light modulator for modulating the coherent light field to generate a coherent light field of the remote sensing image for the ROI; a programmable optical computing device for performing target detection and classification operations on the coherent light field of the remote sensing image according to the type of target to be classified; a second detector for acquiring the operation results of the programmable optical computing device; and an inference circuit for determining the location of the target region based on the information acquired by the second detector and the image segmentation information.

[0044] The detector converts the remote sensing incident light field in the visible, infrared, or ultraviolet spectral bands into digital remote sensing images.

[0045] The image segmentation and driving circuit segments the original digital remote sensing image with M×N pixels into k m×n region of interest images, with adjacent regions of interest overlapping by several pixels.

[0046] The number of modulation units in the spatial light modulator is P r ×P c Then the value of k is [M / P] r ]×[N / P c [] indicates rounding up, and the number of overlapping pixels does not exceed the diagonal length of the target to be detected as a percentage of the number of pixels in the image.

[0047] The segmented region of interest image is sequentially converted into a data stream in a spatial light modulator readout format according to the time sequence, including the following steps:

[0048] (1) Perform image segmentation according to the spatial order from left to right and from top to bottom;

[0049] (2) The segmented regions of interest are sequentially labeled with numbers 1, 2, ..., k;

[0050] (3) At time t1, the image data with sequence number 1 is copied to the read memory address of the spatial light modulator; at time t2, the image data with sequence number 2 is copied to the read memory address of the spatial light modulator; ..., t k At any given time, the image data with sequence number k is copied to the read memory address of the spatial light modulator.

[0051] The coherent light source and illumination system generate a uniform coherent beam in the visible spectrum or any spectrum, with the beam area covering the surface of the spatial light modulator.

[0052] The spatial light modulator modulates the coherent light field generated by the coherent light source with a single intensity, a single phase, or both intensity and phase simultaneously.

[0053] The programmable optical computing device enables target detection, classification, and inference based on convolutional neural networks. It includes a programmable spatial light modulator, allowing for structural reconstruction and functional updates of the optical computing device. Specifically, the programmable optical computing device comprises two Fourier lenses, a programmable liquid crystal on silicon (LCD), and a nonlinear crystal. The components are arranged sequentially in the order of the input region of interest (ROI) image, the first Fourier lens, the programmable LCD, the second Fourier lens, the nonlinear crystal, and the output target detection result image, with center alignment. The ROI image, the programmable LCD, and the first Fourier lens are spaced one lens focal length apart, as are the programmable LCD, the nonlinear crystal, and the second Fourier lens. The ROI image is focused by the first Fourier lens, forming a spectral distribution image on the surface of the programmable LCD. The programmable LCD loads a neural network convolution kernel pattern for target detection or classification in a specific remote sensing scene, performs convolution calculations on the spectral image, restores the frequency domain convolution image through the second Fourier lens, and performs nonlinear operations through the nonlinear crystal to obtain the result calculated on the ROI image, forming the target detection result image. By changing the convolution kernel pattern loaded by a programmable silicon-based liquid crystal for different remote sensing scenarios, neural network structures and functions for different types of target detection or classification can be realized.

[0054] Example

[0055] A lightweight, programmable, optoelectronic hybrid computing target detection remote sensing system, such as Figure 7As shown, under the existing remote sensing imaging payload system 10, the natural light field 101 of the remote sensing image is focused onto the focal plane of the first detector 103 by the optical system 102. The first detector converts the natural light field of the remote sensing image into a digital image, and the memory 104 stores the digital image. Then, remote sensing image segmentation, spatiotemporal conversion, and coherent light field generation 20 are realized. Specifically, the image segmentation and driving circuit 201 segments the acquired digital image and inputs the segmented images sequentially into a reflective digital micromirror array (DMD) according to the spatial order from left to right and from top to bottom. )202, the coherent light source 203 generates parallel coherent light through the converging lens 204 and collimating lens 205, which illuminates the surface of the reflective digital micromirror array (DMD) 202. The DMD modulates the coherent light field to generate a temporal region of interest image, which is input to the optical computing unit 30 and passes through the aperture stop 301. The optical computing device 302 performs target detection or classification operations on the coherent light field of the region of interest image. The second detector 303 collects the optical computing output results, and the inference circuit 304 determines the location of the target region based on the information collected by the second detector and the image segmentation information.

[0056] The image segmentation and driving circuit segments the digital image into 1024×768 region of interest images, and converts the region of interest images into a data stream in spatial light modulator read format according to the spatial sequence. The images of adjacent regions of interest overlap by 100 pixels.

[0057] The coherent light source generates a laser beam with a wavelength of 532nm.

[0058] The programmable optical computing device includes a 4f system, a programmable liquid crystal on silicon (LCOS), and a nonlinear lithium niobate crystal, enabling updates to the target detection computing function. For example... Figure 8As shown, the components are arranged in the following order: input region of interest image 401, first Fourier lens 402, programmable liquid crystal on silicon (LCOS) 403, second Fourier lens 404, nonlinear lithium niobate crystal 405, and output target detection result image 406, all center-aligned. The region of interest image, programmable liquid crystal on silicon, and first Fourier lens are spaced one lens focal length apart, as are the programmable liquid crystal on silicon, nonlinear crystal, and second Fourier lens. The region of interest image is focused by the first Fourier lens, forming a spectral distribution image on the surface of the programmable liquid crystal on silicon. The programmable liquid crystal on silicon loads a neural network convolution kernel pattern for target detection or classification in a specific remote sensing scenario, performs convolution calculations on the spectral image, restores the image after frequency domain convolution through the second Fourier lens, and performs nonlinear operations through the nonlinear crystal to obtain the result calculated on the region of interest image, forming the target detection result image. For different remote sensing scenarios, the convolution kernel pattern loaded on the programmable liquid crystal can be changed to achieve different neural network structures and functions for target detection or classification.

[0059] The task of the specific embodiment is as follows: Figure 9 As shown, the correct output is the category of the four-class geometric target.

[0060] The results of the specific embodiments are as follows: Figure 10 As shown, images containing triangles, circles, squares, and hexagons are acquired sequentially. The images output by the optical computing device have the highest brightness in the triangle, circle, square, and hexagon categories, respectively, indicating that the target category is correctly output.

[0061] Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make possible changes and modifications to the technical solutions of the present invention by utilizing the methods and techniques disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solutions of the present invention shall fall within the protection scope of the technical solutions of the present invention.

Claims

1. A lightweight, programmable, optoelectronic hybrid computing target detection remote sensing system, characterized in that... include: An optical system is used to focus the incoherent light field of a remotely sensed image onto the focal plane of the detector; The first detector is used to convert the incoherent light field of the remote sensing image into a digital remote sensing image; Memory, used to store digital remote sensing images; Image segmentation and driving circuitry is used to construct a sliding window for digital remote sensing images, sequentially selecting regions of interest and inputting them into a spatial light modulator. Coherent light source and illumination system for generating a uniformly distributed coherent light field and illuminating the surface of a spatial light modulator; Spatial light modulators are used to modulate coherent light fields to generate coherent light fields for remote sensing images of regions of interest. Programmable optical computing devices are used to perform target detection or classification operations on the coherent light field of remote sensing images according to the type of target to be classified. The second detector is used to collect the computation results of the programmable optical computing device; The inference circuit is used to determine the location of the target area based on the information acquired by the second detector and the image segmentation information.

2. The system according to claim 1, characterized in that: The first detector includes, but is not limited to, converting remotely sensed incident light fields in the visible, infrared, or ultraviolet spectral bands into digital remote sensing images.

3. The system according to claim 1, characterized in that: The image segmentation and driving circuit segments the original digital remote sensing image with M×N pixels into k m×n region of interest images, and converts the region of interest images into a data stream in spatial light modulator read format according to the time sequence. The images of adjacent regions of interest overlap by several pixels.

4. The system according to claim 3, characterized in that: The number of modulation units in the spatial light modulator is P r ×P c Then the value of k is [M / P] r ]×[N / P c [] indicates rounding up, and the number of overlapping pixels does not exceed the diagonal length of the target to be detected as a percentage of the number of pixels in the image.

5. The system according to claim 3, characterized in that: The segmented region of interest image is sequentially converted into a data stream in a spatial light modulator readout format according to the time sequence, including the following steps: (1) Perform image segmentation according to the spatial order from left to right and from top to bottom; (2) The segmented regions of interest are sequentially labeled with numbers 1, 2, ..., k; (3) At time t1, the image data with sequence number 1 is copied to the read memory address of the spatial light modulator; at time t2, the image data with sequence number 2 is copied to the read memory address of the spatial light modulator; ..., t k At any given time, the image data with sequence number k is copied to the read memory address of the spatial light modulator.

6. The system according to claim 1, characterized in that: The spatial light modulator modulates the coherent light field generated by the coherent light source with a single intensity, a single phase, or both intensity and phase simultaneously, to generate a temporally segmented image coherent light field.

7. The system according to claim 1, characterized in that: The programmable optical computing device includes two Fourier lenses, a programmable liquid crystal on silicon (LCD), and a nonlinear crystal. The input region of interest (ROI) image, the first Fourier lens, the programmable LCD, the second Fourier lens, the nonlinear crystal, and the output target detection result image are arranged sequentially and centered. The ROI image, the programmable LCD, and the first Fourier lens are spaced one lens focal length apart, as are the programmable LCD, the nonlinear crystal, and the second Fourier lens. The ROI image is focused by the first Fourier lens, forming a spectral distribution image on the surface of the programmable LCD. The programmable LCD loads a neural network convolution kernel pattern for target detection or classification in a specific remote sensing scene, performs convolution calculations on the spectral image, restores the frequency domain convolution image through the second Fourier lens, and performs nonlinear operations through the nonlinear crystal to obtain the result calculated on the ROI image, forming the target detection result image.

8. The system according to claim 7, characterized in that: By changing the convolution kernel pattern loaded by a programmable silicon-based liquid crystal for different remote sensing scenarios, neural network structures and functions for different types of target detection or classification can be realized.

Images