Multi-aperture imaging detection device and method for moving target in low-illumination environment

Abstract

The application relates to a multi-aperture imaging detection device and method of a moving target in a low-illumination environment, which comprises K sub-aperture cameras arranged side by side in a 1xK configuration, wherein K is an odd number; an imaging unit configured with a low-light detection chip and used for acquiring images; a synchronous exposure control unit used for controlling synchronous exposure of the K sub-aperture cameras; a mode switching control unit used for controlling switching of imaging modes of the K sub-aperture cameras, wherein the imaging modes include a parallel light path mode and a splicing mode; and an image processing unit used for acquiring high-resolution images and target information. The application solves the contradiction between a large field of view and high resolution, and can balance the large field of view while improving the resolution.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of space optical multi-aperture imaging technology, and in particular to a multi-aperture imaging detection device and method for moving targets in a low-illumination environment. BACKGROUND

[0002] Space remote sensing optical detector technology provides a new means for detecting targets from a space perspective, and in particular, with the improvement of optical platform and detector capabilities, remote sensing satellites can achieve short-time multi-frame continuous shooting, which makes it possible to implement super-temporal imaging. The concept of super-temporal imaging can be traced back to Amit Kumar's monograph "Land Cover Classification Methods Based on Super-Temporal Images" published in 2011. In the book, images with a revisit period of less than one day are referred to as super-temporal images. Therefore, using the continuous shooting mode of remote sensing satellites, multiple low-resolution images with high overlap can be obtained within a short time (minutes or even seconds). By using different high-frequency component information in the multiple images for image reconstruction, the ground spatial resolution and imaging quality of the camera can be further improved, and the target detection accuracy can be improved.

[0003] However, existing detection imaging devices mostly use single-aperture remote sensing detectors. Based on single-aperture remote sensing detectors, algorithms combining image quality improvement and target detection can improve the detection accuracy of targets, and good results have been achieved in the detection of stationary targets and targets with slow motion. However, when the target moves at a high speed, the use of this method will result in the loss of moving targets.

[0004] At the same time, for the requirement of all-time detection, single visible light detection cannot meet the requirement, especially in the detection of targets in the morning and evening and starlight-level environmental conditions. Compared with visible light detectors, low-light detectors have obvious advantages in night-time environmental detection. Low-light imaging technology can convert near-infrared radiation, extremely weak starlight, and other scene images into visible light images with enhanced brightness through low-light detectors for spectral and photoelectric conversion, thereby widening the effective working time of optical remote sensors. However, even if low-light imaging technology is used, the images formed in a weak light environment will still have low signal-to-noise ratio and low resolution.

[0005] Therefore, there is an urgent need for a detection method that can meet low-illumination detection. SUMMARY

[0006] To solve the technical problems existing in the prior art, the present application aims to provide a multi-aperture imaging detection device and method for moving targets in a low-illumination environment, which adopts a multi-aperture imaging mode, collects light beams emitted by remote objects by using a plurality of small-aperture subsystems arranged in a certain form array, and can obtain a plurality of super-time images at one time of exposure.

[0007] To achieve the above-mentioned application purposes, the present application provides a multi-aperture imaging detection device for moving targets in a low-illumination environment, which comprises:

[0008] K sub-aperture cameras arranged side by side in a 1xK configuration, wherein K is an odd number;

[0009] An imaging unit configured with a low-light detection chip for acquiring images;

[0010] A synchronous exposure control unit for controlling the K sub-aperture cameras to be exposed synchronously;

[0011] A mode switching control unit for controlling the imaging mode of the K sub-aperture cameras to be switched, wherein the imaging mode includes a parallel light path mode and a splicing mode;

[0012] An image processing unit for acquiring high-resolution images and target information.

[0013] According to one technical solution of the present application, the image processing unit at least comprises an image splicing unit, an image denoising and enhancement unit, a cloud detection and fusion unit, a target detection unit and a super-resolution reconstruction unit.

[0014] According to one technical solution of the present application, the image splicing unit is used for splicing the images acquired by the sub-aperture cameras;

[0015] The image denoising and enhancement unit is used for denoising and enhancing the spliced images;

[0016] The cloud detection and fusion unit is used for detecting and fusing the images after denoising and enhancement to remove clouds;

[0017] The target detection unit is used for detecting targets in the images after cloud detection and fusion removal processing;

[0018] The super-resolution reconstruction unit is used for obtaining high-resolution images containing targets and obtaining target information.

[0019] According to one aspect of the present application, an imaging detection method using the multi-aperture imaging detection device for moving targets in low-illumination environments is provided, which determines a detection mode based on current illumination, and acquires target information, wherein the detection mode includes daytime detection and nighttime detection.

[0020] According to one technical solution of the present application, the nighttime detection includes the following steps:

[0021] In step S101, a first time sequence of remote sensing images is acquired by using a parallel light path mode, wherein K sub-aperture cameras are used to capture images at the same time.

[0022] In step S102, a moving target is detected based on the K first time sequence of remote sensing images, and target information is acquired.

[0023] According to one technical solution of the present application, the daytime detection includes the following steps:

[0024] In step S201, a plurality of second time sequence of remote sensing images is acquired by using a stitching mode.

[0025] In step S202, a moving target is detected based on the plurality of second time sequence of remote sensing images.

[0026] In step S203, the stitching mode is switched to the parallel light path mode by using a mode switching control unit, and then the steps S101 and S102 are executed in sequence.

[0027] According to one technical solution of the present application, in step S201, K×J second time sequence of remote sensing images is acquired by using K apertures, and the resolution of a sub-aperture is M×N, wherein K is an odd number, and J represents different time points.

[0028] According to one technical solution of the present application, in step S202, the following steps are included:

[0029] In step S202, the second time sequence of remote sensing images acquired by the K apertures at J time points is stitched based on the positions of the sub-aperture cameras by using the image stitching unit, to obtain a second stitched image at time point j, and the resolution is M×(K×N-(K-1)t), wherein t is the length of the overlapping area, and j=1, 2, …, J. i

[0030] In step S202, the J second stitched images are subjected to image denoising and image enhancement processing by using the image denoising and enhancement unit.

[0031] In step S202, cloud detection and fusion removal are performed on the images after denoising and enhancement processing by using the cloud detection and fusion unit, to obtain a sequence of remote sensing images after cloud removal.

[0032] ​Based on the sequence remote sensing image after J amplitude cloud removal, target feature analysis is carried out, and the target is extracted.

[0033] According to one technical scheme of the present application, the original high-resolution image z is reconstructed by using a degradation model of the super-resolution reconstruction unit, and the degradation model is expressed as:

[0034] g k = tau * D * H * M * z + n k M k z + n k , k = 1,..., K

[0035] Wherein, g k is the observed low-resolution image of the kth sub-aperture camera; tau represents an integral factor; D represents a down-sampling matrix; H k represents a block circulant matrix used to represent a blur degradation process, at least including defocus, motion blur or blur caused by an optical system transfer function; M k is a matrix representing the position offset and geometric deformation of the kth image relative to the reference image, M k is a constant; z is the original high-resolution image; n k represents additive noise; and Q is the number of low-resolution images.

[0036] According to one technical scheme of the present application, the degradation model is solved by minimizing the objective function, and then:

[0037]

[0038] The maximum a posteriori probability method is used to solve formula (1), and then:

[0039]

[0040] Wherein, theta is a prior model parameter, and lambda is a weight parameter.

[0041] The reconstruction method based on the Huber Markov random field model is used to solve formula (2), and the high-resolution image containing the target is obtained.

[0042] Compared with the prior art, the present application has the following beneficial effects:

[0043] The low-illumination environment moving target multi-aperture imaging detection device and imaging detection device and method of the application, by arranging K sub-aperture cameras side by side, using a mode switching control unit to control switching the K sub-aperture cameras between parallel light path mode and splicing mode, when using splicing mode, the detection range can be expanded to realize large field of view detection, when using parallel light path mode, high resolution imaging is realized through a synchronous exposure control unit, so that high resolution images of the target are obtained by using super resolution technology, solving the contradiction between large field of view and high resolution, while improving resolution, large field of view can be considered.

[0044] Using multi-aperture high resolution imaging mode, the problem of loss of moving target when using traditional super resolution technology to improve quality while using multiple continuous images is solved, and the detection accuracy of the target is improved.

[0045] Using multi-aperture imaging mode, based on the image characteristics of the weak light detection chip, the image processing unit is configured with multiple images of synchronous exposure at the same time to perform quality improvement algorithm, improve image definition and signal-to-noise ratio, and can image in low-illumination environment, solving the problem of low signal-to-noise ratio and definition in weak light imaging. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0047] Figure 1 A flow chart schematically showing an imaging detection method realized by the low-illumination environment moving target multi-aperture imaging detection device in the embodiment of the application;

[0048] Figure 2 A large field of view imaging mode of the low-illumination environment moving target multi-aperture imaging detection device in the embodiment of the application is schematically shown;

[0049] Figure 3 A high resolution imaging mode of the low-illumination environment moving target multi-aperture imaging detection device in the embodiment of the application is schematically shown;

[0050] Figure 4 A schematic diagram of the low-illumination environment moving target multi-aperture imaging detection device in the embodiment of the application is schematically shown. DETAILED DESCRIPTION

[0051] The description of the embodiments of this specification should be taken in conjunction with the accompanying drawings, which form a part of this specification. In the drawings, the proportions of the shapes or thicknesses of the embodiments can be exaggerated and illustrated in a simplified or convenient manner. Furthermore, portions of the structures in the drawings will be described separately, and it should be noted that elements not shown or described in the drawings or by text are in forms known to those skilled in the art.

[0052] The description of the embodiments herein makes any reference to directions and orientations only for the convenience of description and should not be understood as any limitation on the scope of protection of the present application. The following description of the preferred embodiments will involve combinations of features, which can exist independently or in combination, and the present application is not particularly limited to the preferred embodiments. The scope of the present application is defined by the claims.

[0053] As shown in the drawings, Figures 1 to 4 A multi-aperture imaging detection device for moving targets in a low-illumination environment according to the present application comprises:

[0054] K sub-aperture cameras arranged side by side in a 1xK configuration, wherein K is an odd number;

[0055] An imaging unit composed of a CMOS detection chip and a lens, for acquiring images;

[0056] A synchronous exposure control unit for controlling the K sub-aperture cameras to expose synchronously;

[0057] A mode switching control unit for controlling the switching of the imaging modes of the K sub-aperture cameras, the imaging modes including a parallel light path mode and a stitching mode;

[0058] An image processing unit for acquiring high-resolution images and target information.

[0059] By arranging the K sub-aperture cameras side by side, the mode switching control unit is used to control the switching of the K sub-aperture cameras between the parallel light path mode and the stitching mode. When the stitching mode is used, the detection range can be expanded to realize large field-of-view detection. When the parallel light path mode is used, high-resolution imaging is realized through the synchronous exposure control unit, so that high-resolution images of the target are acquired using super-resolution technology, solving the contradiction between large field-of-view and high resolution, and enabling large field-of-view to be considered while improving resolution.

[0060] Using a multi-aperture high-resolution imaging mode can solve the problem of loss of moving targets when using traditional super-resolution technology to improve quality while using multiple consecutive images, improving the detection accuracy of the target.

[0061] The multi-aperture imaging mode is adopted, the image processing unit is configured with multiple frames of images for quality improvement algorithm based on the image characteristics of the weak light detection chip, the image definition and signal-to-noise ratio are improved, imaging can be realized in a low-illumination environment, and the problems of low signal-to-noise ratio and definition in weak light imaging are solved.

[0062] The mode switching control unit can control the switching of the parallel light path mode and the splicing mode by using a large-area fast mirror cascading splicing technology or a relay switch.

[0063] In some embodiments of the application, the image processing unit at least includes an image splicing unit, an image denoising and enhancement unit, a cloud detection and fusion unit, a target detection unit and a super-resolution reconstruction unit.

[0064] The image splicing unit is used for splicing the images obtained by the sub-apertures.

[0065] The image denoising and enhancement unit is used for denoising and enhancing the spliced images.

[0066] The cloud detection and fusion unit is used for cloud detection and fusion removal on the denoised and enhanced images.

[0067] The target detection unit is used for target detection on the images after cloud detection and fusion removal processing.

[0068] The super-resolution reconstruction unit is used for obtaining a high-resolution image containing a target and obtaining target information.

[0069] According to one aspect of the application, an imaging detection method for a multi-aperture imaging detection device for moving targets in a low-illumination environment is provided, which uses any of the above technical solutions, determines the detection mode based on the current illumination, and obtains target information, wherein the detection mode includes daytime detection and nighttime detection.

[0070] Based on the current illumination (light intensity), it is determined whether the current illumination is low, and if the current illumination is low, the nighttime detection mode is selected, otherwise the daytime detection mode is selected.

[0071] In the daytime detection, the splicing mode is used first by the mode switching control unit, which can expand the detection range and realize large field of view detection, after the target is determined, the parallel light path mode is switched by the mode switching control unit, and high-resolution imaging is realized by the synchronous exposure control unit to obtain target information.

[0072] For example, the sub-aperture camera plane array of the multi-aperture imaging detection device is 512*1024 in size, the sub-apertures are arranged in a 1*5 configuration in a line, and the overlapping area between the sub-apertures is 100.

[0073] In some embodiments of the present application, the night detection comprises the following steps:

[0074] In step S101, K images of first time sequence remote sensing images taken by the same sub-aperture camera at the same time are obtained in a parallel light path mode.

[0075] In step S102, a moving target is detected based on the K images of the first time sequence remote sensing images to obtain target information.

[0076] In night detection, a high-resolution mode of the multi-aperture camera is directly used to obtain high-resolution images z, target detection is performed based on the obtained sequence high-resolution images z to obtain target information, the background image can be eliminated by using the correlation of multiple images, the aerial moving object is separated from the ground background based on the background registration principle, clustering analysis, template matching and other methods are used to further extract the target of interest from the moving object, the similarity learning method is used, the pixels with high similarity are classified into a class by using the aggregation method, and the cumulative value or mean value estimation method of the class is used to detect whether the class sample is the target of interest.

[0077] The high-resolution mode is directly used in the night mode, which can avoid the problem of low target detection rate caused by low signal-to-noise ratio, thereby improving the detection accuracy and solving the problems of low signal-to-noise ratio and low definition in weak light imaging.

[0078] In some embodiments of the present application, the day detection comprises the following steps:

[0079] In step S201, a plurality of second time sequence remote sensing images are obtained in a stitching mode with the characteristics of large field of view detection.

[0080] K apertures obtain K*J images of the second time sequence remote sensing images, and the resolution of the sub-aperture is M*N, wherein K is an odd number, and J represents different time points. Figure 2 As shown in the accompanying drawings, 5*J images of the second time sequence remote sensing images are obtained, and the resolution of the sub-aperture is 512*1024.

[0081] The images obtained by the sub-aperture at j1 are stitched, the length of the overlapping area is 100, and the resolution of the stitched image is 512*(5*1024-400), i.e. 512*4720.

[0082] In step S202, a moving target is detected based on the plurality of second time sequence remote sensing images, specifically comprising:

[0083] The image stitching unit is used to stitch the second time sequence remote sensing images obtained at J time points based on the sub-aperture camera position, to obtain a second stitching image at time point j i , and the resolution is Mx(KxN-(K-1)t), t is the length of the overlapping area, and j=1, 2, …, J.

[0084] The image denoising and enhancement unit is used for image denoising and image enhancement processing on the J second stitching images, including: obtaining j2, j3, …, j J images, assuming that J=8, the obtained images are subjected to image denoising and image enhancement processing to preliminarily improve the image quality, a low-illumination image enhancement method based on the Retinex model is selected, and the illumination component and the reflection component of the image are constrained, and an optimization method is used to solve the objective function.

[0085] The cloud detection and fusion unit is used for cloud detection and fusion removal on the images after denoising and enhancement processing to obtain the sequence remote sensing images after cloud removal, including:

[0086] The obtained images are subjected to cloud detection and fusion removal, a method based on the U-Net network model is used to map the remote sensing image with clouds into an image without clouds and a cloud image, a robust principal component analysis (RPCA) method is used, the sparsity characteristics of the clouds due to dynamic changes in the time sequence are utilized, and the cloud detection result of the U-Net deep learning model is combined to realize cloud removal.

[0087] Based on the J sequence remote sensing images after cloud removal, the target is subjected to feature analysis and extraction, including: according to the J sequence remote sensing images, the target is subjected to spatio-temporal feature analysis and extraction, the background image can be eliminated by using the correlation of multiple images, the aerial moving object is separated from the ground background based on the background registration principle, clustering analysis, template matching and other methods are used to further extract the target of interest from the moving object, such as using a similarity learning method, using aggregation means to divide the pixels with high similarity into a class, and then using the cumulative value or mean value estimation method of the class to detect whether the class samples are the target of interest.

[0088] In step S203, the mode switching control unit is used to switch from the stitching mode to the parallel light path mode, and then the steps S101 and S102 are sequentially executed.

[0089] During daytime detection, the large field of view mode is used first to detect the moving target, and then the mode is switched (if no moving target is detected, the mode is not switched), the high-resolution mode is used to obtain the target information, and the detection accuracy is improved.

[0090] In some embodiments of the present application, in the step S102, the original high-resolution image z is reconstructed by using a degradation model of the super-resolution reconstruction unit, and the degradation model is represented as:

[0091] g k = τDH k M k z + n k , k = 1, …, K

[0092] where g k is the observed low-resolution image of the kth sub-aperture camera; τ represents an integral factor; D represents a down-sampling matrix; H k represents a block circulant matrix used to represent a blurring degradation process, at least including defocus, motion blur or blurring caused by an optical system transfer function; M k is a matrix representing the positional misalignment and geometric deformation of the kth image relative to the reference image, M k is a constant; z is the original high-resolution image; n k represents additive noise; and Q is the number of low-resolution images.

[0093] In some embodiments of the present application, in the step S102, the degradation model is solved by minimizing an objective function, and then:

[0094]

[0095] The formula (1) is solved by using a maximum a posteriori probability method, and then:

[0096]

[0097] where θ is a prior model parameter, and λ is a weight parameter;

[0098] P represents a probability, the formula (2) is solved by using a reconstruction method based on a Huber Markov random field model, and a high-resolution image containing a target is obtained.

[0099] The present application discloses a multi-aperture imaging detection device and method for a moving target in a low-illumination environment. The K sub-aperture cameras are arranged side by side, and a mode switching control unit is used to control the switching of the K sub-aperture cameras between a parallel light path mode and a splicing mode. When the splicing mode is adopted, the detection range can be expanded, and large field of view detection can be realized. When the parallel light path mode is adopted, high-resolution imaging can be realized through a synchronous exposure control unit. Therefore, the super-resolution technology is used to obtain a high-resolution image of the target, and the contradiction between large field of view and high resolution is solved. The resolution is improved while the large field of view is considered.

[0100] The multi-aperture high-resolution imaging mode can solve the problem that the traditional super-resolution technology causes the loss of moving targets while improving the quality by using multiple frames of continuous images, and improves the detection accuracy of the target.

[0101] The multi-aperture imaging mode is based on the image characteristics of the weak light detection chip, the image processing unit is configured with multiple frames of images exposed at the same time for quality improvement algorithm, the image definition and signal-to-noise ratio are improved, and the imaging can be performed in a low-illumination environment, thereby solving the problems of low signal-to-noise ratio and definition in weak light imaging.

[0102] The imaging mechanism of the multi-aperture structure solves the problem of motion blur caused by long integration time of a single aperture, the imaging detection device of the present application collects light beams emitted by remote objects through a plurality of small-aperture subsystems arranged in a certain form array, and obtains multiple images through one-time exposure, so as to observe fast-changing targets.

[0103] The above shows and describes the basic principles, main features and advantages of the present application. It should be understood by those skilled in the art that the present application is not limited to the above-mentioned embodiments, the above-mentioned embodiments and descriptions in the specification are only preferred examples of the present application, and are not intended to limit the present application, various changes and improvements can be made without departing from the spirit and scope of the present application, and these changes and improvements all fall within the scope of the present application. The scope of protection of the present application is defined by the appended claims and their equivalents.

Claims

1. A method for detecting moving targets in low-light environments using a multi-aperture imaging detection device, characterized in that, The multi-aperture imaging detection device includes: K A single-aperture camera, using a 1× K The sub-aperture cameras are arranged side-by-side, with overlapping regions between them to achieve sub-pixel-level misalignment. K It is an odd number; The imaging unit is equipped with a low-light detection chip for acquiring images; Synchronous exposure control unit, used to control K Simultaneous exposure of the sub-aperture cameras; The mode switching control unit is used to control the switching. K The imaging modes of the sub-aperture camera include a parallel optical path mode with high resolution and a stitching mode with a large field of view. Image processing unit for acquiring high-resolution images z and target information; wherein, the image processing unit includes an image stitching unit; the image stitching unit is used to stitch together images acquired by the sub-aperture in stitching mode; The multi-aperture imaging detection method includes: determining a detection mode based on the current illumination and acquiring target information, wherein the detection mode includes daytime detection and nighttime detection; The nighttime detection employs a parallel optical path mode and includes the following steps: Step S101, obtain from K Images captured simultaneously by various sub-aperture cameras within a single aperture camera system. K The first time-series remote sensing images are a series of images. Step S102, based on K Using the first time-series remote sensing image, moving targets are detected and target information is obtained; The daytime detection employs a parallel optical path mode and a stitching mode, and includes the following steps: Step S201: Employing a stitching mode to expand the detection range and achieve a wide field of view, multiple second time-series remote sensing images are acquired, among which... K Acquisition of aperture K × J The second time-series remote sensing image has a sub-aperture resolution of [missing information]. M × N , K It is an odd number. J Indicates different times; Step S202: Based on multiple second time-series remote sensing images, detect moving targets; wherein, using the image stitching unit, based on the sub-aperture camera position pair... K Aperture J The second time-series remote sensing images acquired at each time point are stitched together to obtain... J The second mosaic image has a resolution of [resolution value missing]. M ×( K × N -( K -1) t ), t The length of the overlapping region. j =1,2,…, J ; Step S203: When a moving target is detected, the mode switching control unit switches from splicing mode to parallel optical path mode, and then executes steps S101 and S102 in sequence; otherwise, no mode switching is performed.

2. The imaging detection method according to claim 1, characterized in that, The image processing unit also includes an image denoising and enhancement unit, a cloud detection and fusion unit, and a target detection unit; Step S202 further includes: Using the image denoising and enhancement unit to J The second stitched image undergoes image denoising and image enhancement processing. The cloud detection and fusion unit is used to perform cloud detection and fusion removal on the denoised and enhanced images. A method based on the U-Net network model is used to map the remote sensing image with clouds into a cloudless image and a cloud map. Robust principal component analysis is used to take advantage of the sparsity of clouds due to dynamic changes in the time series. Combined with the cloud detection results of the U-Net deep learning model, cloud removal is achieved to obtain the sequence of remote sensing images with cloud removal. based on J After removing clouds, the sequence of remote sensing images is used to perform feature analysis on the target and extract the target.

3. The imaging detection method according to claim 1, characterized in that, The image processing unit includes a super-resolution reconstruction unit, which reconstructs the original high-resolution image using a degradation model. In step S102, the original high-resolution image is reconstructed using the degradation model of the super-resolution reconstruction unit. z， The degradation model is expressed as: in, g k It is the first observed k Low-resolution images from a small aperture camera; Indicates the integrating factor; D Represents the downsampling matrix; H k A block cyclic matrix is ​​used to represent the blur degradation process, including at least defocusing, motion blur, or blur caused by the optical system transfer function. M k It means the first k The matrix representing the positional misalignment and geometric deformation of the image relative to the reference image. M k It is a constant; z It is the original high-resolution image; n k Indicates additive noise; K It represents the number of low-resolution images.

4. An imaging detection device for moving targets in low-light environments, used to implement the imaging detection method as described in any one of claims 1 to 3, characterized in that, include: The K Single-aperture camera; the imaging unit; The synchronous exposure control unit; the mode switching control unit; The image processing unit.

Images