Opto-electronic imaging system and imaging method thereof

Abstract

The application discloses a photoelectric imaging system, which comprises an imaging system control unit, a gyro angle measurement unit, a photoelectric imaging unit and a signal processing unit. The control unit controls the synchronous and cooperative work of the gyro angle measurement unit and the photoelectric imaging unit. The gyro angle measurement unit outputs camera Euler angles to the signal processing unit. The photoelectric imaging unit outputs original images to the signal processing unit. The gyro angle measurement unit outputs the attitude information of the photoelectric system. The image signal real-time processing unit performs fusion and enhancement processing on continuous image frames. The signal processing unit outputs images. The application also discloses a photoelectric imaging method. The application adopts multiple exposures and integrates gyro angle measurement information to perform image information fusion in the time domain, thereby further improving the image quality on the basis of the traditional optical imaging mode.

Description

Technical Field

[0001] The present invention belongs to the technical field of photoelectric imaging, and in particular relates to a photoelectric imaging system and an image signal processing method thereof. Background Art

[0002] Optoelectronic imaging systems are affected by factors such as sensor noise and complex conditions in the imaging path, resulting in severe image degradation and thus affecting their range. Especially in applications involving small and dim targets, the target energy captured in a single frame of image information acquired by an optoelectronic system is extremely low. In existing traditional optical imaging systems, increasing the aperture or exposure time can only provide limited improvements in single-shot image quality within the constraints of the optoelectronic system design.

[0003] Single-frame image enhancement schemes are currently the most commonly used schemes in optoelectronic systems. Although simple to implement, such schemes often rely on prior knowledge. While emphasizing certain details in the image, they also enhance noise and erase details, making it difficult to effectively improve the image signal-to-noise ratio.

[0004] For existing imaging systems, it is not difficult to obtain continuous image frames. By reorganizing and fusing a large amount of redundant information in continuous image frames, the image signal-to-noise ratio can be more effectively improved and details can be enhanced. Summary of the Invention

[0005] In response to the above-mentioned deficiencies in the prior art, the present invention provides an optoelectronic real-time computational synthesis imaging system, which aims to combine the traditional optical imaging system with a gyroscope to generate a time-domain fusion image by real-time computational synthesis, thereby improving the signal-to-noise ratio of the existing optoelectronic system and thus increasing its effective range.

[0006] The technical solution adopted by the present invention to solve its technical problems is: a photoelectric imaging system, including an imaging system control unit and a gyroscope angle measurement unit and a photoelectric imaging unit respectively connected to the control unit, the gyroscope angle measurement unit and the photoelectric imaging unit are simultaneously connected to a signal processing unit, the control unit controls the synchronous and coordinated operation of the gyroscope angle measurement unit and the photoelectric imaging unit in terms of timing, the gyroscope angle measurement unit completes the angle deflection integration within the frame interval time, and outputs the camera Euler angle to the signal processing unit at the frame time interval, the photoelectric imaging unit collects continuous image frames at a set frame rate and outputs the original image to the signal processing unit, the gyroscope angle measurement unit and the photoelectric imaging unit synchronously output the posture information of the photoelectric system, the real-time image signal processing unit performs fusion enhancement processing on the continuous image frames, and the signal processing unit outputs the image.

[0007] The present invention also provides an imaging method of the above imaging system, comprising the following steps:

[0008] S1, transforms the image at time t1 to align with the pixels at time t2 through the coordinate transformation matrix between consecutive image frames, and calculates the projection transformation matrix;

[0009] S2, performs Gaussian pyramid and Laplacian pyramid processing on the images at time t1 and t2;

[0010] S3, using the camera Euler angle information provided by the gyroscope, uses the DIS optical flow algorithm to estimate the inter-frame motion of the images at time t1 and t2, and calculates the image coordinate transformation between the two times;

[0011] S4, fuse the images at time t1 and t2;

[0012] S5, image reconstruction using Laplacian pyramid:

[0013] S51, upsampling the Gaussian pyramid constructed in step S2, and then performing Gaussian convolution filtering;

[0014] S52, adding each layer image obtained in step S51 to the residual stored in the Laplacian pyramid in step S2;

[0015] S53, repeating steps S51 to S51 to obtain the image after Laplace reconstruction.

[0016] Furthermore, the step S1 is specifically as follows:

[0017] Step S11, taking the ship's bow direction as y w Axis, vertically upward is z w Axis, establish a three-dimensional rectangular coordinate system as the world coordinate system, any imaging unit vector It can uniquely correspond to a target point in the coordinate system, and the camera optical axis direction of the gyroscope angle measurement unit (2) Corresponding to the sensor center point O0, assuming that the imaging vector of any point P on the imaging target surface is The relationship between pixel coordinates and imaging vectors can be derived from imaging geometry:

[0018]

[0019] Where f is the focal length of the camera, Δu is the pixel spacing, Δv and the center point coordinates u0, v0 are determined by camera calibration;

[0020] Step S12: Establish a camera coordinate system at the optical center of the imaging system, with the camera optical axis in the y direction of the coordinate system. c The gyroscope installed on the imaging platform can sense the Euler angle of the imaging device. Taking the ZYX rule as an example, the rotation matrix of the optoelectronic system is:

[0021]

[0022] θ x ,θ y ,θ z Respectively around x c ,y c ,z c The three-axis rotation angle corresponds to the yaw angle, pitch angle and roll angle of the optoelectronic system. According to the law of rigid body motion, the imaging unit vector in the world coordinate system is Expressed as:

[0023]

[0024] For telephoto systems At two adjacent exposure times t1, t2, for the telephoto system

[0025] At two adjacent exposure times t1, t2, the camera posture is represented by the rotation matrix as Any two points with the same name in the two exposures are P1 and P2. Assume that the two points with the same name correspond to the same imaging vector. Then there is the following relationship

[0026]

[0027] The image coordinate transformation T(t1→t2) of two adjacent exposure times t1 and t2 can be expressed by the following projective transformation:

[0028]

[0029] Furthermore, the step S2 is specifically as follows:

[0030] S21, performing Gaussian blur processing on the original image;

[0031] S22, by formula G i =Down(G i-1 ) Downsample the blurred image to obtain the original image with the same length and width. Figure 1 / 2 compressed image;

[0032] S23, using the compressed image as the initial image, repeating steps S21 to S22 N times to complete the construction of an N-layer Gaussian pyramid;

[0033] S24, processing the Gaussian pyramid from the bottom layer, taking an upsampling process to obtain an image with the same size as the second-to-last layer of the Gaussian pyramid;

[0034] S25, using Gaussian convolution filtering to process the sampled image, and subtracting the filtered image from the image of the same size as the Gaussian pyramid to obtain a difference image As the i-th layer image of the Laplacian pyramid, repeat the above steps to complete the construction of the Laplacian pyramid.

[0035] Furthermore, the step S3 is specifically as follows:

[0036] S31, build the image Gaussian pyramid and start calculating the optical flow of each layer from top to bottom;

[0037] S32, each layer of the Gaussian pyramid is divided into blocks according to a certain block size and overlap ratio, each block is assigned an initial vector value, and the displacement vector of each block obtained by calculating the projection transformation matrix in step S1 is used as the initial value of the top layer of the Gaussian pyramid. The lower layers of the Gaussian pyramid are substituted with the calculation results of the upper layers of the Gaussian pyramid;

[0038] S33, use the LK reverse search algorithm to calculate the translation vector of each block and obtain the optical flow field on the sparse grid;

[0039] S34, computes dense optical flow from optical flow on a sparse grid using a densification algorithm;

[0040] S35, if the current Gaussian pyramid level is the bottom level, output the result; otherwise, multiply the optical flow calculation result by 2, stretch it to the same resolution as the next lower level, and then pass it to the next lower level of the Gaussian pyramid as the initial value, and jump to step S32.

[0041] Furthermore, the step S4 is specifically as follows:

[0042] Cache a frame of fused image, register and fuse the newly acquired image, and then generate a new fused image; the previous frame and the current frame Laplacian pyramid level l are respectively recorded as and The vector field obtained by DIS optical flow calculation is A, A l It is a vector field with the size of the l-th layer pyramid, which aligns the Laplacian pyramid of the previous frame fusion image to the registration pyramid after the current frame Fused Laplacian Pyramid Expressed as in and They are the interpolation boundaries of the current frame and the previous frame, and the parameter settings To obtain a sharpening effect and cover up the blur caused by slight pixel misalignment during accumulation, It is an interpolation function. The design of this function takes into account various spatiotemporal effects in order to improve the signal-to-noise ratio of the fusion result and reduce artifacts. For this purpose, the design considers the expected value of sensor noise and the image difference between the two pyramids. And considering the registration error To avoid introducing artifacts in the fusion result when no good match is found, the interpolation function is synthesized from the two in is the registration error, Using the sigmoid function: Parameter m is the noise midpoint estimate, parameter C e Used to adjust the balance between image clarity, signal-to-noise ratio and fusion artifacts.

[0043] The beneficial effects of the present invention are:

[0044] 1. The present invention adopts short-time exposure to suppress the energy dispersion during the imaging integration time caused by the motion of the carrier. The short-time exposure continuously acquired images are unified by inertial coordinates and then processed (weighted) to improve the signal-to-noise ratio of the target observation.

[0045] 2. The method of the present invention can reduce the energy dispersion in different pixels caused by the motion of the carrier on the one hand, and on the other hand, reduce the noise in the non-target area that follows the Gaussian distribution.

[0046] 3. In the cooperation with synchronous clocks, the present invention utilizes gyroscope information to perform grayscale reconstruction and optimal weighting on multiple frames of images, thereby improving the signal-to-noise ratio of the target image.

[0047] 4. The real-time pipeline processing flow in the present invention can be deployed in an embedded SoC. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 is a block diagram of the imaging system of the present invention;

[0049] Figure 2 It is a simplified sea scene;

[0050] Figure 3 This is a flow chart of pipeline image sequence fusion of the present invention.

[0051] The reference numerals are: 1—control unit, 2—gyroscope angle measurement unit, 3—photoelectric imaging unit, 4—signal processing unit. DETAILED DESCRIPTION

[0052] The present invention will be described in further detail below with reference to the accompanying drawings.

[0053] Reference Figure 1As shown, the present invention discloses an optoelectronic imaging system, including an imaging system control unit 1 and a gyroscope angle measurement unit 2 and an optoelectronic imaging unit 3 respectively connected to the control unit 1. The gyroscope angle measurement unit 2 and the optoelectronic imaging unit 3 are simultaneously connected to a signal processing unit 4. The control unit 1 controls the synchronous and coordinated operation of the gyroscope angle measurement unit 2 and the optoelectronic imaging unit 3 in terms of timing. The gyroscope angle measurement unit 2 completes the angle deflection integration within the frame interval time and outputs the camera Euler angle to the signal processing unit 4 at the frame time interval. The optoelectronic imaging unit 3 captures continuous image frames at a set frame rate and outputs the original image to the signal processing unit 4. The gyroscope angle measurement unit 2 and the optoelectronic imaging unit 3 synchronously output the attitude information of the optoelectronic system. The real-time image signal processing unit performs fusion enhancement processing on the continuous image frames, and the signal processing unit 4 outputs the image.

[0054] The detailed processing flow of the image signal of the synthetic imaging system of the present invention is as follows: Figure 3 As shown, the detailed implementation of each part is as follows.

[0055] S1, a method for calculating the coordinate transformation matrix between consecutive image frames.

[0056] The coordinate transformation of two consecutive frames of images in the image coordinate system is calculated using the angle information provided by the gyroscope. Assuming that the imaging target is very far away and there is almost no movement of the target in the two frames of imaging, under this assumption, the target can be considered to be on the infinitely distant celestial sphere. The sea scene is simplified as follows: Figure 2 shown.

[0057] The ship's bow direction is y w Axis, vertically upward is z w Axis, establish a three-dimensional rectangular coordinate system, that is, the world coordinate system. However, since the target is considered to be very far away and the influence of distance is ignored, in this solution the three-dimensional rectangular coordinate system is simplified to a horizontal coordinate system, such as Figure 2 As shown, for any imaging unit vector It can uniquely correspond to a target point in the horizontal coordinate system.

[0058] is the optical axis direction of the camera, which corresponds to the center point O0 of the sensor. Assume that the imaging vector of any point P on the imaging target surface is The relationship between pixel coordinates and imaging vectors can be derived from imaging geometry:

[0059]

[0060] Wherein, the camera focal length f, pixel spacing Δu, Δv and center point coordinates u0, v0 are determined by camera calibration.

[0061] The camera coordinate system is established at the optical center of the imaging system, and the camera optical axis is in the y c The gyroscope mounted on the imaging platform can sense the Euler angle of the imaging device. Taking the ZYX rule as an example, the rotation matrix of the optoelectronic system is:

[0062]

[0063] θ x ,θ y ,θ z Respectively around x c ,y c ,z c The three-axis rotation angle corresponds to the yaw angle, pitch angle and roll angle of the optoelectronic system. According to the law of rigid body motion, the imaging unit vector in the world coordinate system is Expressed as:

[0064]

[0065] For telephoto systems In addition, in this model, the translation vector of the imaging system can be ignored for infinitely far targets. Therefore, at two adjacent exposure times t1 and t2, the camera posture is represented by the rotation matrix as follows: Any two points with the same name in the two exposures are P1 and P2. Assume that the two points with the same name correspond to the same imaging vector. Therefore, the following relationship exists:

[0066]

[0067] The image coordinate transformation T(t1→t2) of two adjacent exposure times t1 and t2 can be expressed by the following projective transformation:

[0068]

[0069] The above formula can be used to transform the image at time t1 to align the pixels with those at time t2.

[0070] S2, establish Gaussian pyramid and Laplacian pyramid, the process is as follows.

[0071] S21, performs Gaussian blur on the original image.

[0072] S22, downsample the blurred image to get the original length and width Figure 1 / 2 compressed image, the calculation process is expressed as G i =Down(G i-1 ).

[0073] S23, using the compressed image as the initial image, repeating steps S21 and S22 N times to complete the construction of an N-layer Gaussian pyramid.

[0074] S24, processing the Gaussian pyramid from the bottom layer, taking an upsampling process to obtain an image with the same size as the penultimate layer of the Gaussian pyramid.

[0075] S25, the sampled image is processed using Gaussian convolution filtering, and the difference image is obtained by subtracting the filtered image from the image of the same size as the Gaussian pyramid. The calculation process is expressed as This difference image is the i-th layer image of the Laplacian pyramid. Repeat the above steps to complete the construction of the Laplacian pyramid.

[0076] S3, uses the DIS optical flow algorithm to perform inter-frame motion estimation.

[0077] The calculation above shows how to use the Euler angles provided by the gyroscope to calculate the image coordinate transformation between two moments. However, this analysis assumes the target is infinitely distant and static, which can lead to some registration errors in real-world applications. Therefore, this solution proposes to add optical flow calculations between consecutive image frames based on the above projection transformation. The optical flow algorithm uses the DIS optical flow method. For real-time considerations, the algorithm has been simplified, retaining only the reverse LK search and densification steps. The specific algorithm flow is as follows.

[0078] S31, build the image Gaussian pyramid and start calculating the optical flow of each layer from top to bottom.

[0079] In step S32, each layer of the image pyramid is divided into blocks according to a certain block size and overlap rate. Each block is assigned an initial vector value. The displacement vector of each block obtained by calculating the projection transformation matrix in step S1 is used as the initial value of the top pyramid, while the lower pyramids are substituted with the calculation results of the upper pyramid.

[0080] S33, use the LK reverse search algorithm to calculate the translation vector of each block and obtain the optical flow field on the sparse grid.

[0081] S34, uses a densification algorithm to calculate the dense optical flow field from the optical flow field on the sparse grid.

[0082] S35, if the current pyramid level is the bottom level, output the result; otherwise, multiply the optical flow calculation result by 2, stretch it to the same resolution as the next level, and pass it to the next level pyramid as the initial value, and jump to step 2.

[0083] S4, image fusion.

[0084] In order to achieve real-time processing, the present invention adopts recursive fusion: a fused image is cached, and the newly acquired image is registered and fused with the fused image to generate a new fused image. The first layer of the Laplacian pyramid of the previous frame and the current frame is respectively denoted as and The vector field obtained by DIS optical flow calculation is A, A l It is a vector field with the size of the l-th layer pyramid, which aligns the Laplacian pyramid of the previous frame fusion image to the registration pyramid after the current frame The final fused Laplacian pyramid Expressed as:

[0085]

[0086] and They are the interpolation boundaries of the current frame and the previous frame, and the parameter settings This is to achieve a sharpening effect and cover up the blur caused by slight pixel misalignment during accumulation. is an interpolation function that takes into account various spatiotemporal effects in order to improve the signal-to-noise ratio of the fusion result and reduce artifacts. To this end, the design considers the expected value of sensor noise and the image difference between the two pyramids. And considering the registration error This is to avoid introducing artifacts in the fusion result when no good match is found. The interpolation function is a combination of the two in is the registration error, Using the sigmoid function Parameter m is the noise midpoint estimate, parameter C e Used to adjust the balance between image clarity, signal-to-noise ratio and fusion artifacts.

[0087] S5, reconstructs the image using the Laplacian pyramid. The specific process is as follows.

[0088] S51 , upsampling the Gaussian pyramid constructed in step S2 , and then performing Gaussian convolution filtering.

[0089] S52 , adding each layer image obtained in step S51 to the residual stored in the Laplacian pyramid in step S2 .

[0090] S53, repeat the above steps to obtain the image after Laplace reconstruction.

[0091] The above embodiments are merely illustrative of the principles and effects of the present invention, as well as some embodiments of its application. A person skilled in the art may make several modifications and improvements without departing from the inventive concept of the present invention, and all of these modifications and improvements fall within the scope of protection of the present invention.

Claims

1. An imaging method using a photoelectric imaging system, characterized in that: The photoelectric imaging system comprises a control unit (1) and a photoelectric imaging system comprising a gyroscope angle measurement unit (2) and a photoelectric imaging unit (3) respectively connected to the control unit (1), wherein the gyroscope angle measurement unit (2) and the photoelectric imaging unit (3) are simultaneously connected to a signal processing unit (4), the control unit (1) controls the synchronous coordinated operation of the gyroscope angle measurement unit (2) and the photoelectric imaging unit (3) in terms of timing, the gyroscope angle measurement unit (2) completes the angle deflection integration within the frame interval time, and outputs the camera Euler angle to the signal processing unit (4) at the frame time interval, the photoelectric imaging unit (3) collects continuous image frames at a set frame rate and outputs the original image to the signal processing unit (4), the gyroscope angle measurement unit (2) and the photoelectric imaging unit (3) synchronously output the posture information of the photoelectric system, the signal processing unit (4) performs the fusion enhancement processing of the continuous image frames, and the signal processing unit (4) outputs the image; the system comprises the following steps: S1, through the coordinate transformation matrix between consecutive image frames, The image at this moment is transformed to Align pixels at all times and calculate the projection transformation matrix; Step S1 specifically includes: Step S11, taking the ship's bow direction as y w Axis, vertically upward z w Axis, establish a three-dimensional rectangular coordinate system as the world coordinate system, any imaging unit vector Uniquely corresponds to a target point in the coordinate system, the camera optical axis of the gyroscope angle measurement unit (2) Corresponding to the center point of the sensor , from any point on the imaging target surface P Imaging vector To derive the relationship between pixel coordinates and imaging vectors: ， Where, is the camera focal length, is the pixel spacing, And the center point coordinates Determined by camera calibration; Step S12: Establish a camera coordinate system at the optical center of the imaging system. The camera optical axis is in the coordinate system. y c On the axis, the rotation matrix of the optoelectronic system is: ， , , Around three axes The rotation angle corresponds to the yaw angle, pitch angle and flip angle of the optoelectronic system, and the imaging unit in the world coordinate system The vector is represented as: For telephoto systems , at two adjacent exposure times , the camera posture is expressed by the rotation matrix as , any two points with the same name in the two exposures are , two points of the same name correspond to the same imaging vector Then there is the following relationship: ， Two adjacent exposure times Image coordinate transformation It is represented by the following projection transformation: ； S2, yes 、 The image at each moment is processed with Gaussian pyramid and Laplacian pyramid; S21, performing Gaussian blur processing on the original image; S22, by formula Downsample the blurred image to obtain a compressed image with a length and width of 1 / 2 of the original image; S23, using the compressed image as the initial image, repeating steps S21 to S22 N times to complete the construction of an N-layer Gaussian pyramid; S24, processing the Gaussian pyramid from the bottom layer, taking an upsampling process to obtain an image with the same size as the second-to-last layer of the Gaussian pyramid; S25, using Gaussian convolution filtering to process the sampled image, and subtracting the filtered image from the image of the same size as the Gaussian pyramid to obtain a difference image , as the i-th layer image of the Laplacian pyramid, repeat the above steps to complete the construction of the Laplacian pyramid; S3, using DIS optical flow algorithm 、 The image at the moment is subjected to inter-frame motion estimation, and the image coordinate transformation between the two moments is calculated; S31, build the image Gaussian pyramid and start calculating the optical flow of each layer from top to bottom; S32, each layer of the Gaussian pyramid image is divided into blocks according to size and overlap rate, each block is assigned an initial vector value, and the displacement vector of each block obtained by the projection transformation matrix is ​​used as the initial value of the top layer of the Gaussian pyramid. The lower layers of the Gaussian pyramid are substituted into the calculation results of the upper layers of the Gaussian pyramid; S33, use the LK reverse search algorithm to calculate the translation vector of each block and obtain the optical flow field on the sparse grid; S34, computes dense optical flow from optical flow on a sparse grid using a densification algorithm; S35: If the current Gaussian pyramid level is the bottom level, output the result; otherwise, multiply the optical flow calculation result by 2, stretch it to the same resolution as the next level, and then pass it to the next level of the Gaussian pyramid as the initial value, and jump to step S32; S4, yes 、 The image at each moment is fused: a fused image is cached, and the newly acquired image is fused with the fused image to generate a new fused image; the previous frame and the current frame Laplacian pyramid are combined. The layers are denoted as and , the vector field obtained by DIS optical flow calculation is , The size is l The vector field of the layer pyramid size is used to register the Laplacian pyramid of the previous frame fusion image to the registration pyramid after the current frame. , the fused Laplacian pyramid Expressed as ,in and They are the interpolation boundaries of the current frame and the previous frame, and the parameter settings , is the interpolation function, the expected value of the sensor noise and the image difference between the two pyramids are , the registration error is , interpolation function ,in , is the registration error, Using the sigmoid function: ,parameter is the noise midpoint estimate, parameter Used to adjust the balance between image clarity, signal-to-noise ratio and fusion artifacts; S5, reconstructing the image using a Laplacian pyramid; wherein step S5 specifically includes: S51, upsampling the Gaussian pyramid constructed in step S2, and then performing Gaussian convolution filtering; S52, adding each layer of the obtained image to the residual stored in the Laplacian pyramid; S53, repeating steps S51 to S52 to obtain the image after Laplace reconstruction.

Images