Non-vision field imaging detection method and system of fusion event camera

Abstract

The invention discloses a non-vision field imaging detection method and system fused with an event camera, and the method comprises the following steps: enabling the event camera and a standard optical camera on a detection prototype to be aligned with a ground region of a corner, forming a video file based on a tiny pixel color difference amplification technology, and carrying out the calculation and analysis of the video file; processing the video file by using an image matching algorithm, and eliminating noise by using Kalman filtering; establishing a linear system, and then obtaining a reconstructed image by using a forward transmission light intensity function; performing Euler video amplification processing on the reconstructed image, wherein the Euler video amplification comprises motion estimation, spatial decomposition, time filtering and signal enhancement; detecting the detection area, and mapping the motion track of the target hidden behind the corner to a two-dimensional plane; the system comprises a simulation scene construction module, a three-dimensional point cloud set construction module and an information integration module. According to the invention, a micro motion algorithm and optical flow calculation are popularized to non-vision field imaging, so that the optical imaging technology can act on a vision field range and also can act on a non-vision field range.

Description

Technical Field

[0001] The present invention belongs to the technical field of non-line-of-sight imaging, and in particular relates to a non-line-of-sight imaging detection method and system for a fusion event camera. Background Art

[0002] In disaster relief and other events, targets are often obscured or out of the observer's sight (such as street corners). In this case, it is difficult to effectively observe these suspicious targets and activity venues using direct visual methods.

[0003] Traditional visual technology usually uses visible light cameras to collect images, taking advantage of multiple scattering of light between objects and ultrafast laser technology to indirectly detect information about hidden targets. However, the use of visible light cameras still has problems such as the inability to clearly capture high-speed moving objects and the large amount of data.

[0004] Event cameras are a new type of visual sensor that, unlike traditional cameras, do not capture images at a fixed frame rate, but instead capture information in the form of events by detecting brightness changes in a scene. Whenever the brightness in a scene changes, the event camera generates an information packet containing a timestamp, pixel location, and brightness change. This working principle enables event cameras to have high temporal resolution in fast-moving or changing scenes, enabling them to capture subtle dynamic details. In addition, since only changes are recorded instead of the entire image, event cameras usually have a small amount of data and fast processing speed, making them suitable for real-time applications such as robot vision, driverless cars, and dynamic environment monitoring. Event cameras have driven new developments in the field of computer vision by capturing visual information in an efficient way. Summary of the invention

[0005] The present invention aims to provide a non-line-of-sight imaging detection method and system that integrates an event camera, which can realize real-time display of the number, position and motion status of targets within the non-line-of-sight range behind a wall, thereby improving timeliness.

[0006] In order to achieve the purpose of the present invention, on the one hand, the present invention provides a non-line-of-sight imaging detection method of a fusion event camera, comprising the following steps:

[0007] Step S1: Aim the event camera and the standard optical camera on the detection prototype at the ground area at the corner, form a video file based on the tiny pixel color difference amplification technology, and perform calculation analysis on the video file;

[0008] Step S2: Processing the video file using an image matching algorithm and eliminating noise using a Kalman filter;

[0009] Step S3: Establish a linear system, and then use the forward transmission light intensity function to obtain a reconstructed image;

[0010] Step S4: performing Euler video upscaling processing on the reconstructed image, wherein the Euler video upscaling includes motion estimation, spatial decomposition, temporal filtering and signal enhancement.

[0011] On the other hand, the present invention provides a non-line-of-sight imaging detection system fused with an event camera, the system comprising the following modules:

[0012] Build a simulation scene module to set up the simulation scene, determine the location and characteristics of the light source and detector, and the space where the target object is located;

[0013] Construct a 3D point cloud collection module, which is used to set a 3D point cloud collection of a target object, select a point from the point cloud collection, trace the light passing through the point and calculate the information of the light reaching the detector, determine the specific propagation path of the light in a complex environment, and infer the position and shape of the hidden object;

[0014] The information integration module is used to integrate all light information and realize imaging and analysis of hidden objects through precise simulation and calculation.

[0015] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the program, steps of a non-line-of-sight imaging detection method of a fused event camera are implemented.

[0016] A non-transitory computer-readable storage medium stores computer instructions for causing the computer to execute steps of a non-line-of-sight imaging detection method for a fused event camera.

[0017] A computer program product includes computer program instructions. When the computer program instructions are executed on a computer, the computer is enabled to execute the steps of a non-line-of-sight imaging detection method of a fused event camera.

[0018] Compared with the prior art, the significant progress of the present invention is as follows: (1) The present invention introduces an event camera as a non-line-of-sight imaging acquisition device, uses Kalman filtering to eliminate video noise, and uses the Euler video amplification method to enhance the video signal, thereby calculating the trajectory information of the target hidden behind the corner, solving the problems of corner imaging, difficulty in clearly capturing high-speed moving objects, and difficulty in real-time data processing; (2) The present invention can use an intermediate reflective surface to achieve observation and imaging of hidden targets bypassing obstructing objects; (3) The present invention can quickly and effectively discover hidden objects and perform accurate three-dimensional reconstruction in disaster relief, which has important practical significance for various disaster relief.

[0019] In order to more clearly illustrate the functional characteristics and structural parameters of the present invention, further description is given below in conjunction with the accompanying drawings and specific implementation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0021] Figure 1 is a flow chart of the method of the present invention;

[0022] Figure 2 is a schematic diagram of corner imaging of the present invention;

[0023] Figure 3 It is a principle flow chart of the present invention;

[0024] Figure 4 It is a schematic diagram of program operation of the present invention;

[0025] Figure 5 It is a system composition diagram of the present invention. DETAILED DESCRIPTION

[0026] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0027] A non-line-of-sight imaging detection method integrating event cameras Figure 1 , including the following steps:

[0028] Step S1: Aim the event camera and the standard optical camera on the detection prototype at the ground area at the corner, form a video file based on the tiny pixel color difference amplification technology, and perform calculation analysis on the video file;

[0029] Step S2: Processing the video file using an image matching algorithm and eliminating noise using a Kalman filter;

[0030] Step S3: Establish a linear system, and then use the forward transmission light intensity function to obtain a reconstructed image;

[0031] Step S4: performing Euler video magnification processing on the reconstructed image, which can reveal details and information that may be ignored in the original video. This magnification is not a simple pixel-level magnification, but through signal enhancement technology, information hidden in noise or background becomes visible, that is, a processed enhanced image sequence is obtained; the Euler video magnification includes motion estimation, spatial decomposition, temporal filtering and signal enhancement, which can effectively reveal dynamic information in the video;

[0032] Step S5: Detect the detection area and map the movement trajectory of the target hidden behind the corner to a two-dimensional plane.

[0033] Furthermore, in step S2, the video is processed using an image matching algorithm to obtain a transformation matrix between the two images, and the required horizontal displacement, vertical displacement and rotation angle values ​​are extracted from the matrix to obtain a motion trajectory curve; the Kalman filter predicts the next motion through the previous motion, combines the predicted value and the actual value, and adjusts the weights of the two through the Kalman gain, so that the estimated structure is closer to the true value, thereby smoothing the burrs in the trajectory curve:

[0034] Prediction steps:

[0035] Prediction status:

[0036] Prediction covariance: P k|k-1 =AP k-1|k-1 A T +Q;

[0037] Update steps:

[0038] Kalman gain: K k =P k|k-1 H T (HP k|k-1 H T +R) -1 ;

[0039] Update status:

[0040] Update covariance: P k|k =(IK k H)P k|k-1 ;

[0041] in, is the predicted state vector; is the updated state vector; Px |k-1 is the predicted covariance matrix; P k|k is the updated covariance matrix; A is the state transfer matrix; B is the control input matrix; u k-1is the control vector; Q is the process noise covariance; H is the measurement matrix; R is the measurement noise covariance; z k is the measurement vector; K k is the Kalman gain.

[0042] Furthermore, the step S3 is specifically as follows: establishing a linear system between the target image and the light intensity of the diffuse reflection surface:

[0043] y=A(p0)f+b;

[0044] Among them, A(p0) is the light energy transfer matrix, f is the light intensity function in the original image, and y is the reconstructed image matrix.

[0045] Using the forward transmission light intensity function, we write a forward transmission model, obtain the forward transmission matrix A, input the target image, simulate the forward transmission process of light energy, and obtain a reconstructed image of the simulated target. However, the light intensity information of the target function is usually too weak, and we also need to reveal the time changes in the video that are difficult or impossible to see with the naked eye, and display them in an indicative way.

[0046] Further, the step S4 includes the following steps:

[0047] 4-1: Perform motion estimation on the video, estimate the motion between each frame, and obtain the motion M(t):

[0048] M(t)=(u(t),v(t));

[0049] where u(t) and v(t) are the horizontal and vertical motion components at time t;

[0050] 4-2: Use wavelet transform to decompose the video spatially. Use wavelet transform to decompose each frame of the video into low-frequency and high-frequency components:

[0051] I(t,x,y)=L(t,x,y)+H(t,x,y);

[0052] Where I(t, x, y) is the pixel value of the video sequence at time t and spatial coordinates (x, y), L(t, x, y) is the low-frequency component, and H(t, x, y) is the high-frequency component;

[0053] 4-3: Use time filtering to process the decomposed video, process the high-frequency components through the time filter F, and extract dynamic change information:

[0054] H filtered (t,x,y)=F(H(t,x,y));

[0055] 4-4: Using the signal obtained by time filtering, the video signal is enhanced by multiplying it by the amplification factor, combining the filtered high-frequency components with the low-frequency components, and introducing the amplification factor α:

[0056] I amplified (t,x,y)=L(t,x,y)+αH filtered (t, x, y);

[0057] 4-5: The final enhanced image sequence:

[0058] I final (t,x,y)=I(t,x,y)+I amplified (t, x, y).

[0059] On the other hand, the present invention also provides a non-line-of-sight imaging detection system fused with an event camera, the non-line-of-sight imaging detection system comprising the following modules:

[0060] Build a simulation scene module to set up the simulation scene, determine the location and characteristics of the light source and detector, and the space where the target object is located;

[0061] Construct a 3D point cloud collection module, which is used to set a 3D point cloud collection of a target object, select a point from the point cloud collection, trace the light passing through the point and calculate the information of the light reaching the detector, determine the specific propagation path of the light in a complex environment, and infer the position and shape of the hidden object;

[0062] The information integration module is used to integrate all light information and realize imaging and analysis of hidden objects through precise simulation and calculation.

[0063] The specific implementation method of each module of the above system is the same as the non-line-of-sight imaging detection method of the aforementioned fusion event camera, which will not be repeated here.

[0064] In general, the beneficial effects are prominently reflected in capturing changes that are difficult for the human eye to detect, integrating event cameras for image processing, and monitoring hidden targets.

[0065] Example

[0066] Example 1

[0067] like Figure 2 This is a schematic diagram of corner imaging. Specifically, taking the city as the scene, the camera is placed at the corner, the camera is pointed at the ground, and the system is started. The image information collected by the camera will be transmitted to the Rockchip RD-RK3588 development board, and the image information will be processed in real time through the program we developed. The inspector hides behind the corner where the target cannot be found, and uses the camera to detect the light signal on the ground to obtain image information. When the target in the non-visible area moves, the light signal detected on the ground will change, such as Figure 3 The image information is processed according to the flowchart. First, the simulation scene is set up to determine the light source, the position and characteristics of the detector, and the space where the target object is located. Then, a three-dimensional point cloud set of the target object is set up, and points are taken from the point cloud set. The light rays passing through the points are traced and the information of reaching the detector is calculated. The specific propagation path of the light rays in the complex environment is determined, and the position and shape of the hidden object are inferred. The process of judging whether the point cloud is empty and taking points to process the light rays is repeated. When the point cloud is not empty, repeated judgment is performed. When the point cloud set is empty, it is ensured that all points of the target object are fully processed. Finally, all the light information is integrated, and the imaging and analysis of the hidden object are realized through precise simulation and calculation. Alarms are raised in the program we developed, such as Figure 4 ,Specifically, the main interface of our program includes the functions of opening the camera, closing the camera, starting ,computation, stopping the computation, alarm, device selection, threshold setting, ,counting the number of targets, target orientation, moving status and ,visualization of target motion trajectory.

[0068] System components: Figure 5 The camera is fixed on the gimbal, which is fixed on the helmet. As a vibration isolation device, the gimbal can effectively reduce the interference of external vibration on the camera for the original video signal, ensuring that the camera remains stable during operation. The camera transmits the real-time video image to the development board via a data cable. The development board processes the received image with the above algorithm, visualizes the recognition result obtained by the processing, and transmits it to the display terminal for display, so that the user can intuitively obtain the result image collected by the non-viewing area prototype.

[0069] Example 2

[0070] In order to verify that the system of the present invention can adapt to complex outdoor conditions, a diffuse light was used indoors to simulate sunlight, and experimental data were collected in different brightness environments: sunny, cloudy, daytime and nighttime, when the detection target was moving at a speed of 0.5-2m / s at a distance of 3-7m.

[0071] The experimental conclusions are as follows: Optimal test range: According to the above experimental results and field test analysis, the stronger the light intensity, the closer the moving target distance, the faster the moving speed, the better the experimental results. That is, the best effect is achieved when the light intensity is greater than 20,000, the distance range is 3-5m, and the moving speed is 2m / s. In the actual laboratory environment test process, when the distance is fixed at 3-5m and the moving speed reaches 2m / s (fast moving speed), the light intensity is greater than 1000 to obtain a relatively stable test effect, and the effect gradually increases with the increase of light intensity. When the distance increases or the moving speed decreases, the light intensity needs to be increased to obtain a more stable effect. For example, when the distance is 5-7m and the moving speed reaches 0.5m / s (slow moving speed), the light intensity is recommended to be greater than 20,000.

[0072] Extreme test range: According to the actual laboratory environment test, when the light intensity is 100-1000, the moving target distance is 7m, and the moving speed is 0.5m / s (slow moving speed), the alarm can be triggered due to track movement. However, the moving track signal is poor at this time. It is recommended that when used in extreme environments, based on the above conditions, the light intensity should be greater than 1000 or the target distance should be around 5m or the moving speed should be 1.5m / s (normal moving speed).

[0073] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0074] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A non-line-of-sight imaging detection method for fusion event cameras, characterized in that: The following steps are involved: Step S1: Aim the event camera and the standard optical camera on the detection prototype at the ground area at the corner, form a video file based on the tiny pixel color difference amplification technology, and perform calculation analysis on the video file; Step S2: Processing the video file using an image matching algorithm and eliminating noise using a Kalman filter; Step S3: Establish a linear system, and then use the forward transmission light intensity function to obtain a reconstructed image; Step S4: performing Euler video magnification processing on the reconstructed image, wherein the Euler video magnification includes motion estimation, spatial decomposition, temporal filtering and signal enhancement; Step S5: Detect the detection area and map the movement trajectory of the target hidden behind the corner to a two-dimensional plane.

2. The non-line-of-sight imaging detection method of a fusion event camera according to claim 1, characterized in that: In step S2, the video is processed using an image matching algorithm to obtain a transformation matrix between the two images, and the required horizontal displacement, vertical displacement and rotation angle values ​​are extracted from the matrix to obtain a motion trajectory curve; the Kalman filter predicts the next movement through the previous movement, combines the predicted value and the actual value, and adjusts the weights of the two through the Kalman gain, so that the estimated structure is closer to the true value, thereby smoothing the burrs in the trajectory curve.

3. The non-line-of-sight imaging detection method of a fusion event camera according to claim 2, characterized in that: The burrs in the smoothing trajectory curve are specifically calculated by the following method: Prediction steps: Prediction status: Prediction covariance: P k|k-1 =AP k-1|k-1 A T +Q; Update steps: Kalman gain: K k =P k|k-1 H T (HP k|k-1 H T +R) -1 ; Update status: Update covariance: P k|k =(IK k H)P k|k-1 ; in, is the predicted state vector; is the updated state vector; P k|k-1 is the predicted covariance matrix; P k|k is the updated covariance matrix; A is the state transfer matrix; B is the control input matrix; u k-1 is the control vector; Q is the process noise covariance; H is the measurement matrix; R is the measurement noise covariance; z k is the measurement vector; K k is the Kalman gain.

4. The non-line-of-sight imaging detection method of a fusion event camera according to claim 1, characterized in that: The step S3 specifically includes: establishing a linear system between the target image and the light intensity of the diffuse reflection surface: y=A(p0)f+b; Among them, A(p0) is the light energy transfer matrix, f is the light intensity function in the original image, and y is the reconstructed image matrix. By using the forward transmission light intensity function, the forward transmission model is compiled, the forward transmission matrix A is obtained, the target image is input, the forward transmission process of light energy is simulated, and the reconstructed image of the simulation target is obtained.

5. The non-line-of-sight imaging detection method of a fusion event camera according to claim 1, characterized in that: The step S4 comprises the following steps: 4-1: Perform motion estimation on the video, estimate the motion between each frame, and obtain the motion M(t): M(t)=(u(t),v(t)); where u(t) and v(t) are the horizontal and vertical motion components at time t; 4-2: Use wavelet transform to decompose the video spatially. Use wavelet transform to decompose each frame of the video into low-frequency and high-frequency components: I(t,x,y)=L(t,x,y)+H(t,x,y); Where I(t, x, y) is the pixel value of the video sequence at time t and spatial coordinates (x, y), L(t, x, y) is the low-frequency component, and H(t, x, y) is the high-frequency component; 4-3: Use time filtering to process the decomposed video, process the high-frequency components through the time filter F, and extract dynamic change information: H filtered (t,x,y)=F(H(t,x,y)); 4-4: Using the signal obtained by time filtering, the video signal is enhanced by multiplying it by the amplification factor, combining the filtered high-frequency components with the low-frequency components, and introducing the amplification factor α: I amplified (t,x,y)=L(t,x,y)+αH filtered (t,x,y); 4-5: The final enhanced image sequence: I final (t,x,y)=I(t,x,y)+I amplified (t,x,y)。 6. A non-line-of-sight imaging detection system integrating an event camera, characterized in that: For implementing the non-line-of-sight imaging detection method of a fusion event camera according to any one of claims 1 to 5, the non-line-of-sight imaging detection system comprises the following modules: Build a simulation scene module to set up the simulation scene, determine the location and characteristics of the light source and detector, and the space where the target object is located; Construct a 3D point cloud collection module, which is used to set a 3D point cloud collection of a target object, select a point from the point cloud collection, trace the light passing through the point and calculate the information of the light reaching the detector, determine the specific propagation path of the light in a complex environment, and infer the position and shape of the hidden object; The information integration module is used to integrate all light information and realize imaging and analysis of hidden objects through precise simulation and calculation.

7. The non-line-of-sight imaging detection system of the fusion event camera according to claim 6, characterized in that: The module for constructing a three-dimensional point cloud set continuously repeats the process of determining whether the point cloud is empty and taking points to process light. When the point cloud is not empty, repeated determination is performed. When the point cloud set is empty, it ensures that all points of the target object are fully processed.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

9. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to cause the computer to execute any one of claims 1 to 7.

10. A computer program product, comprising computer program instructions, which, when executed on a computer, cause the computer to execute the method according to any one of claims 1 to 7.

Images