High-speed target position prediction method based on event camera and high-speed camera

Abstract

The application provides a high-speed target position prediction method based on an event camera and a high-speed camera, and solves the problem that the speed and position of a high-speed moving target cannot be calculated in real time, so that the target position cannot be predicted in real time. The method comprises the following steps: arranging two camera groups and performing time synchronization; calibrating the camera groups to obtain the internal parameters and external parameters of the event cameras and the high-speed cameras in each camera group; collecting an event stream by the event cameras, and simultaneously collecting a target motion image sequence by the high-speed cameras; coarsely positioning the target to obtain coarse image coordinates and time stamps of the target in each event camera; searching for an image frame corresponding to the time stamp in the motion image sequence collected by the high-speed camera in the same camera group as each event camera, finely positioning the target to obtain accurate image coordinates of the target in each high-speed camera; tracking the target to obtain accurate image coordinates of the target in each high-speed camera changing with time; and calculating key parameters of the target at different time points to complete the prediction of the position of the high-speed target.

Description

Technical Field

[0001] This invention relates to the prediction of target location, and more specifically to a high-speed target location prediction method based on event cameras and high-speed cameras. Background Technology

[0002] In the fields of transportation, security, and military, predicting the location of high-speed moving targets is crucial for enabling security personnel to respond quickly and effectively to emergencies. Currently, single or multiple high-speed cameras are typically used to predict the location of high-speed moving targets. However, high-speed cameras suffer from problems such as large data volumes and slow data transmission, making it impossible to calculate key parameters such as the speed and position of high-speed moving targets in real time, thus hindering real-time location prediction. Summary of the Invention

[0003] The purpose of this invention is to solve the technical problem that high-speed cameras, due to their large data volume and slow data transmission, cannot perform real-time calculation of key parameters such as the speed and position of high-speed moving targets, thus making it impossible to predict the position of high-speed moving targets in real time. The invention provides a high-speed target position prediction method based on event cameras and high-speed cameras.

[0004] To achieve the above objectives, the present invention adopts the following technical solution:

[0005] A high-speed target position prediction method based on event cameras and high-speed cameras, characterized by the following steps:

[0006] Step 1: Arrange two camera groups facing the target observation area using orthogonal perspectives; the camera groups include an event camera and a high-speed camera, and use satellite time synchronization technology to synchronize the two event cameras and the two high-speed cameras.

[0007] Step 2: Calibrate the event camera and high-speed camera to obtain the intrinsic and extrinsic parameters of the event camera and high-speed camera in each camera group;

[0008] Step 3: Use each event camera to collect event streams triggered by high-speed targets, and simultaneously use each high-speed camera to acquire motion image sequences of high-speed targets; use the event streams collected by each event camera to perform coarse localization of the target, and obtain the approximate image coordinates and timestamps of the target in each event camera respectively;

[0009] Step 4: Using the approximate image coordinates and timestamps of the target in each event camera, and the intrinsic and extrinsic parameters of the event camera and high-speed camera in each camera group, find the image frame with the corresponding timestamp in the motion image sequence acquired by the high-speed camera in the same camera group as each event camera, perform fine positioning of the target, and obtain the precise image coordinates of the target in each high-speed camera.

[0010] Step 5: Using the precise image coordinates of the target in each high-speed camera and the motion image sequence acquired by each high-speed camera, target tracking is performed based on the Kalman filter algorithm to obtain the precise image coordinates of the target in each high-speed camera as time changes.

[0011] Step 6: Using the precise image coordinates of the target changing over time in each high-speed camera, as well as the intrinsic and extrinsic parameters of each high-speed camera, calculate the key parameters of the target at different times to complete the prediction of the high-speed target's position.

[0012] Furthermore, in step 2, the event camera and the high-speed camera are calibrated using a binocular camera calibration method.

[0013] Furthermore, step 2 specifically includes the following steps:

[0014] Step 2.1: Hover the UAV at different locations within the target observation area and record the UAV's GPS coordinates. Simultaneously, each event camera and each high-speed camera take pictures, and extract the image coordinates corresponding to each hovering from each event camera and each high-speed camera.

[0015] Step 2.2: Convert the GPS coordinates of the UAV obtained in Step 2.1 to a preset coordinate system to obtain the three-dimensional coordinates of the UAV in the preset coordinate system;

[0016] Step 2.3: Combining the image coordinates obtained in Step 2.1 and the 3D coordinates of the UAV in the preset coordinate system obtained in Step 2.2, the intrinsic and extrinsic parameters of the event camera and high-speed camera in each camera group are calculated using the direct linear transformation method. The intrinsic parameters include the equivalent focal length and the principal point of the image; the extrinsic parameters include the rotation matrix and the translation vector. The intrinsic and extrinsic parameters of the event camera are calculated according to the following formula:

[0017]

[0018] Among them, f x u0, f y v0, R, and T represent the equivalent focal length in the x-direction, principal point in the x-direction, equivalent focal length in the y-direction, principal point in the y-direction, rotation matrix, and translation vector of the event camera, respectively. k Let M be the image coordinates of the drone during its k-th hover in the camera at this event. k Let be the three-dimensional coordinates of the UAV in the preset coordinate system during the k-th hovering. For M k The projection point in the camera image of this event, K is the number of times the drone hovers;

[0019] The calculation methods for the intrinsic and extrinsic parameters of the high-speed camera are the same as those for the event camera.

[0020] Furthermore, step 6 specifically includes the following steps:

[0021] Step 6.1: Using the precise image coordinates of the target in each high-speed camera over time, and the intrinsic and extrinsic parameters of each high-speed camera, simultaneously solve the following equations to calculate t. i The three-dimensional coordinates P of the target in the preset coordinate system i (X i ,Y i Z i ):

[0022]

[0023]

[0024] Among them, f xl u 0l f yl v 0l R l T l Let x be the equivalent focal length in the x-direction, the principal point of the image in the x-direction, the equivalent focal length in the y-direction, the principal point of the image in the y-direction, the rotation matrix, and the translation vector of one of the high-speed cameras. l ,y l ) for t i The image coordinates of the target in the high-speed camera, Z Cl For P i The projection of the distance to the optical center of the high-speed camera along the optical axis; f xr u 0r f yr v 0r R r T r The equivalent focal length in the x-direction of another high-speed camera, the principal point of the image in the x-direction, the rotation matrix, and the translation vector are respectively (x r ,y r ) for t i The image coordinates of the target in another high-speed camera, Z Cr For P i The projection of the distance to the optical center of another high-speed camera along the optical axis;

[0025] Step 6.2: Based on the target's three-dimensional coordinates changing over time in a preset coordinate system, calculate the target's key parameters at different times to predict the high-speed target's position; the key parameters include the target's velocity and target position; calculate t according to the following formula. i The target velocity v at that time i :

[0026]

[0027] Among them, v Xi =(X i+1 -X i-1 ) / (t i+1 -t i-1 ), v Yi =(Y i+1 -Y i-1 ) / (t i+1 -t i-1 ), v Zi =(Z i+1 -Z i-1 ) / (t i+1 -t i-1 ), v Xi ,v Yi ,v Zi t i The velocity components of the target in the X, Y, and Z directions, (X i+1 ,Y i+1 Z i+1 ) for t i+1 The three-dimensional coordinates of the target in the preset coordinate system, (X) i-1 ,Y i-1 Z i-1 ) for t i-1 The three-dimensional coordinates of the target in the preset coordinate system;

[0028] The target position at any time t can be calculated using the following formula:

[0029] X t =X i +v Xi ·(tt i )

[0030] Y t =Y i +v Yi ·(tt i )

[0031] Z t =Z i +v Zi ·(tt i )

[0032] Among them, (X) t ,Y t Z t ) represents the three-dimensional coordinates of the target in the preset coordinate system at time t.

[0033] Furthermore, in step 4, a target detection algorithm is used to precisely locate the target.

[0034] Furthermore, in step 4, the target detection algorithm adopts the YOLOx series of algorithms;

[0035] In step 5, the Kalman filtering algorithm uses linear Kalman filtering.

[0036] The beneficial effects of this invention are:

[0037] This invention provides a high-speed target position prediction method based on an event camera and a high-speed camera. The event camera used in this method is a novel visual sensing mode inspired by the biological retina. The event camera, relative to a single pixel, only outputs event information when the received light intensity changes. This invention combines the advantages of event cameras (low latency, high dynamic range, low power consumption, low data bandwidth) with the high image resolution and lack of motion blur of high-speed cameras, overcoming the problems of large data volume and slow data transmission. It can calculate the velocity and position of high-speed moving targets in real time, thereby predicting the position of high-speed moving targets in real time. Because this invention can predict the target position in real time, it avoids the risk of target tracking failure when the high-speed moving target is occluded. Attached Figure Description

[0038] Figure 1 This is a flowchart of the high-speed target position prediction method based on event camera and high-speed camera of the present invention;

[0039] Figure 2 This is a layout diagram of two camera groups in an embodiment of the present invention;

[0040] Figure 3 This is a schematic diagram of the binocular intersection principle of two high-speed cameras in an embodiment of the present invention;

[0041] Figure 4 This is the preset coordinate system in the embodiments of the present invention.

[0042] Explanation of reference numerals in the attached figures:

[0043] 1-Event camera, 2-High-speed camera, 3-Observation area. Detailed Implementation

[0044] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0045] like Figure 1 As shown, a high-speed target position prediction method based on event cameras and high-speed cameras includes the following steps:

[0046] Step 1, as follows Figure 2As shown, two camera groups facing the target observation area 3 are arranged with orthogonal perspectives; each camera group includes an event camera 1 and a high-speed camera 2, and the two event cameras 1 and the two high-speed cameras 2 are synchronized in time using satellite time synchronization technology.

[0047] Step 2: The two event cameras 1 and two high-speed cameras 2 are calibrated using the binocular camera calibration method to obtain the intrinsic and extrinsic parameters of event camera 1 and high-speed camera 2 in each camera group; for example... Figure 3 The diagram shown illustrates the binocular intersection principle of two high-speed cameras (2), where P is the target point to be observed, and O... CL -X CL ,Y CL Z CL O L -x L ,y L These are the camera coordinate system and image physical coordinate system of one of the high-speed cameras, O. CR -X CR ,Y CR Z CR O R -x R ,y R These are the camera coordinate system and image physical coordinate system of another high-speed camera 2, respectively.

[0048] The function required for binocular camera calibration is to establish the world coordinate system on top of a preset coordinate system, which specifically includes the following steps:

[0049] Step 2.1: Hover the UAV at different locations within the target observation area 3 and record the UAV's GPS coordinates. Simultaneously, each event camera 1 and each high-speed camera 2 take pictures, and extract the image coordinates corresponding to each hovering from each event camera 1 and each high-speed camera 2. Figure 4 This is the preset coordinate system in this embodiment, where the four coordinate points A, B, C, and D are taken from the four corner points of the preset rectangular area, and the origin O is the coordinate system. w The center point of the rectangle is O, and the X-axis coordinate direction is the origin O. w The direction of the line connecting to edge CD is such that the Y-axis coordinate direction is the origin O. w The direction of the line connecting to edge BC is in the Z-axis coordinate direction. The direction of the vector.

[0050] Step 2.2: Transform the UAV's GPS coordinates obtained in Step 2.1 to both the geodetic coordinate system and the preset coordinate system; assuming the UAV's GPS coordinates are P... G (B,L,H), with three-dimensional coordinates P in the geodetic coordinate system. E (X E ,Y E ZE The coordinate transformation needs to be performed using the following relationship:

[0051]

[0052] Where N is the radius of curvature of the Earth's ellipsoid, and E is the first eccentricity of the Earth's ellipsoid.

[0053] Assuming a and b are the lengths of the Earth's semi-major and semi-minor axes, respectively, and χ is the Earth's ellipsoidal flattening rate, then the parameter values ​​are as follows:

[0054]

[0055] The following parameters can then be obtained:

[0056]

[0057] Then, convert the three-dimensional coordinates P in the geodetic coordinate system E (X E ,Y E Z E By translating and rotating to a preset coordinate system, the three-dimensional coordinates P of the UAV in the preset coordinate system are obtained. W (X W ,Y W Z W ).

[0058] Step 2.3: Combining the image coordinates obtained in Step 2.1 and the 3D coordinates of the UAV in the preset coordinate system obtained in Step 2.2, the intrinsic and extrinsic parameters of event camera 1 and high-speed camera 2 in each camera group are calculated using the direct linear transformation method. The intrinsic parameters include the equivalent focal length and the principal point of the image, and the extrinsic parameters include the rotation matrix and the translation vector. The intrinsic and extrinsic parameters of event camera 1 are calculated according to the following formula:

[0059]

[0060] Among them, f x u0, f y v0, R, and T represent the equivalent focal length in the x-direction, the principal point of the image in the x-direction, the equivalent focal length in the y-direction, the principal point of the image in the y-direction, the rotation matrix, and the translation vector of event camera 1, respectively. k Let M be the image coordinates of the UAV in camera 1 during the kth hovering event. k Let be the three-dimensional coordinates of the UAV in the preset coordinate system during the k-th hovering. For M k The projection point in the image of camera 1 during this event, K is the number of times the drone hovers;

[0061] The calculation methods for the intrinsic and extrinsic parameters of high-speed camera 2 are the same as those for event camera 1.

[0062] Step 3: Use each event camera 1 to collect event streams triggered by high-speed targets, and use each high-speed camera 2 to acquire motion image sequences of high-speed targets; use the event streams collected by each event camera 1 to perform coarse localization of the targets, and obtain the approximate image coordinates and timestamps of the targets in each event camera 1.

[0063] When a high-speed target appears in each event camera 1 and each high-speed camera 2, each event camera 1 will trigger a series of event streams. A specific event includes (t, x, y, p), where t is the timestamp, x is the horizontal coordinate of the pixel where the event is located, y is the vertical coordinate of the pixel where the event is located, and p is the polarity of the event, which indicates the change in the light intensity of the event, and p∈{-1,+1}.

[0064] A certain number of event pixels are combined and arranged according to their position information to form a new event image. Preliminary target detection is performed on these event images to obtain the rough image coordinates of the target.

[0065] Step 4: Using the approximate image coordinates and timestamps of the target in each event camera 1, and the intrinsic and extrinsic parameters of event camera 1 and high-speed camera 2 in each camera group, find the image frame with the corresponding timestamp in the motion image sequence acquired by the high-speed camera 2 which is in the same camera group as each event camera 1. Then, use a target detection algorithm to perform precise target localization, obtaining the accurate image coordinates of the target in each high-speed camera 2. In this embodiment, the target detection algorithm uses the YOLOx series algorithm.

[0066] Step 5: Using the precise image coordinates of the target in each high-speed camera 2 and the motion image sequences acquired by each high-speed camera 2, target tracking is performed based on the Kalman filter algorithm to obtain the precise image coordinates of the target changing over time in each high-speed camera 2. In this embodiment, the Kalman filter algorithm used is linear Kalman filtering.

[0067] Kalman filtering dynamically estimates the state at the next time step in an iterative manner, while simultaneously considering historical states and observation data. For target tracking, Kalman filtering models the target's motion and can be used to estimate the target's position in the next frame. Kalman filtering itself is an estimator used for state estimation; its core is information fusion, and this estimator is dynamically changing, continuously updating the model.

[0068] Step 6: Using the precise image coordinates of the target in each high-speed camera 2 over time, and the intrinsic and extrinsic parameters of each high-speed camera 2, calculate the key parameters of the target at different times to complete the prediction of the high-speed target's position; specifically including the following steps:

[0069] Step 6.1: Using the precise image coordinates of the target in each high-speed camera 2 as a function of time, and the intrinsic and extrinsic parameters of each high-speed camera 2, simultaneously solve the following equations to calculate t. i The three-dimensional coordinates P of the target in the preset coordinate system i (X i ,Y i Z i ):

[0070]

[0071]

[0072] Among them, f xl u 0l f yl v 0l R l T l Let x be the equivalent focal length in the x-direction, the principal point of the image in the x-direction, the equivalent focal length in the y-direction, the principal point of the image in the y-direction, the rotation matrix, and the translation vector of one of the high-speed cameras 2. l ,y l ) for t i The image coordinates of the target in the high-speed camera 2, Z Cl For P i The projection of the distance to the optical center of the high-speed camera 2 along the optical axis; f xr u 0r f yr v 0r R r T r The equivalent focal length in the x-direction of another high-speed camera 2, the principal point of the image in the x-direction, the rotation matrix, and the translation vector are respectively (x r ,y r Let Z be the image coordinates of the target in another high-speed camera 2 at time ti. Cr For P i The projection of the distance to the optical center of another high-speed camera 2 in the direction of the optical axis.

[0073] Step 6.2: Based on the target's three-dimensional coordinates changing over time in a preset coordinate system, calculate the target's key parameters at different times to predict the high-speed target's position. The key parameters include the target's position and velocity. The calculation process for the target's key parameters is as follows:

[0074] First, calculate t according to the following formula. i The velocity components v of the target in the X, Y, and Z directions Xi ,v Yi ,v Zi :

[0075] v Xi=(X i+1 -X i-1 ) / (t i+1 -t i-1 )

[0076] v Yi =(Y i+1 -Y i-1 ) / (t i+1 -t i-1 )

[0077] v Zi =(Z i+1 -Z i-1 ) / (t i+1 -t i-1 )

[0078] Among them, (X) i+1 ,Y i+1 Z i+1 ) for t i+1 The three-dimensional coordinates of the target in the preset coordinate system, (X) i-1 ,Y i-1 Z i-1 ) for t i-1 The three-dimensional coordinates of the target in the preset coordinate system;

[0079] Then calculate t according to the following formula i The target velocity v at that time i :

[0080]

[0081] According to t i The three-dimensional coordinates P of the target in the preset coordinate system i (X i ,Y i Z i ) and the velocity components v in the X, Y, and Z directions Xi ,v Yi ,v Zi It can predict the target position at any time t, specifically calculated according to the following formula:

[0082] X t =X i +v Xi ·(tt i )

[0083] Y t =Y i +v Yi ·(tt i )

[0084] Z t =Z i +vZi ·(tt i )

[0085] Among them, (X) t ,Y t Z t ) represents the three-dimensional coordinates of the target in the preset coordinate system at time t.

[0086] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present invention should be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A high-speed target position prediction method based on event cameras and high-speed cameras, characterized in that, Includes the following steps: Step 1: Arrange two camera groups facing the target observation area (3) with orthogonal perspectives; the camera groups include an event camera (1) and a high-speed camera (2), and use satellite time synchronization technology to synchronize the two event cameras (1) and the two high-speed cameras (2); Step 2: Calibrate the event camera (1) and the high-speed camera (2) to obtain the intrinsic and extrinsic parameters of the event camera (1) and the high-speed camera (2) in each camera group; the intrinsic parameters include the equivalent focal length and the principal point of the image; the extrinsic parameters include the rotation matrix and the translation vector; calculate the intrinsic and extrinsic parameters of the event camera (1) according to the following formula: ； in, , , , , , Let x be the equivalent focal length in the x-direction, the principal point of the image in the x-direction, the equivalent focal length in the y-direction, the principal point of the image in the y-direction, the rotation matrix, and the translation vector of the event camera (1). Let be the image coordinates of the UAV in the event camera (1) during the kth hovering event. Let be the three-dimensional coordinates of the UAV in the preset coordinate system during the k-th hovering. for The projection point in the image of the event camera (1), K is the number of times the drone hovers; Step 3: Use each event camera (1) to collect the event stream triggered by the high-speed target, and use each high-speed camera (2) to obtain the motion image sequence of the high-speed target; use the event stream collected by each event camera (1) to perform coarse localization of the target, and obtain the rough image coordinates and timestamps of the target in each event camera (1); Step 4: Using the approximate image coordinates and timestamps of the target in each event camera (1), and the intrinsic and extrinsic parameters of the event camera (1) and high-speed camera (2) in each camera group, find the image frame with the corresponding timestamp in the motion image sequence acquired by the high-speed camera (2) which is in the same camera group as each event camera (1). Use the target detection algorithm to perform fine positioning of the target and obtain the precise image coordinates of the target in each high-speed camera (2); the target detection algorithm adopts the YOLOx series algorithm. Step 5: Using the precise image coordinates of the target in each high-speed camera (2) and the motion image sequence acquired by each high-speed camera (2), target tracking is performed based on the Kalman filter algorithm to obtain the precise image coordinates of the target in each high-speed camera (2) as time changes. Step 6: Using the precise image coordinates of the target changing over time in each high-speed camera (2), and the intrinsic and extrinsic parameters of each high-speed camera (2), calculate the key parameters of the target at different times to complete the prediction of the high-speed target position; the key parameters include the target velocity and the target position; calculated according to the following formula. Target speed at time : ； in, , , , They are respectively The target at time Velocity components in three directions, for The three-dimensional coordinates of the target in the preset coordinate system. for The three-dimensional coordinates of the target in the preset coordinate system; Calculate any time using the following formula. Target location: ； in, for The three-dimensional coordinates of the target in the preset coordinate system.

2. The high-speed target position prediction method based on event camera and high-speed camera according to claim 1, characterized in that: In step 2, the event camera (1) and the high-speed camera (2) are calibrated using the binocular camera calibration method.

3. The high-speed target position prediction method based on event camera and high-speed camera according to claim 2, characterized in that, Step 2 specifically includes the following steps: Step 2.1: Hover the UAV at different locations within the target observation area (3) and record the GPS coordinates of the UAV. At the same time, each event camera (1) and each high-speed camera (2) simultaneously take pictures and extract the image coordinates corresponding to each hovering from each event camera (1) and each high-speed camera (2). Step 2.2: Convert the GPS coordinates of the UAV obtained in Step 2.1 to a preset coordinate system to obtain the three-dimensional coordinates of the UAV in the preset coordinate system; Step 2.3: Combining the image coordinates obtained in Step 2.1 and the three-dimensional coordinates of the UAV in the preset coordinate system obtained in Step 2.2, the intrinsic and extrinsic parameters of the event camera (1) and the high-speed camera (2) in each camera group are calculated using the direct linear transformation method. The calculation methods for the intrinsic and extrinsic parameters of the high-speed camera (2) are the same as those for the event camera (1).

4. The high-speed target position prediction method based on event camera and high-speed camera according to claim 3, characterized in that, Step 6 specifically includes the following steps: Step 6.1: Using the precise image coordinates of the target in each high-speed camera (2) over time, and the intrinsic and extrinsic parameters of each high-speed camera (2), solve the following equations simultaneously to calculate... The three-dimensional coordinates of the target in the preset coordinate system : ； ； in, , , , , , Let x be the equivalent focal length in the x-direction, the principal point of the image in the x-direction, the equivalent focal length in the y-direction, the principal point of the image in the y-direction, the rotation matrix, and the translation vector of one of the high-speed cameras (2). for The image coordinates of the target in the high-speed camera (2) at that time. for The projection of the distance to the optical center of the high-speed camera (2) along the optical axis; , , , , , Let x be the equivalent focal length in the x-direction of another high-speed camera (2), x-direction image principal point, rotation matrix, and translation vector, respectively. for The image coordinates of the target in another high-speed camera (2). for The projection of the distance to the optical center of another high-speed camera (2) along the optical axis; Step 6.2: Based on the target's three-dimensional coordinates changing over time in the preset coordinate system, calculate the key parameters of the target at different times to complete the prediction of the high-speed target's position.

5. The high-speed target position prediction method based on event camera and high-speed camera according to claim 4, characterized in that: In step 5, the Kalman filtering algorithm uses linear Kalman filtering.

Images