Time-to-collision estimation method based on event camera, and electronic device and storage medium

Abstract

A time-to-collision estimation method based on an event camera. The method comprises: acquiring an events stream in real time by means of an event camera of a host vehicle, acquiring an image of the front of the host vehicle in real time by means of a frame camera of the host vehicle, tracking a bounding box of a target front vehicle in real time, removing an event occurring outside the bounding box, and determining front vehicle events triggered by contour points of the target front vehicle; extracting events within a time Δt from the front vehicle events to serve as target events; performing time-varying affine transformation on normalized coordinates of target contour points corresponding to the target events, so as to obtain normalized coordinates that are at a reference moment tref; and determining optimal a by means of an objective function, determining optimal az on the basis of the optimal a, and calculating the reciprocal of the optimal az, so as to obtain a time-to-collision ttcref that is for the host vehicle and the target front vehicle tref at the reference moment, and calculating a time-to-collision ttcd that is for the host vehicle and the target front vehicle at the current moment. Further provided are an electronic device and a storage medium, which are used for executing the time-to-collision estimation method based on an event camera.

Description

Collision time estimation method, electronic device and storage medium based on event camera Technical Field

[0001] The present invention belongs to the technical field of automobile assisted driving, and in particular relates to a collision time estimation method based on an event camera, an electronic device and a storage medium. Background Art

[0002] As more and more vehicles are equipped with driver assistance systems, various driver assistance tasks have been defined and implemented, such as lane following, pedestrian recognition, etc. Among these tasks, the implementation of vehicle collision warning system is very important and challenging.

[0003] Currently, a large number of research cases have used monocular cameras to capture continuous images and estimate the time to collision (TTC) based on the continuous images. Since the image size of the preceding vehicle changes when there is relative motion with the host vehicle, the TTC can be obtained by the change in image size. However, the problem with this approach is that the update rate of TTC is limited by the frame rate of standard cameras. Considering cost, bandwidth, and energy consumption, standard cameras used in automated assisted driving systems typically operate at a frequency of around 10 Hz, and the interval between two consecutive exposures is around 100 ms. Even without considering the calculation time of the applied TTC algorithm, this is a very large delay for the collision warning system, especially when the relative speed increases sharply.

[0004] Summary of the Invention

[0005] Based on this, in order to solve the above technical problems, a collision time estimation method based on an event camera, an electronic device and a storage medium are provided.

[0006] The technical solution adopted in the present invention is as follows:

[0007] As a first aspect of the present invention, a collision time estimation method based on an event camera is provided, characterized by comprising:

[0008] S101, acquiring an event stream in real time through the vehicle's event camera, acquiring a front image in real time through the vehicle's frame camera, tracking a target frame of a preceding vehicle in real time, eliminating events outside the target frame, and determining preceding vehicle events triggered by the contour points of the preceding vehicle;

[0009] S102: extracting an event within a time period of Δt from the preceding vehicle event as a target event, where Δt is the difference between a first moment and a second moment, the first moment representing the current moment, and the second moment being earlier than the first moment;

[0010] S103, pass The normalized coordinates of the target contour points corresponding to each target event are transformed into the reference time t through the time-varying affine transformation ref The normalized coordinates of p k Represents the normalized coordinates of the target contour points, t ref Between the first moment and the second moment, t k Represents the timestamp of the target event, v = [v x ,v y ,v z ] represents the reference time t ref The relative instantaneous speed of the target vehicle in the vehicle coordinate system, v x ,v y ,v z They represent the components of v in the three directions of the vehicle coordinate system xyz axis, Z(t ref ) represents the contour point of the target vehicle at the reference time t ref The z-coordinate of

[0011] S104, through the objective function Determine the optimal a, and determine the optimal a based on the optimal a z , for the optimal a z Find the reciprocal and get the value at the reference time t ref The collision time between the vehicle and the target vehicle is ttc ref , calculate the collision time ttc between the current vehicle and the target vehicle in front d =ttc ref -(t d -t ref ),in, The function is used to calculate the reference time t for each target contour point ref and the distance reference time t ref The time difference between the timestamps of the most recent events triggered by this target contour point, e k represents the target event, ε represents the set of target events, t d Represents the current moment.

[0012] As a second aspect of the present invention, an electronic device is provided, comprising a storage module, wherein the storage module comprises instructions loaded and executed by a processor, wherein when the instructions are executed, the processor executes a collision time estimation method based on an event camera according to the first aspect.

[0013] As a third aspect of the present invention, a computer-readable storage medium is provided, which stores one or more programs. When the one or more programs are executed by a processor, the method for estimating collision time based on an event camera according to the first aspect is implemented.

[0014] The present invention estimates the collision time based on the event stream captured by the event camera, overcoming the problem of very large delay in the prior art, and improving the collision time estimation accuracy and robustness in autonomous driving scenarios. At the same time, the low energy consumption feature makes the method better suitable for embedded scenarios, reducing procurement and operating costs. The TTC output frequency of the method in this embodiment can reach 200 Hz, which has great application potential in real-time TTC tasks and is more suitable for real-time TTC tasks with sudden changes in relative speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments:

[0016] FIG1 is a flow chart of a collision time estimation method based on an event camera according to an embodiment of the present invention;

[0017] FIG2 is a schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The following will illustrate the implementation of the present invention in conjunction with the drawings in the specification. It should be noted that the implementation methods involved in this specification are not exhaustive and do not represent the only implementation methods of the present invention. The following corresponding embodiments are only for the purpose of clearly illustrating the invention content of the patent of this invention and are not intended to limit its implementation methods. For ordinary technicians in this field, different forms of changes and modifications can be made based on the description of this embodiment. Any obvious changes or modifications that belong to the technical concept and invention content of the present invention are also within the scope of protection of the present invention.

[0019] As shown in FIG1 , an embodiment of the present invention provides a collision time estimation method based on an event camera. The specific process is as follows:

[0020] S101. Acquire an event stream in real time through the vehicle's event camera, acquire a front image in real time through the vehicle's frame camera, track the target frame of the target preceding vehicle in real time, eliminate events outside the target frame, and determine the preceding vehicle event triggered by the contour point of the target preceding vehicle.

[0021] Compared to traditional frame cameras, event cameras offer low latency, high dynamic range, and extremely low power consumption. They output changes in pixel brightness. When the cumulative brightness change of a pixel reaches a certain threshold, an event is generated. An event has three elements: a timestamp, pixel coordinates, and polarity (brightening or dimming). When a large number of pixel brightness changes in a scene due to object motion or lighting changes, a series of events are generated, which are output as an event stream.

[0022] In this embodiment, the target vehicle in front of the front image is identified by YOLOv5, and the target vehicle in front is tracked by DeepSort algorithm, thereby achieving tracking of the target frame of the target vehicle in front.

[0023] S102 : Extract the event within a time period of Δt from the preceding vehicle event as the target event.

[0024] Where Δt is the difference between the first and second moments. The first moment represents the current moment, and the second moment is earlier than the first. Δt can initially be a preset value, such as 0.1 seconds. When the number of preceding vehicle events meets the preset number, Δt is dynamically determined. For example, if the number of preceding vehicle events within the previous 0.08 seconds has met the preset number, Δt is adjusted to 0.08 seconds.

[0025] In this embodiment, the target event is represented by an LTS image, that is, the target event is rendered into an LTS image, and the pixel value of a pixel point of the LTS image is determined by the following formula:

[0026] Among them, x represents the coordinates of the pixel point, that is, the normalized coordinates of the target contour point corresponding to the target event, ε x Represents the set of events triggered by the pixel, t i Represents ε x Event e in i The trigger time, t ref represents the reference time, which is between the first time and the second time. In this embodiment, the reference time t ref Take the middle value between the first moment and the second moment.

[0027] Different from the image-like representation of event data in time surface (TS), we propose a linear time surface (LTS). Different from the ordinary time surface representation, the LTS image stores the reference time t ref and the distance reference time t ref The time difference between the timestamps of the most recent triggered events, that is, the above formula (1) can be understood as calculating the reference time t for each pixel ref and the distance reference time t refThe time difference between the timestamps of the most recent event triggered by the pixel point is used as the pixel value of the pixel point.

[0028] The LTS proposed here, on the one hand, has the property of distance field in a similar sense to TS, which enables us to effectively establish the association of event contours. On the other hand, LTS enhances the continuity of distance field gradient in a different way. Unlike TS, LTS simply takes the reference time t ref Set to the median timestamp of all relevant events, and TS uses a smoothing kernel to avoid the unilateral truncation of TS gradient. This design brings two advantages. First, the contour points at the reference time t ref There is no offset in the true position of , so there is no bias in the registration result. Secondly, the resulting distance transform becomes a signed function, which will lead to more accurate registration results.

[0029] Although most background events outside the target box of the target vehicle have been removed in S101, in order to further resist noise and outliers, the pixels of the LTS image are filtered in the following way:

[0030] (i) Determine the first-order gradient amplitude and second-order gradient amplitude of each point in the LTS image.

[0031] (ii) If the first-order gradient magnitude of the same point is greater than a preset first threshold and the second-order gradient magnitude is less than a preset second threshold, then the point is retained as a valid point, where the first threshold is greater than or equal to 0.0001 and less than or equal to 0.1, and the second threshold is greater than or equal to 0.00000001 and less than or equal to 0.0001. For example, the first threshold and the second threshold are 0.0001 and 0.000001, respectively.

[0032] S103, pass The normalized coordinates of the target contour points corresponding to each target event are transformed into the reference time t through the time-varying affine transformation ref The normalized coordinates of p k Represents the normalized coordinates of the target contour points, t k Represents the timestamp of the target event, v = [v x ,v y ,v z ] represents the reference time t ref The relative instantaneous speed between the vehicle and the target vehicle, v x ,v y ,v z Represents the components of v in the xyz direction, Z(t ref) represents the contour point of the target vehicle at the reference time t ref The Z coordinate of .

[0033] When the relative distance changes, especially within a short period of time, the key to fitting a geometric model of an event is to use an accurate model. Therefore, the embodiment of the present invention proposes a time-varying affine model based on the real dynamics of the contour points. The model is expressed in continuous time form, and the relative instantaneous speed of the vehicle and the preceding vehicle is v = [v x ,v y ,v z ], the 3D contour point coordinates of the front vehicle in the scene are B P =[X,Y,Z], p = [x,y] T =[X / Z,Y / Z] T 3D contour points B The normalized coordinates of the image of P, the optical flow u (optical flow is the instantaneous speed of the pixel movement of a moving object in space on the observation imaging plane) can be defined as the ordinary differential equation (ODE) of p, which is expressed as follows:

[0034] Based on this expression, t0 is the starting time, p is at the reference time t ref The position of can be found exactly by the following integral:

[0035] By solving the ODE expression, substituting t ref The boundary conditions at (t ref ,t0 integration interval) to obtain the general solution:

[0036] in, It is a time-varying affine model, which can be regarded as the average optical flow in the time interval, and is used to accurately guide the distortion of events in the spatiotemporal domain. Therefore, in S103, The normalized coordinates of the target contour points corresponding to the target event are transformed to the reference time t through the time-varying affine transformation ref Normalized coordinates, in order to simplify the calculation, no p is added to each target contour point k Based on the above formula (3),

[0037] S104, through the objective function Determine the optimal a, and determine the optimal a based on the optimal a z , for the optimal a z Find the reciprocal and get the value at the reference time t ref The collision time between the vehicle and the target vehicle is ttc ref , calculate the collision time ttc between the current vehicle and the target vehicle in frontd =ttc ref -(t d -t ref ), where W(p k ,t k ,t ref ; a) is the warping function, which can be approximated as Right now The function is used to calculate the reference time t for each target contour point ref and the distance reference time t ref The time difference between the timestamps of the most recent events triggered by this target contour point, e k represents the target event, ε represents the set of target events, t d Represents the current moment.

[0038] It should be pointed out that by obtaining the optimal a, the optimal time-varying affine model is obtained, so that all involved events can be correctly mapped to the reference time t ref The position of the time-space registration is realized. z The reciprocal of Z(t ref ) / v z , that is, at the reference time t ref The distance between the vehicle and the preceding vehicle is divided by the relative instantaneous speed in the z direction of the two vehicles, and the result is the velocity at the reference time t ref The collision time between the vehicle and the target vehicle is ttc ref .

[0039] In order to more efficiently determine the optimal a, the embodiment of the present invention provides a high-quality initial value a for the objective function, and obtains the optimal a based on the initial value a. The specific process is as follows:

[0040] (1) Three valid points are selected from the LTS image, and the normal flow vectors of the three valid points are calculated based on their gradients. It should be noted that the calculation of the normal flow vector of a pixel point based on the gradient is a well-known technique and will not be described in detail here.

[0041] (2) To overcome the limitation of normal flow, we propose a more efficient geometric metric: the inner product of the full flow vector and the normal flow vector is equal to the square norm of the normal flow vector.

[0042] Since a has three directional components, that is, three unknowns, it is necessary to substitute the normal flow vectors of the three valid points into the following formula respectively and determine the initial value of a using the RANSAC method:

[0043] Among them, n k represents the normal flow vector, n k,x and n k,y Represents n k Components in the x and y directions, δt k is the pixel value of the target point.

[0044] Take pixel position p k and time t k Taking the triggered event as an example, the omnidirectional flow vector is calculated using formula (6):

[0045] The above measurement method is expressed by formula (7):

[0046] Where n=[n x ,n y ] T Represents the normal flow of the event optical flow, which is expressed in Equation 7 by replace The linear state vector can be directly established, namely the above equation (5).

[0047] (3) Each time the optimal a is determined, determine whether the difference between the first moment of this Δt and the first moment of the previous Δt is greater than the previous Δt. If so, re-execute steps (1) and (2) to calculate the initial value a. Otherwise, the next time the optimal a is determined, the previous optimal a is used as the initial value.

[0048] It should be noted that in the actual application of step 101, due to the frame rate of the frame camera, the algorithm tracking method cannot satisfy the requirement of obtaining the target frame of the target vehicle in front at any time. Therefore, a target frame of the target vehicle in front needs to be predicted:

[0049] At the current moment, the omnidirectional flow is calculated based on the penultimate target frame and the penultimate target frame of the target vehicle in front obtained by real-time tracking (the expansion rate of the target frame on the x and y axes is calculated based on the two target frames to obtain the omnidirectional flow), and the time interval Δt1 between the current moment and the moment of the penultimate target frame is calculated. The omnidirectional flow is multiplied by Δt1 and then added to the size of the penultimate target frame. Finally, multiplied by the preset coefficient to obtain the size of the predicted target frame of the preceding vehicle, that is, the coordinates of the predicted target frame of the preceding vehicle are obtained (assuming that the center coordinates of the target frame of the preceding vehicle remain unchanged).

[0050] The predicted target frame of the preceding vehicle may become larger or smaller than the previous target frame, but in order to better obtain the event triggered by the preceding vehicle outline, the preset coefficient is set to 120%.

[0051] The method of the embodiment of the present invention estimates the collision time based on the event stream collected by the event camera, overcoming the problem of very large delay in the existing technology, and improving the accuracy of collision time estimation and robustness in autonomous driving scenarios. At the same time, the low energy consumption feature makes the method better suitable for embedded scenarios, reducing procurement and operating costs. The TTC output frequency of the method of this embodiment can reach 200hz, which has great application potential in real-time TTC tasks and is more suitable for real-time TTC tasks with sudden changes in relative speed.

[0052] Similar to the above concept, an embodiment of the present invention further provides a schematic structural block diagram of an electronic device.

[0053] Exemplarily, the electronic device includes a storage module 21 and a processor 22, the storage module 21 includes instructions loaded and executed by the processor 22, and when the instructions are executed, the processor 22 performs the steps of various exemplary embodiments of the present invention described in the above-mentioned part of the event camera-based collision time estimation method.

[0054] It should be understood that the processor 22 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0055] A computer-readable storage medium is also provided in an embodiment of the present invention. The computer-readable storage medium stores one or more programs. When the one or more programs are executed by a processor, the steps of various exemplary embodiments of the present invention described in the above-mentioned method for estimating collision time based on an event camera are implemented.

[0056] It will be understood by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware implementation, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable storage medium, which may include a computer-readable storage medium (or a non-transitory medium) and a communication medium (or a temporary medium).

[0057] As is well known to those skilled in the art, the term computer-readable storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer-readable storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by a computer. Furthermore, as is well known to those skilled in the art, communication media typically contains computer-readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0058] For example, the computer-readable storage medium may be an internal storage unit of the electronic device of the aforementioned embodiment, such as a hard disk or memory of the electronic device. The computer-readable storage medium may also be an external storage device of the electronic device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc., provided on the electronic device.

[0059] The electronic devices and computer-readable storage media provided in the aforementioned embodiments estimate the collision time based on the event stream collected by the event camera, overcoming the problem of very large delay in the prior art and improving the accuracy of collision time estimation and robustness in autonomous driving scenarios. At the same time, the low energy consumption feature makes this method better suitable for embedded scenarios, reducing procurement and operating costs. The TTC output frequency of the method in this embodiment can reach 200 Hz, which has great application potential in real-time TTC tasks and is more suitable for real-time TTC tasks with sudden changes in relative speed.

[0060] Obviously, those skilled in the art may make various changes and modifications to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is intended to include these modifications and variations.

Claims

1. A collision time estimation method based on event camera, characterized in that: include: S101, acquiring an event stream in real time through an event camera of the vehicle, acquiring a front image in real time through a frame camera of the vehicle, tracking a target frame of a target front vehicle in real time, eliminating events outside the target frame, and determining a front vehicle event triggered by a contour point of the target front vehicle; S102, extracting an event within a time period of Δt from the preceding vehicle event as a target event, wherein Δt is a difference between a first moment and a second moment, wherein the first moment represents the current moment, and the second moment is earlier than the first moment; S103, pass The normalized coordinates of the target contour points corresponding to each target event are transformed to the reference time t through the time-varying affine transformation ref The normalized coordinates of p k represents the normalized coordinates of the target contour points, t ref Located between the first moment and the second moment, t k represents the timestamp of the target event, v = [v x ,v y ,v z ] represents the reference time t ref The relative instantaneous speed of the target vehicle in front of the vehicle in the coordinate system, v x ,v y ,v z They represent the components of v in the three directions of the vehicle coordinate system xyz axis, Z(t ref ) represents the contour point of the target vehicle at the reference time t ref The z-coordinate of S104, through the objective function Determine the optimal a, and determine the optimal a based on the optimal a z , for the optimal a z Find the reciprocal and get ref The collision time between the vehicle and the target vehicle is ttc ref , calculate the collision time ttc between the current vehicle and the target vehicle in front d =ttc ref -(t d -t ref ),in, The function is used to calculate the reference time t for each target contour point ref and the distance reference time t ref The time difference between the timestamps of the most recent events triggered by this target contour point, e k represents the target event, ε represents the set of target events, t d Represents the current moment.

2. The method for estimating collision time based on an event camera according to claim 1, characterized in that: The target frame for real-time tracking of the target front vehicle further includes: The target frame of the target vehicle in front is detected in real time in the front image through the target detection model, and the target frame is tracked in real time through the target tracking algorithm.

3. The method for estimating collision time based on an event camera according to claim 2, characterized in that: The target frame for real-time tracking of the target front vehicle further includes: At the current moment, the omnidirectional flow is calculated based on the penultimate target frame and the penultimate target frame of the target vehicle in front obtained by real-time tracking, and the time interval Δt1 between the current moment and the moment when the penultimate target frame is located is calculated. The omnidirectional flow is multiplied by Δt1 and then added to the size of the penultimate target frame, and finally multiplied by the preset coefficient to obtain the predicted size of the target frame of the vehicle in front.

4. The method for estimating collision time based on an event camera according to claim 1, characterized in that: The Δt is initially a preset value, and when the number of the preceding vehicle events meets the preset number, the Δt is dynamically determined.

5. The method for estimating collision time based on event camera according to claim 4, characterized in that: The reference time t ref Take the middle value between the first moment and the second moment.

6. The method for estimating collision time based on event camera according to claim 5, characterized in that: The extracting the event from the first moment to the second moment from the preceding vehicle event as the target event further includes: Render the target event into an LTS image, and determine the pixel value of the pixel point of the LTS image by the following formula: Among them, x represents the coordinate of the pixel point, ε x Represents the set of events triggered by the pixel, t i Represents ε x Events in i The trigger time.

7. The method for estimating collision time based on event camera according to claim 6, characterized in that: The step of extracting the event from the first moment to the second moment from the preceding vehicle event as the target event further includes: Determine the first-order gradient amplitude and the second-order gradient amplitude of each point in the LTS image; If the first-order gradient amplitude of the same point is greater than the preset first threshold and the second-order gradient amplitude is less than the preset second threshold, the point is retained as a valid point, wherein the first threshold is greater than or equal to 0.0001 and less than or equal to 0.1, and the second threshold is greater than or equal to 0.00000001 and less than or equal to 0.0001.

8. The method for estimating collision time based on event camera according to claim 7, characterized in that: Also includes: (1) selecting three valid points from the LTS image, and calculating the normal flow vectors of the three valid points respectively according to the gradients of the three valid points; (2) Substitute the normal flow vectors of the three effective points into the following formulas respectively, and determine the initial value of a by the RANSAC method: Among them, n k represents the normal flow vector, n k,x and n k,y Represents n k The components in the x and y directions, δt k is the pixel value of the target point; (3) After each determination of the optimal a, determine whether the difference between the first moment of this Δt and the first moment of the previous Δt is greater than the previous Δt. If so, re-execute steps (1) and (2) to calculate the initial value a. Otherwise, the next time the optimal a is determined, the previous optimal a is used as the initial value.

9. An electronic device, characterized in that: A storage module is included, wherein the storage module includes instructions loaded and executed by a processor, and when the instructions are executed, the processor executes a collision time estimation method based on an event camera according to any one of claims 1-8.

10. A computer-readable storage medium storing one or more programs, characterized in that: When the one or more programs are executed by the processor, the collision time estimation method based on an event camera according to any one of claims 1 to 8 is implemented.

Images