Collision time estimation method, electronic device and storage medium based on event camera

Abstract

The present invention provides a collision time estimation method based on an event camera, an electronic device and a storage medium, which estimate the collision time based on the event stream collected by the event camera, overcome the problem of very large delay in the prior art, and improve the accuracy of collision time estimation, as well as the robustness in autonomous driving scenarios. At the same time, the low energy consumption feature enables the method to be better applicable to embedded scenarios, reducing procurement and operating costs. The TTC output frequency of the method in this embodiment can reach 200 Hz, which has great application potential in real-time TTC tasks and is more suitable for real-time TTC tasks with sudden changes in relative speed.

Description

Technical Field

[0001] The present invention belongs to the technical field of automobile assisted driving, and in particular relates to a collision time estimation method based on an event camera, an electronic device and a storage medium. Background Art

[0002] As more and more vehicles are equipped with driver assistance systems, various driver assistance tasks are defined and implemented, such as lane following, pedestrian recognition, etc. Among these tasks, the implementation of vehicle collision warning system is very important and challenging.

[0003] At present, there are a large number of research cases that use monocular cameras to obtain continuous images and estimate the time to collision (TTC) based on the continuous images. Since the image size of the front vehicle will change when there is relative motion with the vehicle, the TTC can be obtained by the change in image size. However, the problem with this method is that the update rate of TTC is limited by the frame rate of the standard camera. Considering the cost, bandwidth and energy consumption, the standard camera used in the automatic assisted driving system usually runs at a frequency of about 10hz, and the interval between two consecutive exposures is about 100ms. Even without considering the calculation time of the applied TTC algorithm, it is also a very large delay for the collision warning system, especially when the relative speed increases sharply. Summary of the invention

[0004] Based on this, in order to solve the above technical problems, a collision time estimation method based on an event camera, an electronic device and a storage medium are provided.

[0005] The technical solution adopted by the present invention is as follows:

[0006] As a first aspect of the present invention, a collision time estimation method based on an event camera is provided, characterized by comprising:

[0007] S101, acquiring an event stream in real time through an event camera of the vehicle, acquiring a front image in real time through a frame camera of the vehicle, tracking a target frame of a target front vehicle in real time, eliminating events outside the target frame, and determining a front vehicle event triggered by a contour point of the target front vehicle;

[0008] S102, extracting an event within a time period of Δt from the preceding vehicle event as a target event, wherein Δt is a difference between a first moment and a second moment, wherein the first moment represents the current moment, and the second moment is earlier than the first moment;

[0009] S103, pass The normalized coordinates of the target contour points corresponding to each target event are transformed to the reference time t through the time-varying affine transformation ref The normalized coordinates of p k Represents the normalized coordinates of the target contour points, t ref Located between the first moment and the second moment, t k represents the timestamp of the target event, v = [v x ,v y ,v z ] represents the reference time t ref The relative instantaneous speed of the target vehicle in front of the vehicle in the coordinate system, v x ,v y ,v z They represent the components of v in the three directions of the vehicle coordinate system xyz axis, Z(t ref ) represents the contour point of the target vehicle at the reference time t ref The z-coordinate of

[0010] S104, through the objective function Determine the optimal a, and determine the optimal a based on the optimal a z , for the optimal a z Find the reciprocal and get ref The collision time between the vehicle and the target vehicle is ttc ref , calculate the collision time ttc between the current vehicle and the target vehicle in front d =ttc ref -(t d -t ref ),in, The function is used to calculate the reference time t for each target contour point ref and the distance reference time t ref The time difference between the timestamps of the most recent events triggered by this target contour point, e k represents the target event, ε represents the set of target events, t d Represents the current moment.

[0011] As a second aspect of the present invention, an electronic device is provided, comprising a storage module, wherein the storage module comprises instructions loaded and executed by a processor, wherein when the instructions are executed, the processor executes a collision time estimation method based on an event camera according to the first aspect.

[0012] As a third aspect of the present invention, a computer-readable storage medium is provided, which stores one or more programs. When the one or more programs are executed by a processor, the method for estimating collision time based on an event camera according to the first aspect is implemented.

[0013] The present invention estimates the collision time based on the event stream collected by the event camera, which overcomes the problem of very large delay in the prior art and improves the accuracy of collision time estimation and the robustness in autonomous driving scenarios. At the same time, the low energy consumption feature enables the method to be better applicable to embedded scenarios, reducing procurement and operating costs. The TTC output frequency of the method in this embodiment can reach 200 Hz, which has great application potential in real-time TTC tasks and is more suitable for real-time TTC tasks with sudden changes in relative speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments:

[0015] Figure 1 A flowchart of a collision time estimation method based on an event camera provided in an embodiment of the present invention;

[0016] Figure 2 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0017] The following will illustrate the implementation methods of the present invention in conjunction with the drawings in the specification. It should be noted that the implementation methods involved in this specification are not exhaustive and do not represent the only implementation methods of the present invention. The following corresponding embodiments are only for the purpose of clearly illustrating the invention content of the patent of the present invention, and are not intended to limit its implementation methods. For ordinary technicians in this field, different forms of changes and modifications can be made on the basis of the description of this embodiment. All obvious changes or modifications that belong to the technical concept and invention content of the present invention are also within the scope of protection of the present invention.

[0018] like Figure 1 As shown, an embodiment of the present invention provides a collision time estimation method based on an event camera, and the specific process is as follows:

[0019] S101, obtaining an event stream in real time through the event camera of the vehicle, obtaining a front image in real time through the frame camera of the vehicle, tracking a target frame of the target front vehicle in real time, eliminating events outside the target frame, and determining a front vehicle event triggered by a contour point of the target front vehicle.

[0020] Compared with traditional frame cameras, event cameras have the characteristics of low latency, high dynamic range, and extremely low power consumption. They output the change in pixel brightness. When the cumulative brightness change of a pixel reaches a certain threshold, an event is output. An event has three elements: timestamp, pixel coordinates, and polarity (brightening or darkening). When a large number of pixel brightness changes in the scene due to object movement or lighting changes, a series of events will be generated, which are output in the form of event streams.

[0021] In this embodiment, the target front vehicle in the front image is identified by YOLOv5, and the target front vehicle is tracked by DeepSort algorithm, so as to achieve tracking of the target frame of the target front vehicle.

[0022] S102: extracting events within a time period of Δt from the preceding vehicle events as target events.

[0023] Wherein, Δt is the difference between the first moment and the second moment, the first moment represents the current moment, and the second moment is earlier than the first moment. Δt can be a preset value initially, such as 0.1 seconds. When the number of preceding vehicle events meets the preset number, Δt is dynamically determined. For example, if the number of preceding vehicle events within the first 0.08 seconds has met the preset number, Δt is adjusted to 0.08 seconds.

[0024] In this embodiment, the target event is represented by an LTS image, that is, the target event is rendered into an LTS image, and the pixel value of a pixel point of the LTS image is determined by the following formula:

[0025]

[0026] Among them, x represents the coordinates of the pixel point, that is, the normalized coordinates of the target contour point corresponding to the target event, ε x Represents the set of events triggered by the pixel, t i Represents ε x Events in i The trigger time, t ref represents a reference time, which is between the first time and the second time. In this embodiment, the reference time t ref Take the middle value between the first moment and the second moment.

[0027] Different from image-like representations of event data in a time surface (TS), we propose a linear time surface (LTS). Different from ordinary time surface representations, LTS images store the reference time t ref and the distance reference time t ref The time difference between the timestamps of the most recent triggered events, that is, the above formula (1) can be understood as calculating the reference time t for each pixel ref and the distance reference time t ref The time difference between the timestamps of the most recent event triggered by the pixel is used as the pixel value of the pixel.

[0028] The LTS proposed here, on the one hand, has the property of distance field in a similar sense to TS, which enables us to effectively establish the association of event contours. On the other hand, LTS enhances the continuity of distance field gradient in a different way. Unlike TS, LTS simply takes the reference time tref is set to the median timestamp of all relevant events, and TS uses a smoothing kernel to avoid the unilateral truncation of TS gradient. This design brings two advantages. First, the contour points at the reference time t ref There is no offset in the true position of , so there is no bias in the registration result. Secondly, the resulting distance transformation becomes a signed function, which will lead to a more accurate registration result.

[0029] Although most background events outside the target box of the target vehicle in front have been removed in S101, in order to further resist noise and outliers, the pixels of the LTS image are screened in the following way:

[0030] (i) Determine the first-order gradient amplitude and second-order gradient amplitude of each point in the LTS image.

[0031] (ii) If the first-order gradient amplitude of the same point is greater than a preset first threshold, and the second-order gradient amplitude is less than a preset second threshold, the point is retained as a valid point, wherein the first threshold is greater than or equal to 0.0001 and less than or equal to 0.1, and the second threshold is greater than or equal to 0.00000001 and less than or equal to 0.0001. Exemplarily, the first threshold and the second threshold are 0.0001 and 0.000001, respectively.

[0032] S103, pass The normalized coordinates of the target contour points corresponding to each target event are transformed to the reference time t through the time-varying affine transformation ref The normalized coordinates of p k Represents the normalized coordinates of the target contour points, t k represents the timestamp of the target event, v = [v x ,v y ,v z ] represents the reference time t ref The relative instantaneous speed between the vehicle and the target vehicle, v x ,v y ,v z Represents the components of v in the xyz direction, Z(t ref ) represents the contour point of the target vehicle at the reference time t ref The Z coordinate of .

[0033] When the relative distance changes, especially in a short period of time, the key to fitting the geometric model of the event is to use an accurate model. Therefore, the embodiment of the present invention proposes a time-varying affine model based on the real dynamics of the contour points. The model is expressed in continuous time form, and the relative instantaneous speed of the vehicle and the preceding vehicle is v = [vx ,v y ,v z ], the 3D contour point coordinates of the front vehicle in the scene are B P=[X,Y,Z],p=[x,y] T =[Y / Z,Y / Z] T 3D contour points B The normalized coordinates of the image of P, the optical flow u (optical flow is the instantaneous speed of the pixel movement of a moving object in space on the observation imaging plane) can be defined as the ordinary differential equation (ODE) of p, and its expression is:

[0034]

[0035] Based on this expression, t0 is the start time, p is at the reference time t ref The position of can be found exactly by the following integral:

[0036]

[0037] By solving the ODE expression, substituting t ref The boundary conditions at (t ref ,t0 integration interval) to obtain the general solution:

[0038]

[0039] in, It is a time-varying affine model, which can be regarded as the average optical flow in the time interval, and is used to accurately guide the distortion of events in the spatiotemporal domain. Therefore, in S103, The normalized coordinates of the target contour points corresponding to the target event are transformed to the reference time t through the time-varying affine transformation ref Normalized coordinates. To simplify the calculation, no p is added to each target contour point. k Based on the above formula (3),

[0040] S104, through the objective function Determine the optimal a, and determine the optimal a based on the optimal a z , for the optimal a z Find the reciprocal and get ref The collision time between the vehicle and the target vehicle is ttc ref , calculate the collision time ttc between the current vehicle and the target vehicle in front d =ttc ref -(t d -t ref ), where W(p k ,t k ,t ref; a) is the warping function, which can be approximated as Right now The function is used to calculate the reference time t for each target contour point ref and the distance reference time t ref The time difference between the timestamps of the most recent events triggered by this target contour point, e k represents the target event, ε represents the set of target events, t d Represents the current moment.

[0041] It should be pointed out that by obtaining the optimal a, the optimal time-varying affine model is obtained, so that all the events involved can be correctly mapped to the reference time t ref The position of the , to achieve spatiotemporal registration. z The reciprocal of Z(t ref ) / v z , that is, at the reference time t ref The distance between the vehicle and the preceding vehicle divided by the relative instantaneous speed in the z direction of the two vehicles is the result at the reference time t ref The collision time between the vehicle and the target vehicle is ttc ref .

[0042] In order to determine the optimal a more efficiently, the embodiment of the present invention provides a high-quality initial value a for the objective function, and obtains the optimal a according to the initial value a. The specific process is as follows:

[0043] (1) Three valid points are selected from the LTS image, and the normal flow vectors of the three valid points are calculated respectively according to the gradients of the three valid points. It should be noted that calculating the normal flow vector of a pixel point according to the gradient is a well-known technique and will not be described in detail here.

[0044] (2) To overcome the limitation of normal flow, we propose a more effective geometric metric method: the inner product of the full flow vector and the normal flow vector is equal to the square norm of the normal flow vector.

[0045] Since a has three directional components, that is, three unknowns, it is necessary to substitute the normal flow vectors of the three valid points into the following formula respectively and determine the initial value of a by the RANSAC method:

[0046]

[0047] Among them, n k represents the normal flow vector, n k,x and n k,y Represents nk The components in the x and y directions, δt k is the pixel value of the target point.

[0048] Take the pixel position p k and time t k Taking the triggered event as an example, the omnidirectional flow vector is calculated using formula (6):

[0049]

[0050] The above measurement method is expressed by formula (7):

[0051]

[0052] Where n = [n x ,n y ] T Represents the normal flow of the event optical flow, which is expressed in Equation 7 by replace The linear state vector can be directly established, namely the above equation (5).

[0053] (3) Each time the optimal a is determined, determine whether the difference between the first moment of this Δt and the first moment of the previous Δt is greater than the previous Δt. If so, re-execute steps (1) and (2) to calculate the initial value a. Otherwise, the next time the optimal a is determined, the previous optimal a is used as the initial value.

[0054] It should be noted that in the actual application of step 101, due to the frame rate of the frame camera, the algorithm tracking method cannot satisfy the requirement of obtaining the target frame of the target vehicle in front at any time, so a target frame of the vehicle in front needs to be predicted at this time:

[0055] At the current moment, the omnidirectional flow is calculated based on the penultimate target frame and the penultimate target frame of the target vehicle in front obtained by real-time tracking (the expansion rate of the target frame on the x and y axes is calculated respectively according to the two target frames to obtain the omnidirectional flow), and the time interval Δt1 between the current moment and the moment when the penultimate target frame is located is calculated. The omnidirectional flow is multiplied by Δt1 and then added to the size of the penultimate target frame, and finally multiplied by the preset coefficient to obtain the predicted size of the target frame of the vehicle in front, that is, the predicted coordinates of the target frame of the vehicle in front are obtained (assuming that the center coordinates of the target frame of the vehicle in front remain unchanged).

[0056] The predicted target frame of the preceding vehicle may become larger or smaller than the previous target frame, but in order to better obtain the event triggered by the preceding vehicle outline, the preset coefficient is set to 120%.

[0057] The method of the embodiment of the present invention estimates the collision time based on the event stream collected by the event camera, overcomes the problem of very large delay in the prior art, and improves the accuracy of collision time estimation and the robustness in autonomous driving scenarios. At the same time, the low energy consumption feature enables the method to be better suitable for embedded scenarios, reducing procurement and operating costs. The TTC output frequency of the method of this embodiment can reach 200 Hz, which has great application potential in real-time TTC tasks and is more suitable for real-time TTC tasks with sudden changes in relative speed.

[0058] Similar to the above concept, an embodiment of the present invention further provides a schematic structural block diagram of an electronic device.

[0059] Exemplarily, the electronic device includes a storage module 21 and a processor 22, the storage module 21 includes instructions loaded and executed by the processor 22, and when the instructions are executed, the processor 22 performs the steps of various exemplary embodiments of the present invention described in the above-mentioned part of the event camera-based collision time estimation method.

[0060] It should be understood that the processor 22 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0061] A computer-readable storage medium is also provided in an embodiment of the present invention, which stores one or more programs. When the one or more programs are executed by a processor, the steps of various exemplary embodiments of the present invention described in the above-mentioned collision time estimation method based on an event camera are implemented.

[0062] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware implementation, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable storage medium, which may include a computer-readable storage medium (or a non-transitory medium) and a communication medium (or a temporary medium).

[0063] As is known to those of ordinary skill in the art, the term computer-readable storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. In addition, it is known to those of ordinary skill in the art that communication media typically contain computer-readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and may include any information delivery medium.

[0064] Exemplarily, the computer-readable storage medium may be an internal storage unit of the electronic device of the aforementioned embodiment, such as a hard disk or memory of the electronic device. The computer-readable storage medium may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card (SmartMediaCard, SMC), a secure digital (SecureDigital, SD) card, a flash memory card (FlashCard), etc. equipped on the electronic device.

[0065] The electronic device and computer-readable storage medium provided in the aforementioned embodiments estimate the collision time based on the event stream collected by the event camera, which overcomes the problem of very large delay in the prior art and improves the accuracy of collision time estimation and the robustness in autonomous driving scenarios. At the same time, the low energy consumption feature enables the method to be better suitable for embedded scenarios, reducing procurement and operating costs. The TTC output frequency of the method in this embodiment can reach 200 Hz, which has great application potential in real-time TTC tasks and is more suitable for real-time TTC tasks with sudden changes in relative speed.

[0066] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A collision time estimation method based on event camera, characterized in that: include: S101, acquiring an event stream in real time through an event camera of the vehicle, acquiring a front image in real time through a frame camera of the vehicle, tracking a target frame of a target front vehicle in real time, eliminating events outside the target frame, and determining a front vehicle event triggered by a contour point of the target front vehicle; S102, extracting an event within a time period of Δt from the preceding vehicle event as a target event, wherein Δt is a difference between a first moment and a second moment, wherein the first moment represents the current moment, and the second moment is earlier than the first moment; S103, pass The normalized coordinates of the target contour points corresponding to each target event are transformed to the reference time t through the time-varying affine transformation ref The normalized coordinates of p k Represents the normalized coordinates of the target contour points, t ref Located between the first moment and the second moment, t k represents the timestamp of the target event, v = [v x , v y , v z ] represents the reference time t ref The relative instantaneous speed of the target vehicle in front of the vehicle in the coordinate system, v x , v y , v z They represent the components of v in the three directions of the vehicle coordinate system xyz axis, Z(t ref ) represents the contour point of the target vehicle at the reference time t ref The z-coordinate of S104, through the objective function Determine the optimal a, and determine the optimal a based on the optimal a z , for the optimal a z Find the reciprocal and get ref The collision time between the vehicle and the target vehicle is ttc ref , calculate the collision time ttc between the current vehicle and the target vehicle in front d =ttc ref -(t d -t ref ),in, The function is used to calculate the reference time t for each target contour point ref and the distance reference time t ref The time difference between the timestamps of the most recent events triggered by this target contour point, e k represents the target event, ε represents the set of target events, t d Represents the current moment.

2. The method for estimating collision time based on an event camera according to claim 1, characterized in that: The target frame for real-time tracking of the target front vehicle further includes: The target frame of the target vehicle in front is detected in real time in the front image through the target detection model, and the target frame is tracked in real time through the target tracking algorithm.

3. The method for estimating collision time based on event camera according to claim 2, characterized in that: The target frame for real-time tracking of the target front vehicle further includes: At the current moment, the omnidirectional flow is calculated based on the penultimate target frame and the penultimate target frame of the target vehicle in front obtained by real-time tracking, and the time interval Δt1 between the current moment and the moment when the penultimate target frame is located is calculated. The omnidirectional flow is multiplied by Δt1 and then added to the size of the penultimate target frame, and finally multiplied by the preset coefficient to obtain the predicted size of the target frame of the vehicle in front.

4. The method for estimating collision time based on an event camera according to claim 1, characterized in that: The Δt is initially a preset value, and when the number of the preceding vehicle events meets the preset number, the Δt is dynamically determined.

5. The method for estimating collision time based on event camera according to claim 4, characterized in that: The reference time t ref Take the middle value between the first moment and the second moment.

6. The method for estimating collision time based on event camera according to claim 5, characterized in that: The extracting the event from the first moment to the second moment from the preceding vehicle event as the target event further includes: Render the target event into an LTS image, and determine the pixel value of the pixel point of the LTS image by the following formula: Among them, x represents the coordinate of the pixel point, ε x Represents the set of events triggered by the pixel, t i Represents ε x Events in i The trigger time.

7. The method for estimating collision time based on event camera according to claim 6, characterized in that: The step of extracting the event from the first moment to the second moment from the preceding vehicle event as the target event further includes: Determine the first-order gradient amplitude and the second-order gradient amplitude of each point in the LTS image; If the first-order gradient amplitude of the same point is greater than the preset first threshold and the second-order gradient amplitude is less than the preset second threshold, the point is retained as a valid point, wherein the first threshold is greater than or equal to 0.0001 and less than or equal to 0.1, and the second threshold is greater than or equal to 0.00000001 and less than or equal to 0.0001.

8. The method for estimating collision time based on event camera according to claim 7, characterized in that: Also includes: (1) selecting three valid points from the LTS image, and calculating the normal flow vectors of the three valid points respectively according to the gradients of the three valid points; (2) Substitute the normal flow vectors of the three effective points into the following formulas respectively, and determine the initial value of a by the RANSAC method: Among them, n k represents the normal flow vector, n k，x and n k，y Represents n k The components in the x and y directions, δt k is the pixel value of the target point; (3) After each determination of the optimal a, determine whether the difference between the first moment of this Δt and the first moment of the previous Δt is greater than the previous Δt. If so, re-execute steps (1) and (2) to calculate the initial value a. Otherwise, the next time the optimal a is determined, the previous optimal a is used as the initial value.

9. An electronic device, characterized in that: A storage module is included, wherein the storage module includes instructions loaded and executed by a processor, and when the instructions are executed, the processor executes a collision time estimation method based on an event camera according to any one of claims 1-8.

10. A computer-readable storage medium storing one or more programs, characterized in that: When the one or more programs are executed by the processor, the collision time estimation method based on an event camera according to any one of claims 1 to 8 is implemented.

Images