A road pedestrian detection and tracking method and system based on event camera

Abstract

The present invention discloses a method and system for detecting and tracking pedestrians on a road based on an event camera, and relates to the technical field of detecting and tracking pedestrians on a road. The method comprises: collecting target area information through an event camera, and determining an image to be detected; performing edge detection processing on the image to be detected to determine preliminary image features; inputting the preliminary image features into a feature extraction model to determine the current pedestrian features; calculating the position information of the pedestrian on the road according to the current pedestrian features; if the position information of the pedestrian on the road is within the set safety area range, then performing feature tracking on the current pedestrian features to determine the pedestrian features at the next moment, and then updating the current pedestrian features to the pedestrian features at the next moment, and returning to the step of calculating the position information of the pedestrian on the road according to the current pedestrian features; if the position information of the pedestrian on the road is not within the set safety area range, then generating an emergency braking instruction. The present invention realizes more comprehensive and timely protection of pedestrians on the road.

Description

Technical Field

[0001] The present invention relates to the technical field of road pedestrian detection and tracking, and in particular to a road pedestrian detection and tracking method and system based on an event camera. Background Art

[0002] In recent years, traffic accidents caused by vehicles or obstacles blocking the line of sight and forming blind spots account for a relatively large proportion of pedestrian accidents, and the braking reaction time and braking severity in lateral conflicts are shorter and higher than those in longitudinal conflicts. In the existing technology, the application of visual algorithms has the following problems: the FOV field of view is not suitable; the traditional frame camera is not ideal for detecting lateral pedestrians; the amount of data is too large, and the computing power requirements are particularly high, so that it cannot respond quickly; it is greatly affected by sudden changes in light brightness and backlighting; there is no time to respond to pedestrians who suddenly appear in the blind spot of the view blocked by the vehicle or obstacle in front.

[0003] With the rapid development of intelligent driving technology, some methods have emerged to solve or mitigate the above-mentioned problems. For example, V2X technology can sense the surrounding information of the vehicle in advance and take collision avoidance measures to effectively avoid or reduce ghosting. For example, when a pedestrian suddenly appears in the blind spot of the vision blocked by a vehicle or obstacle in front, the vehicle will automatically perform emergency braking, thereby reducing the degree of damage to a certain extent. However, no matter which of the above solutions is used, it is impossible to protect pedestrians on the road in time. Summary of the invention

[0004] The purpose of the present invention is to provide a road pedestrian detection and tracking method and system based on event cameras, which can achieve faster detection response to road pedestrians and track and evaluate the response results to achieve more comprehensive and timely protection for road pedestrians.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] In a first aspect, the present invention provides a method for detecting and tracking pedestrians on a road based on an event camera, comprising:

[0007] Collecting target area information through an event camera and determining an image to be detected; the target area information includes pedestrians on the road;

[0008] Performing edge detection processing on the image to be detected to determine preliminary image features;

[0009] The preliminary image features are input into a feature extraction model to determine the current pedestrian features; the feature extraction model is obtained by training a cascade classifier according to a sample set; the cascade classifier is determined by integrating multiple Ada boost strong classifiers in series; the sample set includes multiple road pedestrian samples and multiple non-pedestrian samples;

[0010] Calculating the position information of the pedestrian on the road according to the current pedestrian characteristics; the position information of the pedestrian on the road includes the distance between the pedestrian on the road and the event camera;

[0011] Determine whether the pedestrian position information on the road is within a set safety area;

[0012] If the position information of the pedestrian on the road is within the set safety area, the feature tracking of the current pedestrian is performed to determine the pedestrian feature at the next moment, and then the current pedestrian feature is updated to the pedestrian feature at the next moment, and the step of returning to the step of calculating the position information of the pedestrian on the road according to the current pedestrian feature is returned;

[0013] If the pedestrian position information on the road is not within the set safety area, an emergency braking command is generated.

[0014] Optionally, the feature tracking of the current pedestrian feature is performed to determine the pedestrian feature at the next moment, specifically including:

[0015] The current pedestrian features are input into the Kalman filter to

[0016] x t|t-1 =Ax t-1|t-1 +Bu t

[0017] Calculate pedestrian features at the next moment;

[0018] Among them, x represents the pedestrian feature state variable, x t|t-1 represents the pedestrian features at the next moment, x t-1|t-1 represents the current pedestrian feature, A represents the state transfer matrix, B represents the gain matrix, u t represents the control variable at time t.

[0019] Optionally, the road pedestrian detection and tracking method further includes:

[0020] According to the formula

[0021] x t|t =x t|t-1 +K t (Z t -Cx t|t-1 )

[0022] K t =P t|t-1 C T (CP t|t-1 C T ) -1

[0023] P t|t=(IK t C)P t|t-1

[0024] Correcting and optimizing the pedestrian features at the next moment to obtain optimized pedestrian features at the next moment; the optimized pedestrian features at the next moment are used to calculate the road pedestrian position information corresponding to the next moment;

[0025] Among them, P t|t Represents the covariance matrix at the next moment, P t|t-1 Represents the covariance matrix at the current moment, A T represents the transpose of the state transfer matrix, Q represents noise, K t represents the Kalman gain, I represents the identity matrix, x t|t Indicates the optimization of pedestrian features at the next moment, Z t represents the pedestrian feature measurement value at the next moment, C represents the observation matrix, C T represents the transpose of the observation matrix, (CP t|t-1 C T ) -1 Indicates (CP t|t-1 C T ) is the inverse matrix of .

[0026] Optionally, edge detection processing is performed on the image to be detected to determine preliminary image features, specifically including:

[0027] Performing Gaussian filtering on the image to be detected to obtain a first feature map;

[0028] Using the Canny operator to calculate the gradient magnitude and gradient direction of each pixel in the first feature map;

[0029] According to the gradient size and gradient direction of each pixel, non-maximum suppression processing and double threshold screening processing are carried out in sequence to determine the preliminary image features.

[0030] Optionally, the training process of the Ada boost strong classifier includes:

[0031] Extracting the HOG features of each sample in the sample set;

[0032] Performing weight initialization processing on all samples in the sample set to determine an initial weight corresponding to each sample;

[0033] Perform multiple iterations of training based on the HOG features of each sample, and after each iteration of training, determine whether the sample error rate reaches a set value; the sample error rate is the probability that the sample is misclassified;

[0034] If the sample error rate does not reach the set value, it is determined whether the current number of iterations reaches the iteration threshold. If the current number of iterations does not reach the iteration threshold, the initial weight corresponding to each sample is updated, and a larger initial weight is given to the wrongly classified samples. When the current number of iterations reaches the iteration threshold, the training ends.

[0035] If the sample error rate reaches the set value, the training is terminated; the results of multiple iterative training corresponding to the sample error rate reaching the set value constitute the Ada boost strong classifier.

[0036] Optionally, extracting the HOG feature of each sample in the sample set specifically includes:

[0037] For each sample in the sample set, color space normalization processing, image gradient calculation, cell gradient histogram calculation, and histogram normalization processing are performed in sequence to determine the HOG feature of the sample.

[0038] Optionally, collecting target area information through an event camera and determining an image to be detected specifically includes:

[0039] The target area information is collected through the event camera, and the light intensity change of each pixel is recorded to determine the image light intensity change;

[0040] When the image light intensity variation is within a set light intensity threshold range, a plurality of pixel points corresponding to the image light intensity variation are output as an image to be detected.

[0041] In a second aspect, the present invention provides a road pedestrian detection and tracking system based on an event camera, comprising:

[0042] An image acquisition module, used to acquire target area information through an event camera and determine an image to be detected; the target area information includes pedestrians on the road;

[0043] An edge detection module is used to perform edge detection processing on the image to be detected to determine preliminary image features;

[0044] A feature extraction module is used to input the preliminary image features into a feature extraction model to determine the current pedestrian features; the feature extraction model is obtained by training a cascade classifier according to a sample set; the cascade classifier is determined by integrating multiple Ada boost strong classifiers in series; the sample set includes multiple road pedestrian samples and multiple non-pedestrian samples;

[0045] A position calculation module, used to calculate the position information of the road pedestrian according to the current pedestrian characteristics; the road pedestrian position information includes the distance between the road pedestrian and the event camera;

[0046] A position determination module is used to determine whether the position information of the pedestrian on the road is within a set safety area;

[0047] A position tracking module, used for tracking the current pedestrian features when the pedestrian position information on the road is within the set safety area to determine the pedestrian features at the next moment, then updating the current pedestrian features to the pedestrian features at the next moment, and returning to the position calculation module;

[0048] The emergency braking module is used to generate an emergency braking instruction when the position information of the pedestrian on the road is not within the set safety area.

[0049] Optionally, the edge detection module specifically includes:

[0050] A filtering processing submodule, used for performing Gaussian filtering on the image to be detected to obtain a first feature map;

[0051] A pixel calculation submodule, used to calculate the gradient magnitude and gradient direction of each pixel in the first feature map using a Canny operator;

[0052] The feature processing submodule is used to determine the preliminary image features through non-maximum suppression processing and double threshold screening processing in sequence according to the gradient size and gradient direction of each pixel.

[0053] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0054] The present invention provides a method and system for detecting and tracking pedestrians on a road based on an event camera. By determining the image to be detected through the event camera, a lot of invalid information can be reduced, computing power and power consumption can be reduced, thereby achieving a faster response. Then, edge detection processing is performed on the image to be detected, and the obtained preliminary image features are input into the feature extraction model to extract the current pedestrian features, wherein the feature extraction model is integrated by a plurality of Ada boost strong classifiers in series, so that more accurate feature extraction can be achieved. Finally, the position information of the pedestrian on the road is determined according to the current pedestrian features, and when the pedestrian is in the set safety area range, the current pedestrian features are tracked to determine the pedestrian features at the next moment, and further determine whether the position information corresponding to the pedestrian features at the next moment is in the set safety area range; when the pedestrian is not in the set safety area range, an emergency braking command is generated to avoid the pedestrian on the road in time, so as to achieve the prediction and tracking of the actions of the pedestrian on the road, thereby achieving more comprehensive and timely protection of the pedestrian on the road. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0056] Figure 1 It is a schematic diagram of the flow of the method for detecting and tracking pedestrians on a road based on an event camera of the present invention;

[0057] Figure 2 It is a schematic diagram of the structure of the road pedestrian detection and tracking system based on event camera of the present invention. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] When there is a blind spot in the field of vision blocked by a vehicle or obstacle ahead, the present invention can reduce or weaken the collision between the vehicle in the blind spot and the lateral and vertical pedestrians by taking advantage of the fast response speed and high dynamic range when facing pedestrians who suddenly appear on the road.

[0060] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0061] Embodiment 1

[0062] like Figure 1 As shown, this embodiment provides a method for detecting and tracking pedestrians on a road based on an event camera, comprising:

[0063] Step 100, collect target area information through the event camera and determine the image to be detected; the target area information includes pedestrians on the road. Specifically, when the vehicle is started, the ECU (Electronic Control Unit) set in the vehicle receives the vehicle start signal, and the vehicle control system, sensors, and event cameras are all turned on.

[0064] You should know that the event camera captures "events", which can be simply understood as "changes in pixel brightness", that is, the event camera outputs changes in pixel brightness. When the event camera collects information about the area in front, it records the positive or negative changes in light intensity at a certain pixel point and outputs a signal of the change in light intensity. And the event camera mainly displays the outer frame of the moving object in front. Since the changing part is mainly in the object frame, the area inside it is likely to remain unchanged. It is equivalent to compressing the image, reducing invalid information, and only outputting dynamic information, which can provide high-quality information with a low-bandwidth line. The dynamic range of the event camera can reach 120dB, or even higher. High dynamic range can help event cameras to maintain effective work in extremely dark light, overexposure, sudden changes in light, etc., adding a safety redundancy to autonomous driving.

[0065] In order to reduce the vibration of the event camera, super glue is used to firmly fix the event camera at a downward tilt in the middle of the upper part of the vehicle's windshield. This will not block the driver's line of sight and can also obtain more information about the road ahead and pedestrians.

[0066] Based on the above content, step 100 specifically includes:

[0067] 1) The target area information is collected through the event camera, and the light intensity change of each pixel is recorded to determine the image light intensity change.

[0068] 2) When the image light intensity variation is within the set light intensity threshold range, multiple pixel points corresponding to the image light intensity variation are output as the image to be detected. Specifically, the electronic control unit ECU performs a threshold judgment on the collected event stream (image light intensity variation). If it is within the set light intensity threshold range, the image to be detected is output; if it is not within the set light intensity threshold range, it is not output, and the pixel light intensity information needs to be repeatedly acquired until it reaches the set light intensity threshold range.

[0069] Before step 200, the image to be detected may be preprocessed, for example, the size of the image to be detected may be adjusted to a certain set size, and image enhancement processing may be performed on the image to be detected after the size adjustment.

[0070] Step 200, edge detection is performed on the image to be detected to determine preliminary image features. There are many commonly used edge detection algorithms, each with its own advantages and disadvantages: the Roberts edge detection operator accurately locates edges but is easily affected by noise; the Prewitt edge detection operator has a good advantage in processing noisy images; and the Canny edge detection operator can accurately locate edges, is less affected by noise interference, and can also optimize images. Therefore, the Canny edge detection operator is used in the specific embodiment of the present invention.

[0071] Specifically, step 200 includes:

[0072] 1) Performing Gaussian filtering on the image to be detected to obtain a first feature map; specifically, the calculation formula of Gaussian filtering is:

[0073]

[0074] Wherein, f(x,y) represents a pixel point in the image to be detected, and x and y represent the x-coordinate and y-coordinate of the pixel point in the image to be detected, respectively; is the standard deviation, Generally, a smaller value is selected to improve the accuracy of edge detection; G represents the first feature map after Gaussian processing.

[0075] 2) The Canny operator is used to calculate the gradient size (gradient amplitude) and gradient direction of each pixel in the first feature map; specifically, the Canny operator is calculated by the finite difference of the first-order partial derivative, and the mathematical expression of the Canny operator is as follows:

[0076] Gx=[f(x,y+1)-f(x,y)+f(x+1,y+1)-f(x+1,y)] / 2

[0077] Gy=[f(x,y)-f(x+1,y)+f(x,y+1)-f(x+1,y+1)] / 2

[0078] The gradient amplitude and gradient direction of the pixel in the image can be further obtained as follows:

[0079]

[0080] θ(x,y)=tan -1 (G y (x,y) / G x (x,y)).

[0081] 3) According to the gradient size and gradient direction of each pixel, non-maximum suppression processing and double threshold screening processing are performed in turn to determine the preliminary image features. Specifically, the edges in the image need to be refined to exclude interfering non-edge points. The key is to find the local maximum of the pixel points, rather than excluding these pixels. Then, two thresholds T1 and T2 are set, and most of the noise is removed according to the two thresholds. The pixels obtained by one of the thresholds are tracked and connected, and the other threshold is used as a supplement. Repeat until all contours in the image are found and the preliminary image features are determined.

[0082] Step 300, input the preliminary image features into a feature extraction model to determine the current pedestrian features; the feature extraction model is obtained by training a cascade classifier based on a sample set; the cascade classifier is determined by integrating multiple Adaboost strong classifiers in series; the sample set includes multiple road pedestrian samples and multiple non-pedestrian samples.

[0083] The Ada boost algorithm is widely used in obstacle recognition due to its high accuracy and speed in target detection. The Ada boost cascade classifier is an iterative algorithm that first trains samples to obtain some weak classifiers, and then combines these weak classifiers to obtain a strong classifier. Finally, multiple strong classifiers are connected in series to form a cascade classifier for image detection, and the detection results are more accurate.

[0084] Based on this, in a specific embodiment of the present invention, the training process of the Ada boost strong classifier includes:

[0085] 1) Extract the HOG feature of the histogram of directional gradients of each sample in the sample set. Specifically, the HOG feature divides the collected sample image into small cells, and then counts the gradient histogram of each cell. The image features can be obtained from the gradient histogram, and all the features can be combined to obtain the features of the entire sample image.

[0086] 2) Perform weight initialization processing on all samples in the sample set to determine the initial weight corresponding to each sample; specifically, assuming that there are n training samples in the sample set, 1 represents a positive sample and 0 represents a negative sample, and each sample is assigned the same initial weight.

[0087] 3) Perform multiple iterations of training based on the HOG features of each sample, and after each iteration of training, determine whether the sample error rate reaches the set value; the sample error rate is the probability that the sample is misclassified. Specifically, the sample is iterated m times, and a weak classifier is obtained through the sample features. Each weak classifier is used to filter negative samples and retain positive sample images. Through multiple weak classifiers, the number of negative samples is gradually reduced, and the positive samples are retained; then the sample error rate of the features in each weak classifier is calculated.

[0088] 4) If the sample error rate does not reach the set value, it is determined whether the current number of iterations reaches the iteration threshold. If the current number of iterations does not reach the iteration threshold, the initial weight corresponding to each sample is updated, and the wrongly classified samples are given a larger initial weight, that is, the weight of the sample is updated. When the current number of iterations reaches the iteration threshold, the training is terminated so that the iterative training will not be infinite and there will be no stopping moment.

[0089] 5) If the sample error rate reaches the set value, the training is terminated; the results of multiple iterative training corresponding to the sample error rate reaching the set value constitute the Ada boost strong classifier.

[0090] Finally, each Ada boost strong classifier is connected in series to form a cascade classifier. When using the cascade classifier, each sample image in the sample set is input into the combined strong classifier, and finally a positive sample image is output, which improves the detection accuracy and reduces the calculation time.

[0091] Furthermore, the above step 1) extracts the HOG features of each sample in the sample set, specifically including:

[0092] For each sample in the sample set, color space normalization processing, image gradient calculation, cell gradient histogram calculation, and histogram normalization processing are performed in sequence to determine the HOG feature of the sample.

[0093] Step 400, calculating the position information of the road pedestrian according to the current pedestrian characteristics; the road pedestrian position information includes the distance between the road pedestrian and the event camera. In a specific embodiment, the distance between the road pedestrian and the event camera can be replaced by the distance between the road pedestrian and the vehicle; the road pedestrian position information also includes the time required for the vehicle to travel to the position of the road pedestrian. In actual applications, the road pedestrian position information can be set as needed.

[0094] Step 500, determine whether the pedestrian position information is within the set safety area. Specifically, it is necessary to perform a risk assessment on the pedestrian position information to determine whether the pedestrian's current position is safe. Then send the judgment result to the electronic control system.

[0095] Step 600, if the road pedestrian position information is within the set safety area, that is, the current status of the road pedestrian is safe, then the current pedestrian features are tracked to determine the pedestrian features at the next moment, and then the current pedestrian features are updated to the pedestrian features at the next moment, and the step of calculating the road pedestrian position information based on the current pedestrian features is returned.

[0096] Specifically, the feature tracking of the current pedestrian feature is performed to determine the pedestrian feature at the next moment, specifically including:

[0097] 1) Input the current pedestrian features into the Kalman filter to calculate the pedestrian features according to the formula x t|t-1 =Ax t-1|t-1 +Bu t Calculate the pedestrian features at the next moment.

[0098] Among them, x represents the pedestrian feature state variable, x t|t-1 represents the pedestrian features at the next moment, that is, the prior estimate of the pedestrian features at time t, x t-1|t-1 represents the current pedestrian feature, A represents the state transfer matrix, B represents the gain matrix, u t represents the control variable at time t.

[0099] Specifically, the Kalman filter contains the state equation: x = Ax K-1 +Bu K +W K And the observation equation: y = Cx K +v K Among them, x K-1 is the system state at time K-1, u K is the control variable at time K, W is the system noise, y is the measurement value at time K, C is the observation matrix, and V is the observation noise.

[0100] According to the above state equation and observation equation, the state prediction equation for predicting the current moment based on the situation at the previous moment can be derived: t|t-1 =Ax t-1|t-1 +Bu t .

[0101] 2) Kalman filter will bring a certain amount of uncertainty in each prediction process. P is used to represent the uncertainty covariance, and Q is the noise brought by the prediction itself. According to the formula

[0102] x t|t =x t|t-1 +K t (Z t -Cx t|t-1 );

[0103] K t =P t|t-1 C T (CP t|t-1 C T ) -1 ;

[0104] P t|t =(IK t C)P t|t-1 ;

[0105] The pedestrian features at the next moment are corrected and optimized to obtain the optimized pedestrian features at the next moment; the optimized pedestrian features at the next moment are used to calculate the pedestrian position information on the road corresponding to the next moment. Further, when the state is updated through the Kalman filter, if the prediction model is trusted more, the weight should be set smaller, and vice versa; and the uncertainty of the prediction in the state update stage is reduced.

[0106] Among them, P t|t Represents the covariance matrix at time t, that is, the covariance matrix at the next moment, P t|t-1 represents the covariance matrix at time t-1, that is, the covariance matrix at the current time, P t|t-1 =AP t-1|t-1 A T +Q;A T represents the transpose of the state transfer matrix, Q represents noise, K t represents the Kalman gain, I represents the identity matrix, x t|t represents the optimized pedestrian features at time t, that is, the optimized pedestrian features at the next moment, Z t represents the pedestrian feature measurement value at time t, that is, the pedestrian feature measurement value at the next moment, Z t is to set y=Z t Substitute the formula y = Cx K +v K Calculated, C represents the observation matrix, C T represents the transpose of the observation matrix, (CP t|t- 1C T ) -1 Indicates (CP t|t-1 C T ) is the inverse matrix of .

[0107] Regardless of whether the pedestrian features at the next moment are directly obtained through the above feature tracking, or the pedestrian features at the next moment are further optimized to obtain the optimized pedestrian features at the next moment, the pedestrian features at the next moment and the optimized pedestrian features at the next moment need to be evaluated for danger again, that is, repeat the above steps 400 and 500 to determine whether the detected pedestrian is in a safe state. If the result is that the pedestrian is in a safe state, feature tracking continues, and the process is repeated to protect the pedestrian.

[0108] Step 700: If the pedestrian position information is not within the set safety area, that is, the pedestrian is currently unsafe, an emergency braking command is generated and sent to the AEB to perform automatic emergency braking to reduce the probability of a collision.

[0109] In summary, the present invention collects the outer frame of the dynamic object in front through the event camera, extracts the pedestrian features, screens out the target detected pedestrians through the classifier, conducts a danger assessment, and then judges whether the current situation and the next moment are safe based on the assessment results. If it is not safe, the signal is immediately transmitted to the automatic emergency braking system for emergency braking. Among them, the pedestrian detection results are transmitted to the Kalman filter, and the Kalman filter algorithm marks the detected pedestrians on the road to obtain their motion trajectory and detect their motion state in real time. The algorithm obeys the Gaussian distribution, and continuously updates and corrects the state variables through the predicted value of the previous moment and the current measured value to obtain more accurate predicted variables. Finally, whether it is the event camera, HOG pedestrian feature extraction, or Kalman filter pedestrian tracking and detection, it can achieve a faster response speed compared to the current technology.

[0110] Embodiment 2

[0111] like Figure 2 As shown, this embodiment provides a road pedestrian detection and tracking system based on an event camera, comprising:

[0112] The image acquisition module 101 is used to acquire target area information through an event camera and determine an image to be detected; the target area information includes pedestrians on the road.

[0113] The edge detection module 201 is used to perform edge detection processing on the image to be detected to determine preliminary image features. Specifically, the edge detection module includes a filter processing submodule, a pixel calculation submodule and a feature processing submodule.

[0114] The filtering processing submodule is used to perform Gaussian filtering on the image to be detected to obtain a first feature map; the pixel calculation submodule is used to use the Canny operator to calculate the gradient size and gradient direction of each pixel in the first feature map; the feature processing submodule is used to determine the preliminary image features according to the gradient size and gradient direction of each pixel, through non-maximum suppression processing and double threshold screening processing in sequence.

[0115] The feature extraction module 301 is used to input the preliminary image features into a feature extraction model to determine the current pedestrian features; the feature extraction model is obtained by training a cascade classifier according to a sample set; the cascade classifier is determined by integrating multiple Ada boost strong classifiers in series; the sample set includes multiple road pedestrian samples and multiple non-pedestrian samples.

[0116] The position calculation module 401 is used to calculate the position information of the road pedestrian according to the current pedestrian characteristics; the road pedestrian position information includes the distance between the road pedestrian and the event camera.

[0117] The position determination module 501 is used to determine whether the position information of the pedestrian on the road is within a set safety area.

[0118] The position tracking module 601 is used to track the current pedestrian features when the road pedestrian position information is within the set safety area to determine the pedestrian features at the next moment, then update the current pedestrian features to the pedestrian features at the next moment, and return to the position calculation module.

[0119] The emergency braking module 701 is used to generate an emergency braking instruction when the position information of the pedestrian on the road is not within the set safety area.

[0120] Compared with the prior art, the present invention also has the following advantages:

[0121] (1) The present invention processes a large number of samples through HOG feature extraction to extract pedestrian features, and uses Adaboost cascade classifier to continuously classify and screen according to HOG features to obtain the final detection result, which has strong reliability and relatively low environmental requirements.

[0122] (2) The event camera used in the present invention generates an event stream based on the change in light intensity on the surface of an object, which reduces a lot of invalid information, greatly reduces computing power and power consumption, and therefore has a higher response speed. At the same time, it has a high dynamic range, and the camera can maintain image clarity under extreme light intensity changes, which greatly improves the problems faced by traditional frame cameras. In addition, the fast response is of great help in solving the problem of pedestrian protection that suddenly appears in the blind spot of vision where there are vehicles or obstacles blocking the line of sight ahead, and can allow vehicles to take collision avoidance measures in advance to avoid the occurrence of traffic accidents or reduce the severity of traffic accidents.

[0123] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0124] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A road pedestrian detection and tracking method based on event camera, characterized in that: The road pedestrian detection and tracking method comprises: Collecting target area information through an event camera and determining an image to be detected; the target area information includes pedestrians on the road; Performing edge detection processing on the image to be detected to determine preliminary image features; The preliminary image features are input into a feature extraction model to determine the current pedestrian features; the feature extraction model is obtained by training a cascade classifier according to a sample set; the cascade classifier is determined by integrating multiple Adaboost strong classifiers in series; the sample set includes multiple road pedestrian samples and multiple non-pedestrian samples; Calculating the position information of the pedestrian on the road according to the current pedestrian characteristics; the position information of the pedestrian on the road includes the distance between the pedestrian on the road and the event camera; Determine whether the pedestrian position information on the road is within a set safety area; If the position information of the road pedestrian is within the set safety area, the feature tracking of the current pedestrian feature is performed to determine the pedestrian feature at the next moment, and then the current pedestrian feature is updated to the pedestrian feature at the next moment, and the step of calculating the position information of the road pedestrian according to the current pedestrian feature is returned; the feature tracking of the current pedestrian feature is performed to determine the pedestrian feature at the next moment, specifically including: inputting the current pedestrian feature into a Kalman filter to calculate the pedestrian feature at the next moment; If the pedestrian position information on the road is not within the set safety area, an emergency braking command is generated; The training process of the Adaboost strong classifier includes: Extracting the HOG features of each sample in the sample set; Performing weight initialization processing on all samples in the sample set to determine an initial weight corresponding to each sample; Perform multiple iterations of training based on the HOG features of each sample, and after each iteration of training, determine whether the sample error rate reaches a set value; the sample error rate is the probability that the sample is misclassified; If the sample error rate does not reach the set value, it is determined whether the current number of iterations reaches the iteration threshold. If the current number of iterations does not reach the iteration threshold, the initial weight corresponding to each sample is updated, and a larger initial weight is given to the wrongly classified samples. When the current number of iterations reaches the iteration threshold, the training ends. If the sample error rate reaches the set value, the training is terminated; the results of multiple iterative training corresponding to the sample error rate reaching the set value constitute the Adaboost strong classifier.

2. The method for detecting and tracking pedestrians on a road based on an event camera according to claim 1, characterized in that: The current pedestrian features are input into the Kalman filter to x t|t-1 =Ax t-1|t-1 +Bu t Calculate pedestrian features at the next moment; Among them, x represents the pedestrian feature state variable, x t|t-1 represents the pedestrian features at the next moment, x t-1|t-1 represents the current pedestrian feature, A represents the state transfer matrix, B represents the gain matrix, u t represents the control variable at time t.

3. The method for detecting and tracking pedestrians on a road based on an event camera according to claim 2, characterized in that: The road pedestrian detection and tracking method further includes: According to the formula x t|t =x t|t-1 +K t (Z t -Cx t|t-1 ) K t =P t|t-1 C T (CP t|t-1 C T ) -1 P t|t =(I-K t C)P t|t-1 Correcting and optimizing the pedestrian features at the next moment to obtain optimized pedestrian features at the next moment; the optimized pedestrian features at the next moment are used to calculate the road pedestrian position information corresponding to the next moment; Among them, P t|t Represents the covariance matrix at the next moment, P t|t-1 Represents the covariance matrix at the current moment, A T represents the transpose of the state transfer matrix, Q represents noise, K t represents the Kalman gain, I represents the identity matrix, x t|t Indicates the optimization of pedestrian features at the next moment, Z t represents the pedestrian feature measurement value at the next moment, C represents the observation matrix, C T represents the transpose of the observation matrix, (CP t|t-1 C T ) -1 Indicates (CP t|t-1 C T ) is the inverse matrix of .

4. The method for detecting and tracking pedestrians on a road based on an event camera according to claim 1, characterized in that: Performing edge detection processing on the image to be detected to determine preliminary image features, specifically including: Performing Gaussian filtering on the image to be detected to obtain a first feature map; Using the Canny operator to calculate the gradient magnitude and gradient direction of each pixel in the first feature map; According to the gradient size and gradient direction of each pixel, non-maximum suppression processing and double threshold screening processing are carried out in sequence to determine the preliminary image features.

5. The method for detecting and tracking pedestrians on a road based on an event camera according to claim 1, characterized in that: Extract the HOG features of each sample in the sample set, specifically including: For each sample in the sample set, color space normalization processing, image gradient calculation, cell gradient histogram calculation, and histogram normalization processing are performed in sequence to determine the HOG feature of the sample.

6. The method for detecting and tracking pedestrians on a road based on an event camera according to claim 1, characterized in that: The method of collecting target area information through an event camera and determining an image to be detected specifically includes: The target area information is collected through the event camera, and the light intensity change of each pixel is recorded to determine the image light intensity change; When the image light intensity variation is within a set light intensity threshold range, a plurality of pixel points corresponding to the image light intensity variation are output as an image to be detected.

7. A road pedestrian detection and tracking system based on event camera, characterized in that: The road pedestrian detection and tracking system comprises: An image acquisition module, used to acquire target area information through an event camera and determine an image to be detected; the target area information includes pedestrians on the road; An edge detection module is used to perform edge detection processing on the image to be detected to determine preliminary image features; A feature extraction module is used to input the preliminary image features into a feature extraction model to determine the current pedestrian features; the feature extraction model is obtained by training a cascade classifier according to a sample set; the cascade classifier is determined by integrating multiple Adaboost strong classifiers in series; the sample set includes multiple road pedestrian samples and multiple non-pedestrian samples; A position calculation module, used to calculate the position information of the road pedestrian according to the current pedestrian characteristics; the road pedestrian position information includes the distance between the road pedestrian and the event camera; A position determination module is used to determine whether the position information of the pedestrian on the road is within a set safety area; The position tracking module is used to perform feature tracking on the current pedestrian features to determine the pedestrian features at the next moment when the position information of the road pedestrian is within the set safety area, and then update the current pedestrian features to the pedestrian features at the next moment, and return to the position calculation module; the feature tracking on the current pedestrian features to determine the pedestrian features at the next moment specifically includes: inputting the current pedestrian features into a Kalman filter to calculate the pedestrian features at the next moment; An emergency braking module, used for generating an emergency braking instruction when the position information of the pedestrian on the road is not within the set safety area; The training process of the Adaboost strong classifier includes: Extracting the HOG features of each sample in the sample set; Performing weight initialization processing on all samples in the sample set to determine an initial weight corresponding to each sample; Perform multiple iterations of training based on the HOG features of each sample, and after each iteration of training, determine whether the sample error rate reaches a set value; the sample error rate is the probability that the sample is misclassified; If the sample error rate does not reach the set value, it is determined whether the current number of iterations reaches the iteration threshold. If the current number of iterations does not reach the iteration threshold, the initial weight corresponding to each sample is updated, and a larger initial weight is given to the wrongly classified samples. When the current number of iterations reaches the iteration threshold, the training ends. If the sample error rate reaches the set value, the training is terminated; the results of multiple iterative training corresponding to the sample error rate reaching the set value constitute the Adaboost strong classifier.

8. The road pedestrian detection and tracking system based on event camera according to claim 7, characterized in that: The edge detection module specifically includes: A filtering processing submodule, used for performing Gaussian filtering on the image to be detected to obtain a first feature map; A pixel calculation submodule, used to calculate the gradient magnitude and gradient direction of each pixel in the first feature map using a Canny operator; The feature processing submodule is used to determine the preliminary image features through non-maximum suppression processing and double threshold screening processing in sequence according to the gradient size and gradient direction of each pixel.

Images