Event camera target detection method and device based on bounding box memory

By obtaining the crossover and comparison of the front and back event frames and bounding boxes in the event camera, determining the reason for the target disappearance, the problem that the event camera cannot detect a stationary or slow target is solved, and low-memory and efficient target detection is achieved.

CN119006801AActive Publication Date: 2024-11-22NAT UNIV OF DEFENSE TECH
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202411465407.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-21
Publication Date
2024-11-22
Estimated Expiration
2044-10-21

AI Technical Summary

Technical Problem

Event cameras cannot detect targets that are relatively stationary or slow to move as the camera, because in this case the event camera does not produce output, and existing methods require designing complex network structures and consuming a large amount of computing resources.

Method used

By obtaining the two event frames before and after when the moving target disappears from the event camera, the intersection ratio between bounding boxes is calculated, the confusion matrix and preset judgment rules are used to determine the cause of disappearance, and under conditions without training and low memory occupancy is low. Complete the object detection.

Benefits of technology

It effectively solves the problem that event cameras cannot detect slow motion or relatively stationary targets, realizes object detection with low memory footprint and no training required, and improves detection accuracy and efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119006801A_ABST
    Figure CN119006801A_ABST
Patent Text Reader

Abstract

The invention discloses an event camera target detection method and device based on bounding box memory, and relates to the technical field of image data processing, and the method comprises the steps: obtaining a front event frame and a rear event frame corresponding to a moving target from an event camera when the moving target disappears from the event camera; based on a preset target detection algorithm and the two event frames before and after, obtaining detection results corresponding to a first event frame and a second event frame in the two event frames before and after; the first event frame is a previous event frame of the second event frame; calculating an intersection-to-union ratio between bounding boxes in the first event frame and the second event frame based on each detection result, and obtaining a confusion matrix; and performing a bounding box judgment process through the confusion matrix and a preset judgment rule, determining a disappearance cause corresponding to the moving target, and completing a target detection operation corresponding to the moving target. Under the conditions that training is not needed and the memory occupancy rate is low, the problem that an event camera cannot detect a moving target which moves slowly or is relatively static is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image data processing, and in particular to an event camera target detection method and device based on bounding box memory. Background Art

[0002] Event cameras are a new type of vision-based sensor that asynchronously captures the instantaneous changes of each pixel. Specifically, when a pixel When the intensity change exceeds the threshold C, the pixel outputs an event point , Indicates the timestamp, is the polarity, indicating whether the light intensity at the pixel is increasing or decreasing. However, unlike frame-based cameras, event cameras capture the motion of the scene. When the event camera is relatively stationary with respect to the moving target, the event camera will not produce any output, which results in event-based object detection being unable to detect targets that are relatively stationary or moving slowly with respect to the camera.

[0003] At present, in order to solve this problem, existing methods mainly use LSTM (Long Short Term Memory) to extract time domain context information to memorize the contextual features of the target. However, this method requires the design of a complex network structure and takes up a lot of computing resources. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide an event camera target detection method and device based on bounding box memory, which can effectively solve the problem that the event camera cannot detect slow-moving or relatively static moving targets without the need for training and with low memory usage. The specific scheme is as follows:

[0005] In a first aspect, the present application provides an event camera target detection method based on bounding box memory, comprising:

[0006] When the moving target disappears from the event camera, obtaining two event frames corresponding to the moving target from the event camera;

[0007] Based on a preset target detection algorithm and the two event frames, a detection result corresponding to a first event frame and a second event frame of the two event frames is obtained; wherein the first event frame is a previous event frame of the second event frame;

[0008] Based on each of the detection results, calculating the intersection-over-union ratio between the bounding boxes in the first event frame and the second event frame, and obtaining a corresponding confusion matrix;

[0009] The corresponding bounding box judgment process is performed through the confusion matrix and the preset judgment rules to determine the disappearance reason corresponding to the moving target, and complete the target detection operation corresponding to the moving target.

[0010] Optionally, acquiring two preceding and following event frames corresponding to the moving target includes:

[0011] By projecting all event data within a preset time window for each pixel value within a corresponding time period, two event frames before and after corresponding to the moving target are obtained.

[0012] Optionally, the detection result includes the corresponding number of bounding boxes and corresponding bounding box coordinate information.

[0013] Optionally, the corresponding bounding box judgment process is performed using the confusion matrix and preset judgment rules, including:

[0014] Obtaining the maximum intersection-over-union ratio corresponding to each row in the confusion matrix;

[0015] Determine whether each of the maximum intersection-and-union ratios is less than a preset intersection-and-union ratio threshold, and obtain a corresponding first determination result;

[0016] When there is a first judgment result indicating that the value is less than the preset intersection-over-union ratio threshold, calculating the average number of event points in the corresponding bounding box to obtain a corresponding calculation result;

[0017] It is determined whether the calculation result is less than a preset average event point quantity threshold value to obtain a corresponding second determination result.

[0018] Optionally, after obtaining the corresponding first judgment result, the method further includes:

[0019] When all the first judgment results indicate that the value is not less than the preset intersection-and-joint ratio threshold, the process jumps again to the step of acquiring the two preceding and succeeding event frames corresponding to the moving target from the event camera.

[0020] Optionally, after obtaining the corresponding second judgment result, the method further includes:

[0021] When each of the second judgment results indicates that the number is not less than the preset average event point number threshold, the process jumps again to the step of acquiring the two preceding and following event frames corresponding to the moving target from the event camera.

[0022] Optionally, when there is the first judgment result indicating that the value is less than the preset intersection-over-combination ratio threshold, the method further includes:

[0023] When there is a first judgment result indicating that the value is less than the preset intersection-over-union ratio threshold, the corresponding bounding box coordinate information corresponding to the first event frame is saved in a preset bounding box memory library.

[0024] Optionally, the determining the disappearance reason corresponding to the moving target includes:

[0025] When the second judgment result indicates that the number of event points is less than the preset average event point number threshold, it is determined that the disappearance reason corresponding to the moving target is that the moving target and the event camera are relatively still or the moving target moves slowly.

[0026] Optionally, the completing the target detection operation corresponding to the moving target includes:

[0027] The corresponding information supplementation operation is completed by adding the corresponding bounding box coordinate information to the detection result corresponding to the second event frame, so as to complete the target detection operation corresponding to the moving target based on the supplemented detection result corresponding to the second event frame.

[0028] In a second aspect, the present application provides an event camera target detection device based on bounding box memory, comprising:

[0029] An event frame acquisition module, used for acquiring two preceding and following event frames corresponding to the moving target from the event camera when the moving target disappears from the event camera;

[0030] An event frame detection module, configured to obtain detection results corresponding to a first event frame and a second event frame of the two event frames based on a preset target detection algorithm and the two event frames; wherein the first event frame is a previous event frame of the second event frame;

[0031] A confusion matrix acquisition module, configured to calculate the intersection-and-union ratio between the bounding boxes in the first event frame and the second event frame based on the detection results, and obtain a corresponding confusion matrix;

[0032] The bounding box judgment module is used to perform a corresponding bounding box judgment process through the confusion matrix and preset judgment rules, determine the disappearance reason corresponding to the moving target, and complete the target detection operation corresponding to the moving target.

[0033] It can be seen that in the present application, when a moving target disappears from an event camera, two event frames before and after corresponding to the moving target are obtained from the event camera; based on a preset target detection algorithm and the two event frames before and after, detection results corresponding to the first event frame and the second event frame in the two event frames before and after are obtained; wherein the first event frame is the previous event frame of the second event frame; based on each of the detection results, the intersection-and-union ratio between the bounding boxes in the first event frame and the second event frame is calculated, and a corresponding confusion matrix is ​​obtained; the corresponding bounding box judgment process is performed through the confusion matrix and the preset judgment rule, the disappearance reason corresponding to the moving target is determined, and the target detection operation corresponding to the moving target is completed. That is, when a moving target disappears from an event camera, the present application obtains two event frames before and after corresponding to the moving target from the event camera, and performs bounding box intersection-and-union ratio calculation and bounding box judgment process by using the detection results obtained by target detection, determines the disappearance reason corresponding to the moving target, and completes the target detection operation corresponding to the moving target. In this way, the problem that event cameras cannot detect slow-moving or relatively still moving targets can be effectively solved without the need for training and with low memory usage. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0035] Figure 1 A flow chart of an event camera target detection method based on bounding box memory provided in this application;

[0036] Figure 2 A schematic diagram of the principle of an event camera provided in this application;

[0037] Figure 3 A schematic diagram of a calculation flow of a bounding box memory module provided in this application;

[0038] Figure 4 A specific visualization diagram of the test results provided in this application;

[0039] Figure 5 A structural diagram of an event camera target detection device based on bounding box memory provided in this application. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] Event cameras are a new type of vision-based sensor that asynchronously captures the instantaneous changes of each pixel. Specifically, when a pixel When the intensity change exceeds the threshold C, the pixel outputs an event point , Indicates the timestamp, is the polarity, indicating whether the light intensity at the pixel is increasing or decreasing. However, unlike frame-based cameras, event cameras capture the movement of the scene. When the event camera is relatively stationary with the moving target, the event camera will not produce any output, which results in the inability of event-based target detection to detect targets that are relatively stationary or moving slowly with the camera. At present, in order to solve this problem, existing methods mainly use LSTM to extract time domain context information to memorize the context features of the target. However, this method requires the design of a complex network structure and requires a lot of computing resources.

[0042] To this end, the present application provides an event camera target detection solution based on bounding box memory, which can effectively solve the problem that event cameras cannot detect slow-moving or relatively stationary moving targets without the need for training and with low memory usage.

[0043] See also Figure 1 As shown, an embodiment of the present invention discloses an event camera target detection method based on bounding box memory, comprising:

[0044] Step S11: when a moving target disappears from the event camera, two preceding and following event frames corresponding to the moving target are acquired from the event camera.

[0045] Specific, combined Figure 2 As shown in the figure, when there is no motion in the scene as shown in the Traditional camera output, the Event camera output is interrupted, that is, the event camera has no output. To solve this problem, considering that for event stream object detection, once the detector successfully detects an object, it is impossible for the object to disappear from the area without any event generation. Therefore, based on the detection results of the previous frame and the number of events in the bounding box of the next frame, it can be determined whether there is a stationary object in the area without events.

[0046] Specifically, in this embodiment, when the moving target disappears from the event camera, two consecutive frames of event data are acquired from the event camera, from t to All event data in are represented as event data t, from arrive All event data in the time window are represented as event data t+1. Event data t and t+1 are projected into event frames. The projection process is: the value of the pixel (x, y) (where x and y are the horizontal and vertical coordinates of the pixel, respectively) is in the time window That is, obtaining the two event frames before and after the moving target includes: obtaining the two event frames before and after the moving target by projecting all event data whose pixel values ​​are within a preset time window in the corresponding time period.

[0047] Step S12, based on a preset target detection algorithm and the two event frames, obtaining detection results corresponding to the first event frame and the second event frame of the two event frames; wherein the first event frame is the previous event frame of the second event frame.

[0048] Specifically, in this embodiment, Yolov5 can be used to detect the event frame. The detection result of the first event frame, frame t, is The detection result of the second event frame, frame t+1, is Where I and J are the total number of bounding boxes in the two frames, represents the coordinates of the upper left and upper right corners of the rectangular bounding box, x1 and y1 represent the horizontal and vertical coordinates of the point in the upper left corner, and x2 and y2 represent the horizontal and vertical coordinates of the point in the upper right corner. That is, the detection result includes the corresponding number of bounding boxes and the corresponding bounding box coordinate information.

[0049] Step S13: Based on the detection results, the intersection-over-union ratio between the bounding boxes in the first event frame and the second event frame is calculated to obtain a corresponding confusion matrix.

[0050] Combination Figure 3 It can be seen that after obtaining the detection results, the intersection over union (IOU) between the bounding boxes of the two frames is calculated for the subsequent judgment process. The specific calculation formula is as follows.

[0051] ;

[0052] ;

[0053] Where M iou is the obtained confusion matrix; is the intersection-over-union ratio between the i-th bounding box in the detection result of event frame t and the j-th bounding box in the detection result of event frame t+1; is the area of ​​the overlap (intersection) of bounding box A and bounding box B, is the area of ​​the union of the two bounding boxes.

[0054] Step S14: Perform a corresponding bounding box judgment process through the confusion matrix and preset judgment rules to determine the disappearance reason corresponding to the moving target, and complete the target detection operation corresponding to the moving target.

[0055] Specifically, in this embodiment, combined with Figure 3 As shown, after obtaining the confusion matrix, it is necessary to calculate each row (m iou ) , which can be obtained by using (That is, Figure 3 maxIOU(i) in the formula) and the corresponding preset intersection-over-union threshold Perform a specific judgment operation. That is, the corresponding bounding box judgment process is performed through the confusion matrix and the preset judgment rule, including: obtaining the maximum intersection-and-union ratio corresponding to each row in the confusion matrix; judging whether each of the maximum intersection-and-union ratios is less than a preset intersection-and-union ratio threshold, and obtaining a corresponding first judgment result; when there is a first judgment result indicating that it is less than the preset intersection-and-union ratio threshold, calculating the average number of event points in the corresponding bounding box to obtain a corresponding calculation result; judging whether the calculation result is less than the preset average number of event points threshold It should be understood that when there is a first judgment result indicating that the value is less than the preset intersection-over-combination ratio threshold, it is determined that the corresponding bounding box disappears in the next frame and is added to the bounding box memory library. That is, when there is the first judgment result indicating that the value is less than the preset intersection-and-joint ratio threshold, the method further includes: when there is the first judgment result indicating that the value is less than the preset intersection-and-joint ratio threshold, saving the corresponding bounding box coordinate information corresponding to the first event frame into a preset bounding box memory library.

[0056] At this point, there may be two reasons why the target disappears. One is that the target moves faster and moves out of the bounding box, and the other is that the target is relatively still with the camera, and the event camera has no output, causing the network to fail to detect the target. Therefore, it is necessary to further calculate the average number of events within each boundary to determine the specific reason for the target disappearance. The specific calculation formula is as follows.

[0057] ;

[0058] In the formula, is the number of points (i.e. events) within the bounding box; is the area of ​​the bounding box. If the average number of events in the bounding box is less than the threshold , it can be considered that the disappearance of the bounding box is due to too few events, and the disappearance reason corresponding to the moving target is determined to be that the moving target is relatively still with the event camera or the moving target moves slowly. Afterwards, the target detection operation corresponding to the moving target is completed, including:

[0059] By adding the corresponding bounding box coordinate information to the detection result corresponding to the second event frame, the corresponding information supplementation operation is completed, so as to complete the target detection operation corresponding to the moving target based on the supplemented detection result corresponding to the second event frame. That is, these bounding boxes are added to the detection result of the next frame. This process can be described as: .

[0060] In addition, in this embodiment, after obtaining the corresponding first judgment result, the method further includes: when each of the first judgment results indicates that the value is not less than the preset intersection-and-parallel ratio threshold, the method further includes: when each of the first judgment results indicates that the value is not less than the preset intersection-and-parallel ratio threshold, the method further includes: when each of the second judgment results indicates that the value is not less than the preset average event point number threshold, the method further includes: when each of the second judgment results indicates that the value is not less than the preset average event point number threshold, the method further includes: when each of the second judgment results indicates that the value is not less than the preset average event point number threshold, the method further includes: when each of the event cameras indicates that the value is not less than the preset average event point number threshold, the method further includes:

[0061] In summary, this embodiment proposes an event camera target detection method based on bounding box memory, which solves the problem that the existing event camera cannot detect slow-moving or stationary targets by fusing the time domain context information of the detection results.

[0062] It can be seen that in the embodiment of the present application, when the moving target disappears from the event camera, the two event frames before and after corresponding to the moving target are obtained from the event camera; based on the preset target detection algorithm and the two event frames before and after, the detection results corresponding to the first event frame and the second event frame in the two event frames before and after are obtained; wherein the first event frame is the previous event frame of the second event frame; based on each of the detection results, the intersection and union ratio between the bounding boxes in the first event frame and the second event frame is calculated, and a corresponding confusion matrix is ​​obtained; the corresponding bounding box judgment process is performed through the confusion matrix and the preset judgment rule, the disappearance reason corresponding to the moving target is determined, and the target detection operation corresponding to the moving target is completed. That is, when the moving target disappears from the event camera, the present application obtains the two event frames before and after corresponding to the moving target from the event camera, and performs the bounding box intersection and union ratio calculation and the bounding box judgment process by using the detection results obtained by the target detection, determines the disappearance reason corresponding to the moving target, and completes the target detection operation corresponding to the moving target. In this way, the problem that event cameras cannot detect slow-moving or relatively still moving targets can be effectively solved without the need for training and with low memory usage.

[0063] Combine the following Figure 3 , Figure 4 The schematic diagram disclosed in the figure specifically illustrates the technical solution of the embodiment of the present application.

[0064] Taking car detection as an example, in this embodiment, experiments are conducted on the GEN1 car detection dataset (GEN1 dataset) and the Prophesee 1Mpx car detection dataset (1 Mpx detection dataset). More precisely, the GEN1 dataset consists of 39 hours of event data (resolution 304×240) recorded by an ATIS (Asynchronous Time Based Image Sensor) camera, containing manual bounding box annotations for 226,719 cars and 27,658 pedestrians. The 1 Mpx detection dataset provides an event stream with a resolution of 1280×720, containing more than 25 million labeled bounding boxes for cars, pedestrians, and two-wheelers, annotated at a frequency of 60 Hz, and providing 11.19 hours of training records, 2.21 hours of validation records, and 2.25 hours of test records.

[0065] The detection performance is evaluated by mAP (mean Average Precision). mAP is the average of AP (Average Precision) values ​​of all categories. AP is the area calculated based on the PR curve (Precision-Recall Curve, a curve drawn with recall as the horizontal axis and precision as the vertical axis). It interpolates the Precision under different recall rates and calculates the area under the interpolation curve. AP is a comprehensive performance indicator for evaluating target detection algorithms under different recall rates.

[0066] Among them, Precision measures how many of all samples detected as positive samples are truly positive samples. The calculation formula of Precision is: Precision=TP / (TP+FP), where TP (True Positives) is the number of true positive samples, and FP (False Positives) is the number of negative samples mistakenly marked as positive samples. A higher Precision means that the algorithm is more accurate in determining positive samples. Recall measures how many of all true positive samples are correctly detected by the algorithm. The calculation formula of Recall is: Recall=TP / (TP+FN), where FN (False Negatives) is the number of positive samples that were mistakenly not detected. A higher Recall means that the algorithm can better detect the real target.

[0067] We first evaluated different bounding box memory module thresholds , Regarding the impact on algorithm performance, YOLOv5 is used as the target detector to detect targets in the scene, and then different thresholds are used and threshold To evaluate the performance of the algorithm, the results are shown in Table 1 and Table 2. As can be seen from the table, when the intersection-union ratio threshold is preset Set to 0.9, preset the average event point number threshold When it is set to 0.8, the algorithm achieves the best performance.

[0068] Table 1

[0069]

[0070] Table 2

[0071]

[0072] Secondly, the detection results of the bounding box memory module are visualized, such as Figure 4As shown in the figure, the previous line is the result of single-frame target detection, and the second line is the detection result after applying the bounding box memory module. It can be seen that when the target is relatively still with the event camera, no event data is generated in the target area, and the target detector cannot detect the target (as shown in the t+2 frame and t+n frame in the first row). After applying the bounding box memory module, based on the detection results of the t frame and the t+1 frame, the algorithm can supplement the target (the supplemented detection results are indicated by a red dotted box in the figure), thereby detecting a stationary target.

[0073] Combination Figure 5 As shown, the embodiment of the present application also discloses an event camera target detection device based on bounding box memory, including:

[0074] The event frame acquisition module 11 is used to acquire the preceding and following event frames corresponding to the moving target from the event camera when the moving target disappears from the event camera;

[0075] An event frame detection module 12 is used to obtain detection results corresponding to a first event frame and a second event frame of the two event frames based on a preset target detection algorithm and the two event frames; wherein the first event frame is a previous event frame of the second event frame;

[0076] A confusion matrix acquisition module 13, configured to calculate the intersection-and-union ratio between the bounding boxes in the first event frame and the second event frame based on the detection results, and obtain a corresponding confusion matrix;

[0077] The bounding box judgment module 14 is used to perform a corresponding bounding box judgment process through the confusion matrix and preset judgment rules, determine the disappearance reason corresponding to the moving target, and complete the target detection operation corresponding to the moving target.

[0078] Among them, for more specific working processes of the above-mentioned modules, please refer to the corresponding contents disclosed in the aforementioned embodiments, which will not be repeated here.

[0079] In summary, when a moving target disappears from an event camera, the present application obtains the two event frames corresponding to the moving target from the event camera, and uses the detection results obtained by target detection to perform bounding box intersection and union calculation and bounding box judgment process to determine the disappearance reason corresponding to the moving target, and completes the target detection operation corresponding to the moving target. In this way, the problem that the event camera cannot detect slow-moving or relatively still moving targets can be effectively solved without the need for training and with low memory usage.

[0080] In some specific embodiments, the event frame acquisition module 11 may specifically include:

[0081] The event frame acquisition unit is used to obtain two event frames corresponding to the moving target by projecting all event data whose pixel values ​​are within a preset time window within a corresponding time period.

[0082] In some specific embodiments, the bounding box determination module 14 may specifically include:

[0083] An intersection-and-union ratio acquisition unit, used to acquire a maximum intersection-and-union ratio corresponding to each row in the confusion matrix;

[0084] An I / O ratio judgment unit, used to judge whether each of the maximum I / O ratios is less than a preset I / O ratio threshold value, and obtain a corresponding first judgment result;

[0085] A first judgment result processing unit is used to calculate the average number of event points in the corresponding bounding box when there is a first judgment result indicating that the value is less than the preset intersection-over-union ratio threshold, so as to obtain a corresponding calculation result;

[0086] The calculation result judgment unit is used to judge whether the calculation result is less than a preset average event point quantity threshold value to obtain a corresponding second judgment result.

[0087] In some specific embodiments, the bounding box determination module 14 may further include:

[0088] The second judgment result processing unit is used to jump back to the step of acquiring the two preceding and following event frames corresponding to the moving target from the event camera when all the first judgment results indicate that the value is not less than the preset intersection-and-joint ratio threshold.

[0089] In some specific embodiments, the bounding box determination module 14 may further include:

[0090] The third judgment result processing unit is used to jump back to the step of acquiring the two preceding and following event frames corresponding to the moving target from the event camera when each of the second judgment results indicates that the number of event points is not less than the preset average event point number threshold.

[0091] In some specific embodiments, the bounding box determination module 14 may further include:

[0092] The information storage unit is used to store the corresponding bounding box coordinate information corresponding to the first event frame in a preset bounding box memory library when there is the first judgment result indicating that the first judgment result is less than the preset intersection-over-combination ratio threshold.

[0093] In some specific embodiments, the bounding box determination module 14 may specifically include:

[0094] The fourth judgment result processing unit is used to determine that the disappearance reason corresponding to the moving target is that the moving target is relatively still with the event camera or the moving target moves slowly when the second judgment result indicates that the number of event points is less than the preset average event point threshold.

[0095] In some specific embodiments, the bounding box determination module 14 may specifically include:

[0096] An information supplement unit is used to complete a corresponding information supplement operation by adding corresponding bounding box coordinate information to the detection result corresponding to the second event frame, so as to complete a target detection operation corresponding to the moving target based on the supplemented detection result corresponding to the second event frame.

[0097] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0098] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0099] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0100] The technical solution provided by the present application is introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technicians in this field, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. An event camera target detection method based on bounding box memory, characterized in that: include: When the moving target disappears from the event camera, obtaining two event frames corresponding to the moving target from the event camera; Based on a preset target detection algorithm and the two event frames, a detection result corresponding to a first event frame and a second event frame of the two event frames is obtained; wherein the first event frame is a previous event frame of the second event frame; Based on each of the detection results, calculating the intersection-over-union ratio between the bounding boxes in the first event frame and the second event frame, and obtaining a corresponding confusion matrix; The corresponding bounding box judgment process is performed through the confusion matrix and the preset judgment rules to determine the disappearance reason corresponding to the moving target, and complete the target detection operation corresponding to the moving target.

2. The event camera target detection method based on bounding box memory according to claim 1, characterized in that: The acquiring of two preceding and following event frames corresponding to the moving target comprises: By projecting all event data within a preset time window for each pixel value within a corresponding time period, two event frames before and after corresponding to the moving target are obtained.

3. The event camera target detection method based on bounding box memory according to claim 1, characterized in that: The detection result includes the corresponding number of bounding boxes and corresponding bounding box coordinate information.

4. The event camera target detection method based on bounding box memory according to any one of claims 1 to 3, characterized in that: The corresponding bounding box judgment process is performed using the confusion matrix and preset judgment rules, including: Obtaining the maximum intersection-over-union ratio corresponding to each row in the confusion matrix; Determine whether each of the maximum intersection-and-union ratios is less than a preset intersection-and-union ratio threshold, and obtain a corresponding first determination result; When there is a first judgment result indicating that the value is less than the preset intersection-over-union ratio threshold, calculating the average number of event points in the corresponding bounding box to obtain a corresponding calculation result; It is determined whether the calculation result is less than a preset average event point quantity threshold value to obtain a corresponding second determination result.

5. The event camera target detection method based on bounding box memory according to claim 4, characterized in that: After obtaining the corresponding first judgment result, the method further includes: When all the first judgment results indicate that the value is not less than the preset intersection-and-joint ratio threshold, the process jumps again to the step of acquiring the two preceding and succeeding event frames corresponding to the moving target from the event camera.

6. The event camera target detection method based on bounding box memory according to claim 4, characterized in that: After obtaining the corresponding second judgment result, the method further includes: When each of the second judgment results indicates that the number is not less than the preset average event point number threshold, the process jumps again to the step of acquiring the two preceding and following event frames corresponding to the moving target from the event camera.

7. The event camera target detection method based on bounding box memory according to claim 4, characterized in that: When there is the first judgment result indicating that the value is less than the preset intersection-over-combination ratio threshold, the method further includes: When there is a first judgment result indicating that the value is less than the preset intersection-over-union ratio threshold, the corresponding bounding box coordinate information corresponding to the first event frame is saved in a preset bounding box memory library.

8. The event camera target detection method based on bounding box memory according to claim 4, characterized in that: The determining of the disappearance reason corresponding to the moving target includes: When the second judgment result indicates that the number of event points is less than the preset average event point number threshold, it is determined that the disappearance reason corresponding to the moving target is that the moving target and the event camera are relatively still or the moving target moves slowly.

9. The event camera target detection method based on bounding box memory according to claim 8, characterized in that: The completing the target detection operation corresponding to the moving target includes: The corresponding information supplementation operation is completed by adding the corresponding bounding box coordinate information to the detection result corresponding to the second event frame, so as to complete the target detection operation corresponding to the moving target based on the supplemented detection result corresponding to the second event frame.

10. An event camera target detection device based on bounding box memory, characterized in that: include: An event frame acquisition module, used for acquiring two preceding and following event frames corresponding to the moving target from the event camera when the moving target disappears from the event camera; An event frame detection module, configured to obtain detection results corresponding to a first event frame and a second event frame of the two event frames based on a preset target detection algorithm and the two event frames; wherein the first event frame is a previous event frame of the second event frame; A confusion matrix acquisition module, configured to calculate the intersection-and-union ratio between the bounding boxes in the first event frame and the second event frame based on the detection results, and obtain a corresponding confusion matrix; The bounding box judgment module is used to perform a corresponding bounding box judgment process through the confusion matrix and preset judgment rules, determine the disappearance reason corresponding to the moving target, and complete the target detection operation corresponding to the moving target.

Citation Information

Patent Citations

  • Street front order event video detection method based on deep learning and motion consistency

    CN108304798A

  • Unmanned aerial vehicle target detection method based on event camera

    CN113762409A

  • Unmanned aerial vehicle high-dynamic target detection method fused with event camera

    CN115761548A

  • Moving target detection method and system based on event data and image data combination

    CN116416602A

  • Automatic driving event camera target detection method based on cross-modal noise perception

    CN116912798A