Method and device for dynamic high-speed target tracking based on an event camera

Abstract

The present invention discloses a dynamic high-speed target tracking method and device based on an event camera. The method includes: obtaining a light brightness transformation map in a high-speed motion scene collected by the event camera and parsing it into an event frame image set; performing target association matching on the current frame image and the previous frame image according to multiple targets recognized by a second detection box in the event frame image set to determine multiple tracked targets; obtaining the predicted position and the actual position of the tracked targets in the next frame image, and determining whether the tracked targets are the same tracking target according to the error magnitude between the predicted position and the actual position to obtain a tracking result; and obtaining and displaying the tracking result in real time. This solution uses the event camera to capture high-speed moving targets and processes the original data of the event camera, so as to effectively detect and continuously track high-speed targets through target recognition detection and target tracking.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular, to a method and device for dynamic high-speed target tracking based on an event camera. Background Art

[0002] With the rapid development of artificial intelligence, target tracking technology has received extensive attention in various fields, such as vehicle tracking in the transportation field. Through target tracking technology, the movement trend of the tracking target can be understood in a timely manner. Therefore, how to achieve timely and effective target tracking is an issue worthy of attention in the field of target tracking.

[0003] In the prior art, target tracking is to continuously track the target in the video captured by the camera using intelligent algorithms. Specifically, based on the input image frame sequence along the time axis, the target features in the image are extracted based on a convolutional neural network, and it is judged whether it is the target to be followed by feature comparison. However, traditional cameras are limited by their imaging principle and chip architecture, and do not have characteristics such as high-speed response and large dynamic range imaging compared with the human eye, resulting in their limited use in some complex scenarios. For example, when the target object moves at high speed, the imaging characteristics based on time integration of traditional cameras will cause blurred imaging; or in a scene with large dynamic illumination, due to its small dynamic range, usually only 60dB, it is unable to obtain effective exposure of both bright and dark areas at the same time. Moreover, traditional cameras image global information, resulting in a lot of meaningless background redundant information.

[0004] Although event cameras can effectively capture high-speed moving targets, since the pixels of event cameras are independently and asynchronously distributed, there are great differences from traditional image frames. Currently, traditional artificial intelligence algorithms cannot fully adapt to this data form.

[0005] Due to the problems that traditional cameras will have afterimages when imaging high-speed and ultra-high-speed targets, or cannot obtain effective target information, and the data of event cameras cannot be directly combined with existing artificial intelligence algorithms, the existing target tracking methods cannot effectively detect and continuously track targets. Summary of the Invention

[0006] The present invention provides a method and device for dynamic high-speed target tracking based on an event camera, which uses the event camera to capture high-speed moving targets and processes the original data of the event camera, solving the problem that the existing target tracking methods cannot effectively detect and continuously track targets.

[0007] To achieve the above object, the present invention adopts the following technical solutions:

[0008] In a first aspect, the present invention provides a method for dynamic high-speed target tracking based on an event camera, the method comprising:

[0009] Obtain the light brightness transformation map in the high-speed motion scene collected by the event camera, and parse the light brightness transformation map into an event frame image set; the event frame image set includes a plurality of grayscale images distributed in chronological order;

[0010] Extract image features and perform classification and regression processing on the grayscale images in the event frame image set through a convolutional neural network to obtain a plurality of first detection frames for identifying different category targets; the first detection frame includes the detection frame position and the detection frame confidence;

[0011] Based on the non-maximum suppression algorithm and the detection frame confidence of each first detection frame, remove the first detection frames with the overlapping degree meeting the first preset threshold to obtain the second detection frames;

[0012] According to the multiple targets identified by the second detection frames in the current frame image and the previous frame image in the event frame image set, perform target association and matching on the current frame image and the previous frame image to determine a plurality of tracked targets;

[0013] According to the tracked targets in the current frame image and the previous frame image, obtain the predicted position of the tracked targets in the next frame image, obtain the actual position of the tracked targets in the next frame image through the second detection frame recognition, and determine whether the tracked targets are the same tracking target according to the error size between the predicted position and the actual position to obtain the tracking result; the tracking result includes the actual positions and identification numbers of the same tracking target in the previous frame image, the current frame image, and the next frame image;

[0014] Obtain and display the tracking result in real time.

[0015] In a possible implementation manner, the parsing of the light brightness transformation map into an event frame image set specifically includes:

[0016] Obtain the grayscale transformation information of multiple light brightness transformation maps based on the differential imaging principle; each grayscale transformation information includes the pixel coordinate values, grayscale values, and trigger times of the trigger positions of multiple events;

[0017] Obtain the corresponding grayscale matrix according to the grayscale transformation information; each grayscale matrix is used to describe a grayscale image, and a plurality of grayscale images distributed in chronological order form an event frame image set.

[0018] In a possible implementation manner, the obtaining of the corresponding grayscale matrix according to the grayscale transformation information specifically includes:

[0019] Construct a grayscale matrix according to the resolution of the event camera, and assign an initial value of zero to each element of the grayscale matrix;

[0020] Traverse all events in each of the light intensity transformation maps, and assign the grayscale value in the grayscale change information to each element in the grayscale matrix corresponding to the pixel coordinate value of the trigger position.

[0021] In a possible implementation manner, after constructing a grayscale matrix according to the resolution of the event camera and assigning an initial value of zero to each element of the grayscale matrix, the method further includes:

[0022] Judge whether the value of each element of the grayscale matrix is less than the corresponding grayscale value in the grayscale transformation information;

[0023] If so, assign a polarity of -1 to the element; if not, assign a polarity of +1 to the element.

[0024] In a possible implementation manner, based on the non-maximum suppression algorithm and the detection box confidence of each of the first detection boxes, remove the first detection boxes whose overlap degree meets the first preset threshold, specifically including:

[0025] For the first detection boxes of the same category target, obtain the first detection box with the largest detection box confidence;

[0026] Calculate the ratio of the intersection to the union of the first detection box with the largest detection box confidence and the remaining first detection boxes of the category target respectively; use the ratio to represent the overlap degree between the first detection boxes;

[0027] Delete the remaining first detection boxes when the ratio is greater than the first preset threshold according to the size of the ratio and the first preset threshold.

[0028] In a possible implementation manner, the target association and matching of the current frame image and the previous frame image according to the multiple targets recognized by the second detection box in the event frame image set specifically includes:

[0029] For all grayscale images in the event frame image set, for each target recognized by the second detection box, extract the feature vector of the target through the feature extraction network; the feature vector is used to describe the appearance and motion characteristics of the target;

[0030] Convert all the recognized targets to the same size through the L2 normalization method;

[0031] Starting from the grayscale image of the second frame of the event frame image set, calculate the similarity between the target in the current frame image and the target in the previous frame image;

[0032] Based on the similarity, associate and match the target in the current frame image with the target in the previous frame image through the Hungarian algorithm.

[0033] In a possible implementation manner, according to the targets being tracked in the current frame image and the previous frame image, obtain the predicted position of the target being tracked in the next frame image, identify the actual position of the target being tracked in the next frame image through the second detection box, and determine whether the target being tracked is the same tracking target according to the error magnitude between the predicted position and the actual position. Specifically:

[0034] For each of the targets being tracked, estimate the state information of the target being tracked through a Kalman filter, and predict the predicted position of each of the targets being tracked in the next frame image according to the state information; the state information includes the position, speed, and acceleration of the target being tracked;

[0035] Identify the target being tracked in the next frame image through the second detection box, and obtain the actual position of the target being tracked;

[0036] When the error between the actual position and the predicted position is less than a preset threshold, determine that the target being tracked is the same tracking target.

[0037] In a possible implementation manner, after determining multiple targets being tracked, the method further includes:

[0038] After initially determining each target being tracked, create a corresponding tracker for the target being tracked; the tracker is used to manage the feature vector and state information of the target being tracked.

[0039] In a possible implementation manner, before performing image feature extraction and classification regression processing on the grayscale images in the event frame image set through a convolutional neural network, the method further includes:

[0040] Perform denoising processing on the multiple grayscale images included in the event frame image set;

[0041] Adjust the resolution of the multiple grayscale images after denoising processing so that the resolutions of the multiple grayscale images are the same.

[0042] In a second aspect, the present invention provides a dynamic high-speed target tracking device based on an event camera, including:

[0043] A data acquisition module, configured to obtain a light brightness transformation map in a high-speed motion scene collected by an event camera, and parse the light brightness transformation map into an event frame image set; the event frame image set includes a plurality of grayscale images distributed in chronological order;

[0044] A first detection module, configured to perform image feature extraction and classification regression processing on the grayscale images in the event frame image set through a convolutional neural network, to obtain a plurality of first detection boxes for identifying different category targets; the first detection box includes a detection box position and a detection box confidence;

[0045] A second detection module, configured to remove first detection boxes with an overlapping degree meeting a first preset threshold based on a non-maximum suppression algorithm and the detection box confidence of each first detection box, to obtain second detection boxes;

[0046] A first tracking module, configured to perform target association and matching on the current frame image and the previous frame image in the event frame image set according to a plurality of targets identified by the second detection boxes in the current frame image and the previous frame image, to determine a plurality of tracked targets;

[0047] A second tracking module, configured to obtain a predicted position of a tracked target in a next frame image according to the tracked targets in the current frame image and the previous frame image, obtain an actual position of the tracked target in the next frame image through identification by a second detection box, and determine whether the tracked target is the same tracking target according to the error magnitude between the predicted position and the actual position, to obtain a tracking result; the tracking result includes the actual positions and identification numbers of the same tracking target in the previous frame image, the current frame image, and the next frame image;

[0048] A data visualization module, configured to obtain and display the tracking result in real time.

[0049] In a possible implementation manner, the data acquisition module is specifically configured to perform:

[0050] Obtain a light brightness transformation map in a high-speed motion scene collected by an event camera, and obtain grayscale transformation information of a plurality of the light brightness transformation maps based on a differential imaging principle; each grayscale transformation information includes pixel coordinate values, grayscale values, and trigger times of trigger positions of a plurality of events;

[0051] Obtain a corresponding grayscale matrix according to the grayscale transformation information; each grayscale matrix is used to describe a grayscale image, and a plurality of grayscale images distributed in chronological order form an event frame image set.

[0052] In a possible implementation, when obtaining the corresponding grayscale matrix according to the grayscale transformation information, the data acquisition module is specifically configured to execute:

[0053] Construct a grayscale matrix according to the resolution of the event camera, and assign an initial value of zero to each element of the grayscale matrix;

[0054] Traverse all events in each of the light intensity transformation maps, and assign the grayscale value in the grayscale change information to each element in the grayscale matrix corresponding to the pixel coordinate value of the trigger position.

[0055] In a possible implementation, after constructing the grayscale matrix according to the resolution of the event camera and assigning an initial value of zero to each element of the grayscale matrix, the data acquisition module is further configured to execute:

[0056] Judge whether the value of each element of the grayscale matrix is less than the corresponding grayscale value in the grayscale transformation information;

[0057] If so, assign the polarity of the element as -1; if not, assign the polarity of the element as +1.

[0058] In a possible implementation, the second detection module is specifically configured to execute:

[0059] For the first detection boxes of the same category target, obtain the first detection box with the highest detection box confidence;

[0060] Calculate the ratio of the intersection to the union of the first detection box with the highest detection box confidence and the remaining first detection boxes of the category target respectively; use the ratio to characterize the overlap degree between the first detection boxes;

[0061] Delete the remaining first detection boxes when the ratio is greater than the first preset threshold according to the size of the ratio and the first preset threshold.

[0062] In a possible implementation, the first tracking module is specifically configured to execute:

[0063] For all grayscale images in the event frame image set, for each target recognized by the second detection box, extract the feature vector of the target through the feature extraction network; the feature vector is used to describe the appearance and motion characteristics of the target;

[0064] Convert all the recognized targets to the same size through the L2 normalization method;

[0065] Starting from the grayscale image of the second frame of the event frame image set, calculate the similarity between the target of the current frame image and the target of the previous frame image;

[0066] Based on the similarity, the targets in the current frame image are associated and matched with the targets in the previous frame image through the Hungarian algorithm.

[0067] In a possible implementation, the second tracking module is specifically configured to execute:

[0068] For each of the tracked targets, the state information of the tracked target is estimated by a Kalman filter, and the predicted position of each of the tracked targets in the next frame image is predicted according to the state information; the state information includes the position, speed, and acceleration of the tracked target.

[0069] The tracked targets in the next frame image are identified through a second detection box, and the actual positions of the tracked targets are obtained.

[0070] When the error between the actual position and the predicted position is less than a preset threshold, it is determined that the tracked targets are the same tracking target.

[0071] In a possible implementation, after determining multiple tracked targets, the second tracking module is further configured to execute:

[0072] After initially determining each tracked target, a corresponding tracker is created for the tracked target; the tracker is used to manage the feature vectors and state information of the tracked target.

[0073] In a possible implementation, the target tracking device further includes a preprocessing module, which is configured to execute before performing image feature extraction and classification regression processing on the grayscale images in the event frame image set through a convolutional neural network:

[0074] Perform denoising processing on the multiple grayscale images included in the event frame image set;

[0075] Adjust the resolution of the multiple grayscale images after denoising processing so that the resolutions of the multiple grayscale images are the same.

[0076] In a third aspect, the present invention provides an electronic device, which includes a processor and a memory. The memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the dynamic high-speed target tracking method based on an event camera described in any one of the above.

[0077] Fourthly, the present invention provides a computer-readable storage medium, in which at least one instruction, at least one program, a code set or an instruction set is stored, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the method for dynamically tracking high-speed targets based on an event camera described in any one of the above items.

[0078] The method for dynamically tracking high-speed targets based on an event camera provided by the embodiments of the present invention collects a light brightness transformation map in a high-speed motion scene through the event camera and parses it into an event frame image set; extracts image features and performs classification and regression processing on the grayscale images in the event frame image set through a convolutional neural network to obtain a plurality of first detection frames for identifying targets of different categories; removes the first detection frames with a high degree of overlap through a non-maximum suppression algorithm and the detection frame confidence of each first detection frame to obtain second detection frames; performs target association and matching on the current frame image and the previous frame image according to the multiple targets identified by the second detection frames in the current frame image and the previous frame image of the event frame image set, so as to determine a plurality of tracked targets; obtains the predicted position of the tracked target in the next frame image according to the tracked targets in the current frame image and the previous frame image, obtains the actual position of the tracked target in the next frame image through the second detection frame, and then determines whether the tracked target is the same tracking target according to the error between the predicted position and the actual position to obtain a tracking result; this solution obtains a light brightness transformation map of a high-speed moving target through an event camera, and through processing the obtained light brightness transformation map, event frame images with a high frame rate hundreds of times that of a traditional camera can be obtained, and these event frame images are clearly imaged, solving the problems of blurring or inability to obtain effective target information when a traditional camera images high-speed and ultra-high-speed targets; in this solution, after obtaining the first detection frame, the highly overlapping detection frames are removed through a non-maximum suppression algorithm and the detection frame confidence of the first detection frame, and during the target tracking process, it is further determined whether the tracked target is the same target through the error between the predicted position and the actual position to improve the accuracy of the target detection and tracking results; and the target tracking is realized based on adjacent grayscale images in the event frame image set to continuously track the target. Description of the Drawings

[0079] Figure 1 It is a flowchart of the steps of the first method for dynamically tracking high-speed targets based on an event camera provided by the embodiments of the present invention;

[0080] Figure 2 It is a grayscale image in the parsed event frame image set provided by the embodiments of the present invention;

[0081] Figure 3The flowchart of steps of the second method for dynamically tracking high-speed targets based on an event camera provided by an embodiment of the present invention;

[0082] Figure 4 The structural block diagram of a device for dynamically tracking high-speed targets based on an event camera provided by an embodiment of the present invention. Detailed implementation manners

[0083] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0084] Hereinafter, the terms "first" and "second" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the embodiments of the present disclosure, unless otherwise specified, the meaning of "a plurality" is two or more. Additionally, the use of "based on" or "according to" is meant to be open and inclusive, as a process, step, calculation, or other action "based on" or "according to" one or more of the stated conditions or values may, in practice, be based on additional conditions or values beyond those stated.

[0085] When traditional cameras image high-speed and ultra-high-speed targets, there will be afterimages, or effective information of the targets cannot be obtained, and there is also the problem that the data of event cameras cannot be directly combined with existing artificial intelligence algorithms, resulting in the inability of existing target tracking methods to effectively detect and continuously track targets.

[0086] To solve the above problems, embodiments of the present invention provide a method and a device for dynamically tracking high-speed targets based on an event camera.

[0087] Figure 1 The flowchart of steps of the first method for dynamically tracking high-speed targets based on an event camera provided by an embodiment of the present invention. As Figure 1 shown, in a first aspect, the present invention provides a first method for dynamically tracking high-speed targets based on an event camera, and the method includes:

[0088] Step 101, obtain a light brightness transformation map in a high-speed motion scene collected by an event camera, and parse the light brightness transformation map into an event frame image set.

[0089] Among them, the event frame image set includes a plurality of grayscale images distributed in chronological order.

[0090] An event camera is a new type of sensor based on biological vision and is very different from traditional cameras. The working principle of an event camera is similar to that of the human eye. It can quickly and accurately capture the changes occurring in the environment and output the changes in pixel-level brightness. Different from traditional cameras, an event camera does not capture the changes in the scene by capturing continuous image frames at fixed time intervals, but obtains the change information of the scene by independently monitoring the time changes of each pixel in the scene. Therefore, event cameras have characteristics such as high frame rate, low power consumption, low data storage requirements, and high dynamic range. In addition, since an event camera only records the changes that occur in the scene rather than the images of the entire scene, the problem of motion blur that appears in the images can be significantly improved.

[0091] The light intensity change map refers to a color image of the pixel-level brightness change of the scene change information obtained by an event camera through monitoring the time changes of each pixel in the scene.

[0092] Figure 2 This is a grayscale image in the set of event frame images obtained by parsing provided by the embodiment of the present invention. As Figure 2 shown, the obtained grayscale image only contains the information of the moving targets in the scene and no additional background information will appear. In Figure 2 , the moving target is a human hand, and only the outline of the human hand can be seen, and no extra background information can be seen.

[0093] That is to say, when there are no moving objects as targets within the shooting field of view of the event camera, in the obtained grayscale image, except for a few noises, no information will be output.

[0094] Step 102: Extract image features and perform classification and regression processing on the grayscale images in the set of event frame images through a convolutional neural network to obtain multiple first detection frames for identifying different category targets.

[0095] Among them, the first detection frame includes the detection frame position and the detection frame confidence.

[0096] A convolutional neural network is a type of feedforward neural network that includes convolutional calculations and has a deep structure, and is one of the representative algorithms of deep learning.

[0097] Through a convolutional neural network, image features can be extracted within the receptive field, receive and respond to the same features returned by different receptive fields, and perform feature extraction and screening; the classification and regression processing quantifies the classification data by assigning numerical values to categories, thereby classifying different image features.

[0098] Identify different category targets in the grayscale image through the first detection frame to obtain multiple first detection frames.

[0099] In this embodiment, downsampling and feature fusion are performed through convolutional and pooling layers to fuse shallow information and deep semantic information, obtaining an effective feature map containing rich target information.

[0100] Position and class regression classification processing are performed on the effective feature map to obtain multiple groups of prediction vectors. The first detection box is described by the prediction vectors, and the prediction vectors include the detection box position, the predicted box size, the predicted box class, and the detection box confidence.

[0101] Step 103: Based on the non-maximum suppression algorithm and the detection box confidence of each first detection box, remove the first detection boxes whose overlap degree meets the first preset threshold to obtain the second detection box.

[0102] Specifically, when different category targets are recognized through the first detection box, the same target may be recognized by multiple first detection boxes at the same time. Therefore, we need to remove the duplicate detection boxes in the recognition of the same target.

[0103] In this embodiment, the non-maximum suppression algorithm is used to process the detection box confidence of the first detection box, and the first preset threshold is used as the screening condition to remove the highly overlapping detection boxes.

[0104] Further, based on the non-maximum suppression algorithm and the detection box confidence of each first detection box, removing the first detection boxes whose overlap degree meets the first preset threshold specifically includes:

[0105] Step 1031: For the first detection boxes of the same category target, obtain the first detection box with the maximum detection box confidence.

[0106] Specifically, for the first detection boxes of the same category, compare and select the first detection box with the maximum detection box confidence among them.

[0107] Step 1032: Calculate the ratio of the intersection to the union between the first detection box with the maximum detection box confidence and the remaining first detection boxes of the category target, and use the ratio to represent the overlap degree between the first detection boxes.

[0108] Specifically, calculate the ratio of the intersection to the union between the first detection box with the maximum detection box confidence selected and the remaining first detection boxes of the same category, that is, the intersection over union ratio.

[0109] Step 1033: According to the size relationship between the ratio and the first preset threshold, delete the remaining first detection boxes when the ratio is greater than the first preset threshold.

[0110] Specifically, if the ratio of the intersection to the union is greater than the first preset threshold, delete the corresponding first detection box, and continue to perform the operation in Step 1032 on the remaining first detection boxes until all first detection boxes are processed.

[0111] In this embodiment, the first preset threshold is taken as 0.5.

[0112] Step 104: Based on the multiple targets recognized by the second detection box in the current frame image and the previous frame image in the event frame image set, perform target association and matching on the current frame image and the previous frame image to determine multiple tracked targets.

[0113] Specifically, since both the current frame image and the previous frame image are grayscale images obtained by parsing, all targets in the current frame image and the previous frame image can be recognized through the second detection box. By performing association and matching on the targets of adjacent two frames, the same targets in the current frame image and the previous frame image can be determined, and these same targets are the tracked targets.

[0114] Furthermore, based on the multiple targets recognized by the second detection box in the current frame image and the previous frame image in the event frame image set, performing target association and matching on the current frame image and the previous frame image specifically includes:

[0115] Step 1041: For all grayscale images in the event frame image set, for each target recognized by the second detection box, extract the feature vector of the target through the feature extraction network.

[0116] Among them, the feature vector is used to describe the appearance and motion characteristics of the target.

[0117] For each target recognized by the second detection box, obtain the feature vector of the target through the feature extraction network, and describe the appearance and motion characteristics of the target through this feature vector.

[0118] In this embodiment, the feature extraction network adopts a CNN backbone feature extraction network.

[0119] Step 1042: Convert all recognized targets to the same size through the L2 normalization method.

[0120] Specifically, in order to better associate and subsequently track the targets, convert all recognized targets to a unified size to improve the accuracy of target tracking.

[0121] Step 1043: Starting from the grayscale image of the second frame of the event frame image set, calculate the similarity between the targets in the current frame image and the targets in the previous frame image.

[0122] Specifically, calculate the similarity between the targets in the current frame image and the targets in the previous frame image through the cosine similarity algorithm or the Euclidean distance algorithm.

[0123] Among them, the cosine similarity algorithm measures the similarity between two targets by measuring the cosine value of the angle between the feature vectors of the targets in the current frame image and the feature vectors of the targets in the previous frame image.

[0124] When the target A in the current frame image and the target B in the previous frame image are regarded as points in the coordinate graph, the similarity between the two is the distance dist(A, B), and the similarity can be calculated by the Euclidean distance algorithm.

[0125] Step 1044: Based on the similarity, the targets in the current frame image and the targets in the previous frame image are associated and matched through the Hungarian algorithm.

[0126] Specifically, the Hungarian algorithm is a combinatorial optimization algorithm for solving the task assignment problem within a polynomial. Through the Hungarian algorithm, the targets in the previous frame image and the targets in the current frame image can be optimally matched.

[0127] Step 105: According to the targets being tracked in the current frame image and the previous frame image, obtain the predicted position of the targets being tracked in the next frame image. Identify the actual position of the targets being tracked in the next frame image through the second detection box, and determine whether the targets being tracked are the same tracking target according to the error size between the predicted position and the actual position, so as to obtain the tracking result.

[0128] Among them, the tracking result includes the actual positions and identification numbers of the same tracking target in the previous frame image, the current frame image, and the next frame image.

[0129] Specifically, according to the targets being tracked in the current frame image and the previous frame image, the appearance features and motion trends of the targets can be obtained. According to the appearance features and motion trends of the targets, the predicted position where the targets will appear in the next frame image can be predicted.

[0130] The actual positions where all the targets being tracked appear in the next frame image can be directly identified through the second detection box.

[0131] According to the error size between the predicted position and the actual position, it can be determined whether the targets appearing in the next frame image and the targets being tracked in the current frame image are the same target, and the same target is identified with the same identification number for distinction from other targets.

[0132] Furthermore, according to the targets being tracked in the current frame image and the previous frame image, obtain the predicted position of the targets being tracked in the next frame image. Identify the actual position of the targets being tracked in the next frame image through the second detection box, and determine whether the targets being tracked are the same tracking target according to the error size between the predicted position and the actual position. Specifically, it includes:

[0133] Step 1051: For each target to be tracked, estimate the state information of the target to be tracked through a Kalman filter, and predict the predicted position of each target to be tracked in the next frame of image based on the state information.

[0134] Among them, the state information includes the position, speed, and acceleration of the target to be tracked.

[0135] The Kalman filter refers to Kalman filtering, which is an algorithm that uses the state equation of a linear system to optimally estimate the system state through the input and output observation data of the system.

[0136] Based on the actual positions and motion states of each target in the previous frame of image and the current frame of image, the predicted position of the target in the next frame of image can be estimated.

[0137] Step 1052: Identify the target to be tracked in the next frame of image through the second detection box, and obtain the actual position of the target to be tracked.

[0138] Specifically, the actual position of the target to be tracked can be directly identified through the second detection box.

[0139] Step 1053: When the error between the actual position and the predicted position is less than the preset threshold, determine that the target to be tracked is the same tracking target.

[0140] Specifically, when tracking targets, multiple targets of the same type may be recognized. To avoid misidentifying different targets of the same type as the same target for tracking, we judge whether the target to be tracked is the same target by the error size between the predicted position and the actual position. If the error exceeds the preset threshold, it means that the tracked targets are not the same target.

[0141] Step 106: Obtain and display the tracking result in real time.

[0142] Specifically, the obtained tracking result is displayed in real time on the display screen in the form of a trajectory diagram.

[0143] Furthermore, after determining multiple targets to be tracked, the method further includes:

[0144] After first determining each target to be tracked, create a corresponding tracker for the target to be tracked; the tracker is used to manage the feature vector and state information of the target to be tracked.

[0145] Specifically, maintain a tracker for each target to be tracked. The tracker is also responsible for managing the appearance features and motion trends of the target. The number of the tracker corresponds one-to-one with the identification number of the target.

[0146] If the corresponding target has not been detected within the preset time period, the target will be marked as missing at the tracker, and the tracking of the target will be stopped.

[0147] If a new target is detected, a new tracker will be created for the target.

[0148] The dynamic high-speed target tracking method based on an event camera provided by an embodiment of the present invention collects a light brightness transformation map in a high-speed motion scene through the event camera and parses it into an event frame image set; extracts image features and performs classification and regression processing on the grayscale images in the event frame image set through a convolutional neural network to obtain multiple first detection frames for identifying different category targets; eliminates the first detection frames with a high degree of overlap through the non-maximum suppression algorithm and the detection frame confidence of each first detection frame to obtain second detection frames; performs target association and matching on the current frame image and the previous frame image according to the multiple targets identified by the second detection frames in the current frame image and the previous frame image of the event frame image set, so as to determine multiple tracked targets; obtains the predicted position of the tracked target in the next frame image according to the tracked targets in the current frame image and the previous frame image, obtains the actual position of the tracked target in the next frame image through the second detection frame recognition, and then determines whether the tracked target is the same tracking target according to the error size between the predicted position and the actual position to obtain the tracking result; this solution obtains a light brightness transformation map of a high-speed moving target through the event camera, and by processing the obtained light brightness transformation map, event frame images with a high frame rate hundreds of times that of a traditional camera can be obtained, and these event frame images are clearly imaged, solving the problem of blurry images or inability to obtain effective information of the target when a traditional camera images high-speed and ultra-high-speed targets; in this solution, after obtaining the first detection frame, the highly overlapping detection frames are removed through the non-maximum suppression algorithm and the detection frame confidence of the first detection frame, and during the target tracking process, it is further determined whether the tracked target is the same target through the error between the predicted position and the actual position to improve the accuracy of the target detection and tracking results; and the target tracking is realized based on adjacent grayscale images in the event frame image set to continuously track the target.

[0149] Figure 3 It is a flowchart of the steps of the second dynamic high-speed target tracking method based on an event camera provided by an embodiment of the present invention. As Figure 3 shown, an embodiment of the present invention also provides a second dynamic high-speed target tracking method based on an event camera, and the method includes:

[0150] Step 301, obtain the light brightness transformation map in the high-speed motion scene collected by the event camera, and obtain the grayscale transformation information of multiple light brightness transformation maps based on the differential imaging principle.

[0151] Among them, each grayscale transformation information includes the pixel coordinate values, grayscale values, and trigger times of the trigger positions of multiple events.

[0152] Specifically, M light intensity transformation diagrams P = {P i |0 < i ≤ M} in a high-speed motion scene are collected through the lens of an event camera, where P i represents the i-th image, M is an integer, and M > 1.

[0153] Based on the principle of differential imaging, the imaging chip of the event camera can sense the change in the grayscale information of the target image P i and obtain the grayscale transformation information of multiple light intensity transformation diagrams.

[0154] Among them, the grayscale transformation information is represented by S, S = {S i |0 < i ≤ M}, where S i represents the corresponding grayscale transformation information of P i ; and S i = {c i,j = (x i,j , y i,j , g i,j , t i,j )|0 < j ≤ N i}, N i > 0, N i represents the number of events included in each image in P i , c i,j represents the j-th event, x i,j and y i,j respectively represent the abscissa and ordinate of the trigger position pixel of c i,j , g i,j represents the grayscale value of c i,j , g i,j ≥ 0, t i,j represents the trigger time of c i,j .

[0155] Step 302: Obtain the corresponding grayscale matrix according to the grayscale transformation information.

[0156] Among them, each grayscale matrix is used to describe a grayscale image, and multiple grayscale images distributed in chronological order form an event frame image set.

[0157] Specifically, according to the grayscale transformation information S, the grayscale transformation information can be converted into the corresponding grayscale matrix, and each element in the grayscale matrix represents a pixel point, and each grayscale matrix can represent a grayscale image as shown in Figure 2 .

[0158] Furthermore, obtaining the corresponding grayscale matrix according to the grayscale transformation information specifically includes:

[0159] S3021. Construct a grayscale matrix according to the resolution of the event camera, and assign an initial value of zero to each element of the grayscale matrix.

[0160] Specifically, construct a grayscale matrix M1 = z(H, W), and let each element m in M1 be 0. Here, H and W represent the camera resolution respectively. In this embodiment, H ≥ 128 and W ≥ 128.

[0161] S3022. Determine whether the value of each element of the grayscale matrix is less than the corresponding grayscale value in the grayscale transformation information; if so, assign the polarity of the element as -1; if not, assign the polarity of the element as +1.

[0162] Specifically, the polarity of the element is represented by p i,j to determine whether g i,j > m xi,j,yi,j holds; if so, let p i,j = +1; if not, let p i,j = -1, where represents the element at the corresponding position in the grayscale matrix M1 corresponding to c i,j .

[0163] S3023. Traverse all events in each light intensity transformation map, and assign the grayscale value in the grayscale change information to each element in the grayscale matrix corresponding to the pixel coordinate value of the trigger position.

[0164] Specifically, let and determine whether j < N i holds; if so, let j = j + 1 and execute S3022; if not, obtain the corresponding grayscale map after parsing. Among them, the address event data stream E i = {e i,j = (x i,j , y i,j , p i,j , t i,j )|0 < i ≤ N i} is used to represent this grayscale map, e i,j is the parsing result of c i,j , p i,j represents the polarity of e i,j . Before S3022, let i = 1 and j = 1.

[0165] Determine whether i < M holds. If it holds, let i = i + 1, let j = 1, and execute S3022; if not, output the event frame image set obtained after parsing.

[0166] Step 303. Denoise the multiple grayscale maps included in the event frame image set.

[0167] Specifically, when any event in each grayscale image occurs, a spatial window around it, that is, the events that occur in a short time among the 8 pixels around this event are regarded as noise, and the noise around this event is removed. The same denoising operation is performed on all events in this grayscale image to obtain a denoised grayscale image.

[0168] Step 304: Adjust the resolution of multiple denoised grayscale images so that the resolutions of the multiple grayscale images are the same.

[0169] Specifically, each grayscale image is downsampled to obtain a grayscale image with a preset resolution size containing target information.

[0170] Step 305: Use a convolutional neural network to perform image feature extraction and classification regression processing on the grayscale images in the event frame image set to obtain multiple first detection frames for identifying different category targets.

[0171] In the embodiment of the present invention, Step 305 can refer to Step 102, which will not be elaborated here.

[0172] Step 306: Based on the non-maximum suppression algorithm and the detection box confidence of each first detection frame, remove the first detection frames whose overlapping degree meets the first preset threshold to obtain second detection frames.

[0173] In the embodiment of the present invention, Step 306 can refer to Step 103, which will not be elaborated here.

[0174] Step 307: According to the multiple targets identified by the second detection frames in the current frame image and the previous frame image in the event frame image set, perform target association and matching on the current frame image and the previous frame image to determine multiple tracked targets.

[0175] In the embodiment of the present invention, Step 307 can refer to Step 104, which will not be elaborated here.

[0176] Step 308: According to the tracked targets in the current frame image and the previous frame image, obtain the predicted positions of the tracked targets in the next frame image, obtain the actual positions of the tracked targets in the next frame image identified by the second detection frames, and determine whether the tracked targets are the same tracking target according to the error size between the predicted positions and the actual positions to obtain the tracking result.

[0177] In the embodiment of the present invention, Step 308 can refer to Step 105, which will not be elaborated here.

[0178] Step 309: Obtain and display the tracking result in real time.

[0179] In the embodiment of the present invention, Step 309 can refer to Step 106, which will not be elaborated here.

[0180] When the dynamic high-speed target tracking method based on an event camera provided by an embodiment of the present invention is applied, since the dynamic range of the event camera is usually twice that of a traditional camera, even in the case of extremely high or extremely low light intensity, it can still capture moving targets in real time and obtain a light brightness transformation map. By analyzing the light brightness transformation map, a clear grayscale map after removing complex background information can be obtained. Therefore, the present invention can still capture moving targets in real time and continuously track them when the light intensity is extremely high or extremely low. Before tracking the target, the present invention removes the background information of the image, which not only reduces the amount of data for image processing, but also improves the accuracy of target recognition.

[0181] Figure 4 It is a structural block diagram of a dynamic high-speed target tracking device based on an event camera provided by an embodiment of the present invention. As Figure 4 shown, in a second aspect, the present invention provides a dynamic high-speed target tracking device based on an event camera, and the device includes:

[0182] A data acquisition module 401, configured to obtain a light brightness transformation map in a high-speed motion scene collected by an event camera, and parse the light brightness transformation map into an event frame image set; the event frame image set includes a plurality of grayscale maps distributed in chronological order;

[0183] A first detection module 402, configured to perform image feature extraction and classification regression processing on the grayscale maps in the event frame image set through a convolutional neural network to obtain a plurality of first detection frames for identifying different category targets; the first detection frame includes a detection frame position and a detection frame confidence;

[0184] A second detection module 403, configured to remove the first detection frames whose overlap degree meets a first preset threshold based on the non-maximum suppression algorithm and the detection frame confidence of each first detection frame to obtain second detection frames;

[0185] A first tracking module 404, configured to perform target association and matching on the current frame image and the previous frame image according to the multiple targets identified by the second detection frames in the event frame image set to determine a plurality of tracked targets;

[0186] A second tracking module 405, configured to obtain the predicted position of the tracked target in the next frame image according to the tracked targets in the current frame image and the previous frame image, obtain the actual position of the tracked target in the next frame image through the second detection frame, and determine whether the tracked target is the same tracking target according to the error size between the predicted position and the actual position to obtain a tracking result; the tracking result includes the actual positions and identification numbers of the same tracking target in the previous frame image, the current frame image, and the next frame image;

[0187] The data visualization module 406 is used to obtain and display the tracking results in real time.

[0188] Furthermore, the data acquisition module 401 is specifically configured to execute:

[0189] Obtain the light brightness transformation map in the high-speed motion scene collected by the event camera, and obtain the gray-scale transformation information of multiple light brightness transformation maps based on the principle of differential imaging; each gray-scale transformation information includes the pixel coordinate values, gray-scale values, and trigger times of the trigger positions of multiple events;

[0190] Obtain the corresponding gray-scale matrix according to the gray-scale transformation information; each gray-scale matrix is used to describe a gray-scale image, and multiple gray-scale images distributed in chronological order form an event frame image set.

[0191] Furthermore, when obtaining the corresponding gray-scale matrix according to the gray-scale transformation information, the data acquisition module 401 is specifically configured to execute:

[0192] Construct a gray-scale matrix according to the resolution of the event camera, and assign the initial value of each element of the gray-scale matrix to zero;

[0193] Traverse all events in each light brightness transformation map, and assign the gray-scale value in the gray-scale change information to each element in the gray-scale matrix corresponding to the trigger position pixel coordinate value.

[0194] Furthermore, after constructing the gray-scale matrix according to the resolution of the event camera and assigning the initial value of each element of the gray-scale matrix to zero, the data acquisition module 401 is further configured to execute:

[0195] Judge whether the value of each element of the gray-scale matrix is less than the corresponding gray-scale value in the gray-scale transformation information;

[0196] If so, assign the polarity of the element to -1; if not, assign the polarity of the element to +1.

[0197] Furthermore, the second detection module 403 is specifically configured to execute:

[0198] For the first detection boxes of the same category of targets, obtain the first detection box with the highest detection box confidence;

[0199] Calculate the ratio of the intersection to the union of the first detection box with the highest detection box confidence and the remaining first detection boxes of the category target respectively; use the ratio to characterize the overlap degree between the first detection boxes;

[0200] Delete the remaining first detection boxes when the ratio is greater than the first preset threshold according to the size of the ratio and the first preset threshold.

[0201] Further, the first tracking module 404 is specifically configured to perform:

[0202] For all grayscale images in the event frame image set, for each target recognized by the second detection box, extract the feature vector of the target through the feature extraction network; the feature vector is used to describe the appearance and motion characteristics of the target;

[0203] Convert all recognized targets to the same size through the L2 normalization method;

[0204] Starting from the grayscale image of the second frame of the event frame image set, calculate the similarity between the targets in the current frame image and the targets in the previous frame image;

[0205] Based on the similarity, associate and match the targets in the current frame image with the targets in the previous frame image through the Hungarian algorithm.

[0206] Further, the second tracking module 405 is specifically configured to perform:

[0207] For each tracked target, estimate the state information of the tracked target through the Kalman filter, and predict the predicted position of each tracked target in the next frame image according to the state information; the state information includes the position, speed, and acceleration of the tracked target;

[0208] Identify the tracked targets in the next frame image through the second detection box, and obtain the actual positions of the tracked targets;

[0209] When the error between the actual position and the predicted position is less than the preset threshold, determine that the tracked targets are the same tracking target.

[0210] Further, after determining multiple tracked targets, the second tracking module 405 is further configured to perform:

[0211] After first determining each tracked target, create a corresponding tracker for the tracked target; the tracker is used to manage the feature vector and state information of the tracked target.

[0212] Further, the target tracking device further includes a preprocessing module, which is configured to perform: before performing image feature extraction and classification regression processing on the grayscale images in the event frame image set through the convolutional neural network

[0213] Denoise the multiple grayscale images included in the event frame image set;

[0214] Adjust the resolution of the multiple denoised grayscale images so that the resolutions of the multiple grayscale images are the same.

[0215] Those skilled in the art can clearly understand that for the convenience and conciseness of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0216] In a third aspect, the present invention provides an electronic device, which includes a processor and a memory. The memory stores at least one instruction, at least one program, a code set, or an instruction set. The at least one instruction, at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the method for dynamically and highly-speed target tracking based on an event camera according to any one of the above.

[0217] In a fourth aspect, the present invention provides a computer-readable storage medium, which stores at least one instruction, at least one program, a code set, or an instruction set. The at least one instruction, at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the method for dynamically and highly-speed target tracking based on an event camera according to any one of the above.

[0218] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions according to the embodiments of the present invention are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that the computer can access, or a data storage device such as a server or data center that includes one or more integrated available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid-state disk (SSD)).

[0219] As described above, the above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions within the technical scope disclosed by the present invention should be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A dynamic high-speed target tracking method based on an event camera, characterized in that, Including: Obtain a light brightness transformation map in a high-speed motion scene collected by an event camera, and parse the light brightness transformation map into an event frame image set; the event frame image set includes a plurality of grayscale images distributed in chronological order; Perform image feature extraction and classification regression processing on the grayscale images in the event frame image set through a convolutional neural network to obtain a plurality of first detection frames for identifying different category targets; The first detection frame includes a detection frame position and a detection frame confidence; Based on the non-maximum suppression algorithm and the detection frame confidence of each first detection frame, remove the first detection frames whose overlap degree meets the first preset threshold to obtain second detection frames; According to the multiple targets identified by the second detection frame in the current frame image and the previous frame image in the event frame image set, perform target association matching on the current frame image and the previous frame image to determine a plurality of tracked targets; According to the tracked targets in the current frame image and the previous frame image, obtain the predicted position of the tracked targets in the next frame image, obtain the actual position of the tracked targets in the next frame image through the second detection frame, and determine whether the tracked targets are the same tracking target according to the error size between the predicted position and the actual position, to obtain a tracking result; the tracking result includes the actual positions and identification numbers of the same tracking target in the previous frame image, the current frame image, and the next frame image; Obtain and display the tracking result in real time.

2. The method according to claim 1, characterized in that, The parsing of the light brightness transformation map into an event frame image set specifically includes: Obtain the grayscale transformation information of multiple light brightness transformation maps based on the differential imaging principle; each grayscale transformation information includes the pixel coordinate values, grayscale values, and trigger times of the trigger positions of multiple events; Obtain a corresponding grayscale matrix according to the grayscale transformation information; each grayscale matrix is used to describe a grayscale image, and a plurality of grayscale images distributed in chronological order form an event frame image set.

3. The method according to claim 2, characterized in that The obtaining of the corresponding grayscale matrix according to the grayscale transformation information specifically includes: Construct a grayscale matrix according to the resolution of the event camera, and assign an initial value of zero to each element of the grayscale matrix; Traverse all events in each light brightness transformation map, and assign the grayscale value in the grayscale change information to each element in the grayscale matrix corresponding to the trigger position pixel coordinate value.

4. The method according to claim 3, characterized in that, After constructing the grayscale matrix according to the resolution of the event camera and assigning an initial value of zero to each element of the grayscale matrix, the method further includes: Judge whether the value of each element of the grayscale matrix is less than the corresponding grayscale value in the grayscale transformation information; If so, assign the polarity of the element to -1; if not, assign the polarity of the element to +1.

5. The method according to claim 1, characterized in that Based on the non-maximum suppression algorithm and the detection frame confidence of each first detection frame, removing the first detection frames whose overlap degree meets the first preset threshold specifically includes: For the first detection frames of the same category target, obtain the first detection frame with the largest detection frame confidence; Calculate the ratio of the intersection to the union between the first detection box with the highest confidence of the detection boxes and the remaining first detection boxes of the category target respectively; use the ratio to characterize the overlap degree between the first detection boxes. Delete the remaining first detection boxes when the ratio is greater than the first preset threshold according to the size relationship between the ratio and the first preset threshold.

6. The method according to claim 1, characterized in that, The step of performing target association and matching on the current frame image and the previous frame image according to the multiple targets recognized by the second detection box in the current frame image and the previous frame image in the event frame image set specifically includes: For all grayscale images in the event frame image set, for each target recognized by the second detection box, extract the feature vector of the target through the feature extraction network; the feature vector is used to describe the appearance and motion characteristics of the target. Convert all the recognized targets to the same size through the L2 normalization method. Starting from the grayscale image of the second frame of the event frame image set, calculate the similarity between the targets in the current frame image and the targets in the previous frame image. Based on the similarity, perform association and matching between the targets in the current frame image and the targets in the previous frame image through the Hungarian algorithm.

7. The method according to claim 6, wherein The step of obtaining the predicted position of the tracked target in the next frame image according to the tracked targets in the current frame image and the previous frame image, identifying the actual position of the tracked target in the next frame image through the second detection box, and determining whether the tracked target is the same tracking target according to the error size between the predicted position and the actual position specifically is: For each tracked target, estimate the state information of the tracked target through the Kalman filter, and predict the predicted position of each tracked target in the next frame image according to the state information; the state information includes the position, speed and acceleration of the tracked target. Identify the tracked target in the next frame image through the second detection box, and obtain the actual position of the tracked target. When the error between the actual position and the predicted position is less than the preset threshold, determine that the tracked target is the same tracking target.

8. The method according to claim 7, wherein After determining the multiple tracked targets, the method further includes: After first determining each tracked target, create a corresponding tracker for the tracked target; the tracker is used to manage the feature vector and state information of the tracked target.

9. The method according to claim 1, characterized in that, Before performing image feature extraction and classification regression processing on the grayscale images in the event frame image set through the convolutional neural network, the method further includes: Perform denoising processing on the multiple grayscale images included in the event frame image set. Adjust the resolution of the multiple denoised grayscale images so that the resolutions of the multiple grayscale images are the same.

10. A dynamic high-speed target tracking device based on an event camera, characterized in that, It includes: A data acquisition module, configured to acquire a light brightness transformation map in a high-speed motion scene collected by an event camera, and parse the light brightness transformation map into an event frame image set; the event frame image set includes multiple grayscale images distributed in chronological order. The first detection module is used to extract image features and perform classification and regression processing on the grayscale images in the event frame image set through a convolutional neural network, so as to obtain a plurality of first detection frames for identifying different category targets; the first detection frame includes the detection frame position and the detection frame confidence; The second detection module is used to remove the first detection frames with an overlapping degree meeting a first preset threshold based on the non-maximum suppression algorithm and the detection frame confidence of each first detection frame, so as to obtain second detection frames; The first tracking module is used to perform target association and matching on the current frame image and the previous frame image according to a plurality of targets identified by the second detection frames in the current frame image and the previous frame image in the event frame image set, so as to determine a plurality of tracked targets; The second tracking module is used to obtain the predicted position of the tracked target in the next frame image according to the tracked targets in the current frame image and the previous frame image, obtain the actual position of the tracked target in the next frame image through the second detection frame, and determine whether the tracked target is the same tracking target according to the error size between the predicted position and the actual position, so as to obtain a tracking result; the tracking result includes the actual positions and identification numbers of the same tracking target in the previous frame image, the current frame image, and the next frame image; The data visualization module is used to obtain and display the tracking result in real time.

Images