A method and system for identifying and tracking fast moving targets

Abstract

The application provides a kind of fast moving target identification tracking method and system, method includes the following steps: S1.get frame data and event data;S2.convert event data into event frame;S3.judge event frame data whether it is initial frame;S4 identification target: the data of traditional camera acquisition is identified target, and the target frame after identification is output to step S5;S5.initialize tracking frame;S6.extract search image feature;S7.extract target image feature;S8.extract search event feature;S9.extract target event feature;S10.search feature fusion;S11.target feature fusion;S12.calculate contrast loss function;S13.result output.The application utilizes event camera and traditional camera data to carry out fusion, can carry out fusion to the advantage of two kinds of camera data, realizes the effective tracking of fast moving target, improves the tracking effect of moving target.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular, to a method and system for identifying and tracking fast-moving targets. Background Art

[0002] As a new type of biomimetic sensor, event cameras offer low latency and high dynamic range, making them advantageous in tracking fast-moving targets. However, due to the limitations of current sensor technology, while event cameras have high temporal resolution, their spatial resolution is low, and their data lacks texture features. These factors restrict their application in fast-moving target tracking. Currently, event cameras primarily utilize three sensor types: ATIS, DAVIS, and DVS, with Samsung and Sony primarily producing these sensors. Traditional cameras, while lacking high temporal resolution and subject to severe motion blur when capturing fast-moving objects, have more mature sensors with higher spatial resolution, enabling the extraction of effective texture features, complementing event camera data.

[0003] The main problem in the existing technology is that in the tracking and identification of fast-moving targets, because the target moves at a fast speed and is affected by factors such as light, the target's own deformation, and background changes during the movement, tracking fails when using traditional cameras to track the target.

[0004] The difficulty in solving the above problems and defects is: Tracking fast-moving targets has always been an inherent challenge in the application of computer vision industry. Simply relying on traditional cameras to collect and process data will cause tracking failures due to severe motion blur. It cannot be improved through simple algorithm optimization and is very difficult to solve.

[0005] The significance of solving the above problems and defects is: by combining event cameras and traditional cameras, the industry problem of tracking fast-moving targets is effectively solved. Compared with the method of high-speed cameras, the method of the present invention is more efficient, low-power, and low-cost. The method and system can be widely used in fields such as drones, autonomous driving, and aerospace, and have high application and practical value. Summary of the Invention

[0006] The present invention provides a method and system for identifying and tracking fast-moving targets. By fusing event camera and traditional camera data, the advantages of the two camera data can be integrated to achieve effective tracking of fast-moving targets and improve the tracking effect of moving targets.

[0007] The technical solutions of the present invention are as follows:

[0008] According to one aspect of the present application, a method for identifying and tracking a fast-moving target is provided, comprising the following steps: S1. obtaining frame data and event data: a traditional camera and an event camera are respectively arranged to collect frame data and event data; S2. converting event data into event frames: discrete data streams are converted into the form of event frames through a dynamic reconstruction method; S3. judging whether the event frame data is an initial frame: if it is an initial frame, step S4 is entered, and if it is not an initial frame, step S6 is entered; S4. identifying a target: target identification is performed on the data collected by the traditional camera, and the identified target frame is output to step S5; S5. initializing a tracking frame: the identified target is used to initialize the target to be tracked, and when the identified target is consistent with a preset target, the identified result is used to initialize the tracking frame, and step S7 is entered; S6. extracting search image features: search image features are extracted from search image frames by using a search image extraction network, and the extracted search image features are output to step S10; S7. extracting target image features: target image features are extracted from target image frames by using a target image extraction network, and an event target image is cut out from the event image, and the event target image is input to step S9; S8. extracting search event features: search event features are extracted from search event frames by using a search event extraction network, and the extracted search event features are output to step S10; S9. extracting target event features: target event features are extracted from target event frames by using a target event extraction network, and the extracted target event features are output to step S11; S10. search feature fusion: image features and search event features are fused, and the fused search features are input to step S11; S11. target feature fusion: target image features and target event features are fused, and the fused target features are input to step S12; S12. calculating a comparison loss function: a loss function value between the fused search features and the fused target features is calculated, and a target with the highest probability is obtained; S13. result output: a candidate result with the highest probability is extracted from step S12 as a result of tracking the fast-moving target, and the output is a tracking result of a current frame, and the tracking steps are repeated.

[0009] Optionally, in the method for identifying and tracking a fast-moving target, in step S1, the traditional camera collects absolute values of pixels, and the event camera only collects relative transformation values of pixels.

[0010] Optionally, in the above-mentioned fast-moving target identification and tracking method, the fusion method of the event features and the image features in steps S11 and S12 is that the feature fusion is performed by using a weight learning method, different weights are assigned to the feature fusion in different scenes, a larger weight is assigned to the event data when the speed of the target to be detected and tracked is fast, a larger weight is assigned to the image data when the speed of the target to be detected and tracked is slow, the adjustment of the weight is positively correlated with the motion speed of the target, and the total weight proportion is added up to equal one.

[0011] Optionally, in the above-mentioned fast-moving target identification and tracking method, in step S12, the dynamic reconstruction of the event data includes: the motion of adjacent frames is evaluated by using structural similarity, the event data of a shorter time period is intercepted for reconstruction when the target moves fast, the event data of a longer time is intercepted for reconstruction when the target moves slowly, and clear event image data is obtained.

[0012] Optionally, in the above-mentioned fast-moving target identification and tracking method, the event target extraction network and the event search network are trained by using an event data set, the image target extraction network and the image search extraction network are trained by using an image data set, features are extracted from the four groups of networks respectively, then the features can be fused, the image target extraction network is trained for target identification, and the trained network is used for target identification.

[0013] According to another aspect of the present application, a fast-moving target identification and tracking system comprises: a traditional camera and an event camera, which are used to collect frame data and event data respectively, the traditional camera collects the absolute value of pixels, and the event camera only collects the relative change value of pixels; an image target extraction network, which is used for target identification training and feature extraction of a target image frame; an event target extraction network, which is used for feature extraction of a target event frame; an image search extraction network, which is used for feature extraction of a search image frame; an event search extraction network, which is used for feature extraction of a search event frame; and a feature fusion network, which is used for feature fusion of the image features of the traditional camera and the event camera data features.

[0014] According to the technical scheme of the present application, the following beneficial effects are achieved:

[0015] The present application provides a fast-moving target identification and tracking method and system using event camera and traditional camera fusion, which can effectively utilize the advantages of traditional cameras and event cameras for complementation, fuse the data of the two modalities, improve the tracking performance of the traditional method in the fast and high-speed movement of the target, and solve a series of challenges such as motion blur and light change in the fast motion tracking algorithm of the traditional method.

[0016] In order to better understand and illustrate the concept, working principle and inventive effect of the present application, the present application will be described in detail below with specific embodiments combined with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the specific embodiments or prior art description will be briefly introduced as follows.

[0018] Figure 1 is a flowchart of the fast moving target recognition and tracking method of the present application;

[0019] Figure 2 is a network framework involved in the method of the present application;

[0020] Figure 3 is the frame data collected by the traditional camera and the event data collected by the event camera, the traditional camera collects the absolute value of the pixel, while the event camera only collects the relative transformation value of the pixel. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical method and advantages of the present application more clear, the present application will be further described in detail below combined with the drawings and specific examples. These examples are only illustrative, and are not limiting to the present application.

[0022] The present application provides an effective fusion event camera and traditional camera fast moving target recognition and tracking method, which fully utilizes the data advantages of the two kinds of sensors, combines the features of the data collected by the traditional camera and the features of the data collected by the event camera, and completes the effective recognition and tracking positioning of the fast moving object, especially the super high speed moving target.

[0023] The working principle of the method of the present application is: according to the different motion and speed of different recognition and tracking targets, the dynamic reconstruction of the event data is realized, the space-time domain of the event data can be adjusted according to different scenes, the target recognition of the target to be tracked is realized by using the traditional camera data, and then the feature fusion of the data of the two modalities is realized, and the twin network is used to realize the tracking of the fast moving target.

[0024] As shown in Figure 1 and Figure 2 , the fast moving target recognition and tracking method of the present application comprises the following steps:

[0025] S1. Acquire frame data and event data: set the traditional camera and the event camera to collect frame data and event data respectively.

[0026] Figure 3The frame data collected by the traditional camera and the event data collected by the event camera. The traditional camera collects the absolute value of the pixel, while the event camera only collects the relative transformation value of the pixel.

[0027] S2. Convert event data into event frames: Convert the asynchronous event data into discrete data streams in the form of event frames through dynamic reconstruction to facilitate subsequent processing.

[0028] Since the event camera outputs a high-speed asynchronous event stream that cannot be processed using a traditional convolutional network, the present invention uses a dynamic reconstruction method to convert the asynchronous event data into an image-like format.

[0029] The dynamic reconstruction of event data includes: using SSIM (structural similarity) to evaluate the motion of adjacent frames. When the target moves faster, event data of a shorter period of time is intercepted for reconstruction. When the target moves slower, event data of a longer period of time is intercepted for reconstruction. This ensures that clear event graph data can be reconstructed regardless of the motion state.

[0030] S3. Determine whether the event frame data and the traditional frame data are initial frames: if they are initial frames, proceed to step S4; if they are not initial frames, proceed to step S6.

[0031] S4. Target recognition: Target recognition is performed on the data collected by the traditional camera, and the recognized target frame is output to step S5.

[0032] The present invention uses the twin network to extract features. After the features are extracted, subsequent recognition and tracking can be performed. It is divided into a reference branch and a test branch. The twin network has a total of four branch networks to extract features. These four branches are the image target extraction network ( Figure 2 Network 1 in the event target extraction network ( Figure 2 Network 2), image search and extraction network ( Figure 2 Network 3) and event search extraction network ( Figure 2 In the network 4), the event target extraction network and the event search network are trained using event datasets, and the image target extraction network and the image search extraction network are trained using image datasets. After extracting features from the four groups of networks, they can be fused. In addition, the image target extraction network is trained for target recognition, and the trained network can perform target recognition.

[0033] S5. Initialize the tracking frame: Use the target recognition results to initialize the target to be tracked. When the recognized target is consistent with the preset target, use the recognition results to initialize the tracking frame, that is, input the target's center point coordinates (x, y) and width and height (w, h) into the tracking network to enter step S7.

[0034] S6. Extract search image features: the newly coming data is subjected to search image feature extraction. The search image frame (i.e. image frame search) in the above image search extraction network is subjected to feature extraction, and the extracted search image features are output to step S10. Figure 2

[0035] S7. Extract target image features: the target image frame (i.e. image frame target) in the above image target extraction network is subjected to feature extraction, and the event target image is cut out from the event image, and the event target image is input to step S9. Figure 2

[0036] S8. Extract search event features: the search event frame (i.e. event frame search) in the above event search extraction network is subjected to feature extraction, and the extracted search event features are output to step S10. Figure 2

[0037] S9. Extract target event features: the target event frame (i.e. event frame target) in the above event target extraction network is subjected to feature extraction, and the extracted target event features are output to step S11. Figure 2

[0038] S10. Search feature fusion: the extracted search image features and search event features are fused, and the fused search features are input to step S11.

[0039] S11. Target feature fusion: the extracted target image features and target event features are fused, and the fused target features are input to step S12.

[0040] The fusion method of the event features and image features in steps S11 and S12 is: the feature fusion is performed by using the weight learning method, different weights are assigned to the feature fusion in different scenes, when the speed of the detected and tracked target is fast, a larger weight is assigned to the event data, when the speed of the detected and tracked target is slow, a larger weight is assigned to the image data, the adjustment of the weight is positively correlated with the motion speed of the target, and the total weight proportion is added to equal one.

[0041] S12. Calculate the contrast loss function: the loss function value between the fused search features and the fused target features is calculated, and the highest probability of the tracked subsequent target is obtained.

[0042] S13. Result output: the highest probability candidate result is extracted from step S12 as the result of tracking the fast moving target, and the output is the tracking result of the current frame, and the tracking step is repeated.

[0043] ​​​​The fast-moving target identification and tracking system of the present invention comprises:

[0044] Traditional cameras and event cameras are used to collect frame data and event data respectively. Traditional cameras collect the absolute values ​​of pixels, while event cameras only collect the relative transformation values ​​of pixels.

[0045] Image target extraction network, target recognition training, and target image frame (i.e. Figure 2 Image frame target in the image frame) for feature extraction;

[0046] Event target extraction network is used to extract target event frames (i.e. Figure 2 Feature extraction of event frame targets in the image;

[0047] Image search extraction network, used to search image frames (i.e. Figure 2 Image frame search in ( ) for feature extraction;

[0048] Event search extraction network, used to search event frames (i.e. ​ Event frame search in

[15] for feature extraction; and

[0049] Feature fusion network is used to fuse the image features of traditional cameras and the data features of event cameras.

[0050] The results of the comparison of the advantages of the solution adopted by the present invention and other methods are shown in Table 1 below, which shows that the present invention can achieve data acquisition and tracking methods that are superior to high-speed cameras with lower power consumption and weight:

[0051] Table 1

[0052]

[0053]

[0054] The present invention discloses a method and system for identifying and tracking fast-moving targets. This method utilizes the fusion of event cameras and traditional cameras for identification and tracking, primarily using data collected by event cameras and traditional cameras for feature fusion to track fast-moving targets. This method is a novel technology for tracking and identifying moving targets using two different types of sensors. Specifically, the method utilizes two types of data for fast-moving target tracking: one type of data is frame data from a traditional camera, and the other type of data is event frame data extracted by a novel sensor. Through three main technical means, namely, dynamic reconstruction of event data, feature extraction by a feature extraction network, and feature fusion, the method achieves dynamic reconstruction of event data and feature fusion with traditional camera data, thereby effectively tracking fast-moving targets.

[0055] The above description is based on the best mode of the present application and working principle. The above description should not be understood as a limitation on the scope of the present application. Other embodiments and combinations of the present application are also within the scope of the present application.

Claims

1. A method for identifying and tracking a fast-moving target, characterized in that: The following steps are involved: S1. Acquiring frame data and event data: Set up a traditional camera and an event camera to collect frame data and event data respectively; S2 converting the event data into an event frame: The asynchronous event data is dynamically reconstructed to convert discrete data streams into event frames; S3 determines whether the event frame data is the initial frame: If it is the initial frame, the process proceeds to step S4, if it is not the initial frame, the process proceeds to step S6; S4 target identification: target recognition data collected by the traditional camera, the target frame after identification is output to step S5; S5. Initialize the tracking frame: Initialize the target to be tracked using the target recognition result. When the recognized target is consistent with the preset target, the tracking frame is initialized using the recognition result, and the process proceeds to step S7. S6 extract search image features: using the image search extraction network to search the image frame feature extraction, and the extracted search image features are output to step S10; S7 target image feature extraction: using the target image extraction network to extract features of the target image frame, and cut out the event target image from the event image, the event target image is input to step S9; S8 extract search event features: using the event search extraction network to extract features of the search event frame, and the extracted search event features are output to step S10; S9. Extracting target event features: Using the event target extraction network to extract features of the target event frame, and outputting the extracted target event features to step S11; S10 search feature fusion: the extracted image features and the search event features are fused, and the fused search features are input into step S11; S11 target feature fusion: the extracted target image features and the target event features are fused, and the fused target features are input into step S12; S12. Compute the contrast loss function: Calculate the loss function between the fused search features and the fused target features to obtain the highest probability for tracking the subsequent target. S13. Result output: Extract the candidate result with the highest probability from step S12 as the result of tracking the fast-moving target and output it as the tracking result of the current frame, and repeat the tracking step.

2. The method for identifying and tracking a fast-moving target according to claim 1, wherein: In step S1 , the conventional camera collects the absolute values ​​of pixels, while the event camera only collects the relative transformation values ​​of pixels.

3. The method for identifying and tracking a fast-moving target according to claim 1, wherein: in, The method for fusing event features and image features in steps S11 and S12 is: using a weight learning method to perform feature fusion, assigning different weights to feature fusion in different scenarios, assigning a larger weight to the event data when the target speed being detected and tracked is faster, and assigning a larger weight to the image data when the target speed being detected and tracked is slower, the weight adjustment is positively correlated with the target's movement speed, and the total weight ratio sums up to one.

4. The method for identifying and tracking a fast-moving target according to claim 1, wherein: in, In step S12, the dynamic reconstruction of event data includes: evaluating the motion of adjacent frames by utilizing structural similarity; when the target moves faster, intercepting event data of a shorter period of time for reconstruction; when the target moves slower, intercepting event data of a longer period of time for reconstruction, so as to obtain clear event graph data.

5. The method for identifying and tracking a fast-moving target according to claim 1, wherein: in, The event target extraction network and the event search network are trained using event data sets, and the image target extraction network and the image search extraction network are trained using image data sets. After extracting features from the four groups of networks respectively, they can be fused and target recognition training can be performed on the image target extraction network. The trained network is used for target recognition.

6. A fast-moving target identification and tracking system, used to implement the fast-moving target identification and tracking method according to any one of claims 1 to 5, characterized in that: include: Traditional cameras and event cameras are used to collect frame data and event data respectively. Traditional cameras collect the absolute values ​​of pixels, while event cameras only collect the relative transformation values ​​of pixels. Image target extraction network, which performs target recognition training and feature extraction on target image frames; Event target extraction network, used to extract features from target event frames; Image search extraction network, used to extract features from search image frames; An event search extraction network for feature extraction of search event frames; and A feature fusion network is used to fuse the image features of the traditional camera and the data features of the event camera.

Images