A low-latency target tracking method, device, medium and product

Abstract

The application discloses a low-delay target tracking method, device, medium and product, relates to the field of target tracking, and comprises the following steps: acquiring a frame image of a target to be tracked and an event set within a preset time period; generating a high-frequency event frame and a low-frequency event frame based on the event set; determining a target tracking frame by using a multi-modal fusion target tracking model based on the low-frequency event frame and the frame image; determining a latest template by using a Gaussian Laplacian operator and a sliding window method based on the target tracking frame; determining a target pixel position at a current moment by using a lightweight image matching network based on the high-frequency event frame, the target tracking frame and the latest template; and estimating the current target pixel position and a pixel speed by using Kalman filtering or Kalman filtering based on an enhanced state based on the target tracking frame and the target pixel position at the current moment. The application realizes low-delay target tracking while maintaining high accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of target tracking, and in particular, to a low-latency target tracking method, device, medium and product. BACKGROUND

[0002] Target tracking technology is one of the core problems of intelligent perception and is also the key to the autonomy of unmanned systems, and has a very wide application in the fields of unmanned aerial vehicles, autonomous driving, etc. However, due to the defects of the imaging principle based on light intensity integration and synchronous output, the traditional camera needs sufficient exposure time to output frame images, which makes it difficult to meet the high-frequency target tracking needs of high-speed unmanned systems. As a new type of bionic visual sensor, the event camera is different from the working mode of the traditional camera. Each pixel point of the event camera independently detects the light intensity change received by itself and asynchronously outputs an event stream containing the pixel position, light intensity change and time information. This imaging method makes the event camera have the characteristics of low latency and high temporal resolution, and can provide microsecond-level delay visual information for target tracking. However, the imaging principle of the event camera makes it only represent sparse scene contour information and cannot completely replace the visible light camera which can provide rich semantic information. Therefore, studying how to effectively fuse the event stream and the frame image to realize low-latency target tracking can effectively improve the intelligent perception ability of unmanned systems in rapidly changing scenes.

[0003] However, the fusion processing of the event stream and the frame image needs to overcome two challenges. On the one hand, the unique imaging characteristics of the event camera make the processing of the event stream quite difficult. Since the event stream output by the event camera has the characteristics of spatial dispersion and temporal asynchrony, traditional computer vision algorithms designed for synchronous and dense images cannot be directly used. And because the light-sensitive device is extremely sensitive, the output results of the event camera may contain noise points caused by light signal particle noise and photoelectric circuit noise; on the other hand, the temporal resolution of the event stream is in the microsecond level, which is quite different from ordinary frame images. Therefore, how to realize efficient and accurate multi-modal fusion while maintaining the low-latency advantage of the event stream is also one of the difficulties to be solved in the field of event stream and frame image fusion target tracking.

[0004] Due to the sparse semantic information, event cameras have inherent disadvantages in pattern recognition tasks such as object detection and tracking. Therefore, a considerable part of the work focuses on the fusion of traditional cameras and event cameras for object tracking, which needs to consider the problems of spatio-temporal registration, feature fusion, and fusion result processing. In the current research, most of the work follows the research results in the field of multi-modal perception, and converts the event stream into an "event frame" through event accumulation, and then realizes the fusion of the two modalities through feature-level fusion. Zhang et al. proposed CDFI, which effectively enhances the multi-modal feature fusion based on sub-attention and cross-domain attention through a new cross-domain attention algorithm, thereby realizing the fusion of event stream and frame image for object tracking; Aydin et al. chose to use a pulse neural network to process the event stream and a convolutional neural network to process the image, realizing a lightweight and high-precision object tracking. However, the above schemes all choose to generate event frames according to the image frequency, although they can achieve high tracking accuracy, but it is difficult to achieve high tracking frequency to cope with rapidly changing scenes. In order to further improve the tracking frequency, Zhang et al. proposed AFNet, which realizes the temporal registration of frame images and event accumulation images through a variable convolution and a style transfer module, so that event frames and images that are not aligned in time can still be fused. However, due to the use of generative neural networks, if the tracking frequency is to be improved, the computational load of the model will also increase, and real-time performance is difficult to guarantee. Grehrig et al. use a linear event generation model to find the region with the smallest luminosity error in the event frame based on the image template, thereby realizing object tracking. Although this method can make the tracking frequency no longer limited by the frame image, it uses convex optimization for image matching, and can only process 17000 event information per second, which cannot be used for real-time object tracking tasks. In summary, how to realize real-time low-delay fusion of event stream and frame image for object tracking is still a difficult problem in the field of intelligent perception based on event cameras.

[0005] In event camera-based object tracking, due to the complementary relationship between event stream and frame image, the current general fusion of event stream and frame image is adopted. Two key problems need to be solved in algorithm design: event stream processing and fusion of event stream and frame image. The former generally uses the method of event frame accumulation, which is currently relatively mature. The latter is still a difficulty in this field due to the problem of temporal registration in multi-modal fusion. On the one hand, temporal alignment of event frames and images can directly solve the problem of temporal registration, but it loses the advantage of low delay of event stream; on the other hand, the method of self-implementing temporal registration by model needs a more complex network structure, which is difficult to meet the real-time performance. Therefore, the needs of low delay and high accuracy need to be considered, and on the basis of existing research, a new algorithm framework is designed to realize low-delay object tracking of event stream and frame image. SUMMARY

[0006] The purpose of the present application is to provide a low-delay target tracking method, device, medium and product to reduce the delay time of target tracking and improve the accuracy of target tracking.

[0007] To achieve the above purpose, the present application provides the following solutions.

[0008] In a first aspect, the present application provides a low-delay target tracking method, comprising:

[0009] obtaining a frame image of a target to be tracked and an event set within a preset time period;

[0010] generating an event frame based on the event set; the event frame includes a high-frequency event frame and a low-frequency event frame;

[0011] determining a target tracking box using a multi-modal fusion target tracking model based on the low-frequency event frame and the frame image;

[0012] determining a latest template using a Gaussian Laplace operator and a sliding window method based on the target tracking box;

[0013] determining a target pixel position at the current time using a lightweight image matching network based on the high-frequency event frame, the target tracking box and the latest template; the lightweight image matching network includes a feature extraction module, a feature fusion module and a coordinate regression module; the feature extraction module includes two encoder-decoder structures; the encoder-decoder structure includes a 4-layer encoder network and a 4-layer decoder network; the feature fusion module includes a splicing operation, a convolutional long short-term memory neural network and a fusion unit; the coordinate regression module includes a multi-layer perception machine;

[0014] estimating the current target state using Kalman filtering or enhanced state-based Kalman filtering based on the target tracking box and the target pixel position at the current time; the target state includes the target pixel position and the pixel speed.

[0015] Optionally, generating an event frame based on the event set specifically includes:

[0016] generating an event frame using the formula ; wherein, is the event frame; is the pixel coordinate of the event frame; is the timestamp of the event frame, is is the timestamp of the most recent event, is the time decay parameter.

[0017] Optionally, the target tracking box is represented as:

[0018] ; wherein, is a target center position coordinate of a target to be tracked on a current frame image; w is a length of a target frame; h is a width of a target frame.

[0019] Optionally, based on the target tracking frame, a latest template is determined using a Gaussian Laplace operator and a sliding window method, and specifically includes:

[0020] determining whether the length or the width of the target tracking frame is greater than the side length of the image block;

[0021] if yes, extracting a gradient map using a Gaussian Laplace operator according to the image frame;

[0022] determining a latest template on the target tracking frame using a sliding window method; the latest template is a region with the highest texture richness on the target tracking frame; the texture richness is determined according to the top-left pixel coordinates of the image block, the side length of the image block, and the gradient map;

[0023] if no, taking the target tracking frame as the latest template.

[0024] Optionally, the texture richness is determined according to the top-left pixel coordinates of the image block, the side length of the image block, and the gradient map, and specifically includes:

[0025] determining the texture richness using a formula ; wherein, is the top-left pixel coordinates of the image block; k represents the k candidate image block; is the side length of the image block; is the gradient map; is the image gradient L1 norm extracted according to the Gaussian Laplace operator at the pixel point .

[0026] Optionally, based on the target tracking frame and the target pixel position at the current time, a current target state is estimated using Kalman filtering or Kalman filtering based on an enhanced state, and specifically includes:

[0027] when the target pixel position at the current time is received, the current target state is determined using Kalman filtering;

[0028] when the target tracking frame is received, the current target state is determined using Kalman filtering based on an enhanced state.

[0029] Optionally, when the target tracking frame is received, the current target state is determined using Kalman filtering based on an enhanced state, and specifically includes:

[0030] constructing an enhanced state model;

[0031] based on the enhanced state model, correcting the target tracking frame at the previous target estimation moment by using Kalman filtering to obtain the current target state.

[0032] In a second aspect, the present application provides a computer device, comprising: a memory, a processor to store a computer program on the memory and run the computer program on the processor, and the processor executes the computer program to implement the low-latency target tracking method in any one of the above.

[0033] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, and the computer program is executed by a processor to implement the low-latency target tracking method in any one of the above.

[0034] In a fourth aspect, the present application provides a computer program product comprising a computer program, and the computer program is executed by a processor to implement the low-latency target tracking method in any one of the above.

[0035] According to the embodiments provided in the present application, the following technical effects are disclosed:

[0036] The present application provides a low-latency target tracking method, device, medium and product, by acquiring a frame image of a target to be tracked and an event set within a preset time period; based on the event set, generating a high-frequency event frame and a low-frequency event frame; based on the low-frequency event frame and the frame image, using a multi-modal fusion target tracking model to determine a target tracking frame; based on the target tracking frame, using a Gaussian Laplacian operator and a sliding window method to determine a latest template; based on the high-frequency event frame, the target tracking frame and the latest template, using a lightweight image matching network to determine a target pixel position at the current moment; based on the target tracking frame and the target pixel position at the current moment, using Kalman filtering or Kalman filtering based on an enhanced state to estimate the current target pixel position and pixel speed. The present application realizes low-latency target tracking while maintaining high accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0038] Figure 1 A flowchart of a low-latency target tracking method provided by an embodiment of the present application.

[0039] Figure 2 This is a schematic diagram of a low-latency target tracking framework based on the fusion of event streams and frame images.

[0040] Figure 3 This is a schematic diagram of the network structure for the template update module.

[0041] Figure 4 This is a schematic diagram of the target tracking results.

[0042] Figure 5 This is a graph comparing the algorithm performance.

[0043] Figure 6 This is an application environment diagram of a low-latency target tracking method according to an embodiment of this application.

[0044] Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0045] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0046] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0047] The low-latency target tracking method of this application achieves low-latency target tracking while maintaining high accuracy. For example... Figure 2 As shown, the entire system consists of three modules: a template update module, a template tracking module, and a state estimation module. The template update module receives time-synchronized event frames and images, and uses a high-precision multimodal target tracking model to obtain the target's position and image appearance at the synchronization moment as the latest template. The high-frequency template tracking module uses even higher-frequency event frames and a lightweight image matching model to obtain the target's pixel position at the current moment based on the latest template. The state estimation module uses a filter under unordered measurements to estimate the target's state at the current moment based on the results of the template update and tracking modules, providing the final tracking result.

[0048] In one exemplary embodiment, such as Figure 1 As shown, a low-latency target tracking method is provided, including:

[0049] S1: obtaining a frame image of a target to be tracked and a set of events within a preset time period.

[0050] In practical applications, a set of events obtained within a period of time is defined as , wherein is the total number of events, is the pixel coordinate, timestamp and polarity of the i-th event, respectively.

[0051] S2: generating an event frame based on the set of events; the event frame includes a high-frequency event frame and a low-frequency event frame.

[0052] In practical applications, the received is used to calculate a time surface (TS) as the event frame :

[0053] .

[0054] , wherein is the pixel coordinate of the event frame; is the timestamp of the i-th event frame, is the timestamp of the most recent event at , and is a time decay parameter. In the present application, the event frame will be generated at two frequencies, high and low. The high-frequency event frame will be published at a frequency of 100-200 Hz, which is used for a low-delay template tracking module; the low-frequency event frame is published synchronously with the frame image, which is used for a template update module based on multi-modal fusion target tracking.

[0055] S3: determining a target tracking frame using a multi-modal fusion target tracking model based on the low-frequency event frame and the frame image.

[0056] In practical applications, the existing multi-modal fusion target tracking model is first used to track the target using the time-synchronized event frame and frame image. In the actual test of the present application, a small sample target tracking model ViPT based on Transformer is selected for template update due to the characteristics of the multi-modal dataset. It should be noted that the present application does not specifically specify the multi-modal fusion target tracking model, which can be selected according to the actual task requirements and dataset conditions.

[0057] Then, the output result of the target tracking model is unified into the form of , wherein is the target center position of the target in the current frame image, are the length and width of the target frame, respectively. ​

[0058] S4: Based on the target tracking box, determine the latest template using the Laplace operator of Gauss and the sliding window method.

[0059] As an optional implementation method, S4 specifically includes:

[0060] Determine whether the length or width of the target tracking box is greater than the side length of the image block.

[0061] If so, then the gradient map is extracted using the Laplacian Gaussian operator based on the image frame.

[0062] On the target tracking bounding box, a new template is determined using the sliding window method; the new template is the region with the highest texture richness on the target tracking bounding box; the texture richness is determined based on the pixel coordinates of the top left corner of the image patch, the side length of the image patch, and the gradient map.

[0063] If not, use the target tracking box as the latest template.

[0064] In practical applications, since the subsequent template tracking module requires image patches of fixed size, a texture-rich region needs to be selected from the tracking results as the latest template. To achieve this, image gradients are used to refine the selected region. Texture richness Evaluation:

[0065] .

[0066] in, The side length of the image patch. The coordinates of the top-left pixel of the image block. Gradient plot; for The L1 norm of the image gradient is extracted based on the Laplacian of Gaussian (LoG) operator. A sliding window method is used to search for the region with the highest texture richness, selecting it as the latest template.

[0067] The entire template update algorithm process is as follows:

[0068] Function patch_serach ( ).

[0069] 1: Obtain the target tracking bounding box: .

[0070] 2: if bbox(t).w>s or bbox(t).h>s then.

[0071] Based on the LoG operator, based on the frame image Extract gradient map: .

[0072] In Search the best region according to the sliding window method: .

[0073] 3: else.

[0074] Take (x, y) as the pixel center of the best region:

[0075] .

[0076] 4: end if.

[0077] Update the template: .

[0078] 14: Return .

[0079] S5: Based on the high-frequency event frame, the target tracking frame, and the latest template, determine the target pixel position at the current time by using a lightweight image matching network; the lightweight image matching network includes a feature extraction module, a feature fusion module, and a coordinate regression module; the feature extraction module includes two encoder-decoder structures; the encoder-decoder structure includes a 4-layer encoder network and a 4-layer decoder network; the feature fusion module includes a splicing operation, a convolutional long short-term memory neural network, and a fusion unit; and the coordinate regression module includes a multi-layer perception.

[0080] In actual application, according to the current latest template, a lightweight image matching network is used to realize low-delay target tracking. The network structure is as shown in Figure 3 , which is composed of a feature extraction module, a feature fusion module, and a coordinate regression module.

[0081] First, feature extraction. Taking the last target tracking result as the center, a TS image block for target tracking is cropped from the received event frame according to the same size as the template. The template and the image block are sent into the same encoder-decoder structure model to extract their respective semantic features. In specific implementation, a 4-layer encoder network is used to first obtain a deep feature vector of dimension 64x64x64, and then a 4-layer decoder network is used to obtain a feature map of dimension 64x64x64 from the feature vector. Then, the feature fusion module is used to fuse the feature maps extracted from the TS image block and the template into a feature map for coordinate regression. In specific implementation, the feature maps extracted by the feature extraction are first spliced in the channel direction to obtain a feature map of dimension 64x64x128.

[0082] Finally, the coordinate regression module is used to regress the target pixel position in the TS image block according to the fused feature map. In specific implementation, a multi-layer perception is used to regress the target pixel position in the TS image block according to the fused feature map.​​ feature map . Then, in order to utilize the inter-frame information, a convolutional LSTM network is utilized to fuse the current and last iteration fusion feature map and the splicing result to obtain the fusion feature map of the current iteration , and the fusion result is a feature vector of dimension.

[0083] Finally, a multi-layer perceptron is adopted to regress the offset of the target relative to the current TS block , and the coordinates of the current target in the pixel coordinate system are calculated.

[0084] S6: based on the target tracking frame and the target pixel position at the current time, a Kalman filter or an augmented state-based Kalman filter is utilized to estimate the current target state; the target state includes the target pixel position and the pixel velocity.

[0085] As an optional implementation, S6 specifically includes:

[0086] When the target pixel position at the current time is received, the Kalman filter is utilized to determine the current target state.

[0087] When the target tracking frame is received, the augmented state-based Kalman filter is utilized to determine the current target state, specifically including:

[0088] constructing an augmented state model.

[0089] based on the augmented state model, the Kalman filter is utilized to correct the target tracking frame at the previous target estimation time to obtain the current target state.

[0090] In actual applications, the neural network scale adopted by the template update and template tracking modules is different, and the inference speed is also different. In actual running process, the template update will obviously lag behind the template tracking, causing the problem of unordered measurement, and the specific performance is as shown in Figure 4 , wherein the first row shows the tracking result of the template update module, at this time, the event frame and the image are at the same time, and the accurate multi-modal fusion tracking performance is shown; the second row shows the tracking result of the template tracking module, at this time, only the event frame is updated, and the accurate inter-frame tracking performance is shown. The present application utilizes an augmented state-based Kalman filter (Augmented state-Kalman Filter, AS-KF) to utilize the tracking results of the template update module and the template tracking module with different update frequencies and non-timed arrival to estimate the target state, and the state estimation quantity , i.e., the target pixel position and the pixel velocity.​

[0091] When receiving the result from the template tracking module (the target pixel position at the current time), the target state is updated in the mode of ordinary Kalman filtering. When receiving the result from the template updating module (the target tracking frame), due to the delay of model reasoning, the result is an out-of-order measurement, and the AS-KF is used to process it:

[0092] First, the enhanced state model is constructed.

[0093] Suppose that the out-of-order measurement result is received at time , and the latest target estimation time is , the enhanced state is set as:

[0094] .

[0095] Wherein, is the state at time , is the state closest to the state at time in the estimated state; is the transpose of ; and is the maximum delay. The actual measurement corresponding to the enhanced state is:

[0096] .

[0097] Wherein, is the measurement result corresponding to the state time .

[0098] Correspondingly, the state transition equation and the observation equation of the model are as follows:

[0099] .

[0100] Wherein, the state transition matrix is in the following form, is the state transition matrix from to .

[0101] .

[0102] The measurement matrix is in the following form:

[0103] .

[0104] Wherein, H k is the measurement matrix corresponding to the state time .​H k-1 is measurement matrix corresponding to the state time instant; is measurement matrix corresponding to the state time instant.

[0105] state transition noise is of the form

[0106] .

[0107] is state transition noise at the time instant.

[0108] measurement noise N is of the form

[0109] .

[0110] is measurement noise corresponding to the state time instant.

[0111] Then, the state update for the out-of-sequence measurement is performed.

[0112] Using the augmented state model constructed above, the state at the last target estimation time instant is corrected using Kalman filtering:

[0113] .

[0114] .

[0115] where, is the posterior state estimate, whose covariance is , is the prior state estimate, whose covariance is :

[0116] .

[0117] .

[0118] and is the Kalman gain:

[0119] .

[0120] Through the above update equation, the current posterior state estimate is obtained, and it is taken as the state estimate of the current target.

[0121] In general, the template updating module utilizes the multi-modal fusion target tracking network to provide a template for template tracking with a lower update frequency but sufficient accuracy, thereby avoiding large deviations in subsequent target tracking. The template tracking module only utilizes event frames, which are not limited by frame image frequency, thereby greatly improving the frequency of target tracking. The state estimation module uses filtering to effectively suppress false tracking results and directly uses the target coordinates obtained by the template updating module with a delay to correct the current target state estimation, thereby further improving the accuracy of target tracking. Therefore, the method proposed in the present application can significantly improve the target tracking frequency without significant degradation in target tracking accuracy compared to using only a multi-modal target tracking network. This greatly reduces the target state update interval and reduces the time delay of target tracking.

[0122] For the task of event stream and frame image fusion target tracking, it is currently still a difficult point in the field to achieve high-precision target tracking while retaining the low-delay advantage of event stream due to the large frequency gap. The present application proposes a new low-delay target tracking method based on event stream and frame image fusion. Low and high frequency event frames are generated from event stream and used for template updating and template tracking modules, respectively. The template updating module uses synchronous event frames and images to achieve low-frequency but accurate target tracking through a multi-modal fusion target tracking network. The template tracking module uses the tracking results of the template updating module as a template to achieve higher frequency target tracking through a lightweight image matching network. The two modules are parallel and are fused through an AS-KF-based state estimation module. The state estimation module can effectively suppress false tracking results and directly use the tracking results of the template updating module with a delay to correct the current target state. Thus, in real-time target tracking tasks, compared to using only existing multi-modal fusion target tracking networks for end-to-end target tracking, this scheme can significantly reduce the interval time of target state estimation update while maintaining the accuracy, thereby achieving low-delay target tracking.

[0123] Taking the ViPT multi-modal fusion target tracking network as the benchmark method, the template updating module of the present application also uses ViPT for comparative experiments. Through multiple experiments, the RMSE of the benchmark method is 8.53 pixels, and the average time interval of target state update is 24.8 ms; the RMSE of the present application is 5.80 pixels, and the average time interval of target state update is 5.1 ms. It can be seen that compared to the benchmark method, the present application significantly reduces the target state update time interval while maintaining high accuracy, thereby achieving low-delay target tracking.

[0124] In the performance comparison experiment, the benchmark method uses the same multi-modal fusion target tracking network ViPT as the template update module, and realizes target state estimation through Kalman filtering. Figure 5 For the tracking trajectories of the method and the benchmark method on the same sequence, it can be seen that the accuracy of the method proposed in the application is higher than that of the benchmark method.

[0125] The low-latency target tracking method provided in the embodiments of the application can be applied in an application environment as shown in the figure. Figure 6 As shown in the figure, the terminal 102 communicates with the server 104 through a network. The data storage system can store the data required to be processed by the server 104. The data storage system can be separately arranged, can be integrated on the server 104, or can be placed on the cloud or other servers. The terminal 102 can send the event frame to be processed and the frame image to the server 104. After receiving the event frame to be processed and the frame image, the server 104 generates event frames of low and high frequencies for the event stream, which are respectively used for the template update and template tracking modules. The template update module uses the synchronous event frame and image to realize low-frequency but accurate target tracking through a multi-modal fusion target tracking network. The template tracking module takes the tracking result of the template update module as a template and realizes higher-frequency target tracking through a lightweight image matching network. The two modules are parallel and are fused through an AS-KF-based state estimation module. The server 104 can feed back the current target state obtained to the terminal 102. In addition, in some embodiments, the low-latency target tracking method can also be realized by the server 104 or the terminal 102 alone, for example, the terminal 102 can directly process the event frame to be processed and the frame image, or the server 104 can obtain the event frame to be processed and the frame image from the data storage system and process the event frame to be processed and the frame image.

[0126] The terminal 102 can be, but is not limited to, various desktop computers, notebook computers, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things device can be a smart speaker, a smart television, a smart air conditioner, a smart vehicle device, etc. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc. The server 104 can be realized by an independent server or a server cluster composed of multiple servers, and can also be a cloud server.

[0127] The application further provides an application scenario applying the low-latency target tracking method.

[0128] In an exemplary embodiment, a computer device is provided, which can be a server or a terminal, and an internal structure diagram thereof can be as shown in FIG. 1. Figure 7 The computer device includes a processor, a memory, an input / output interface (I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The database of the computer device is configured to store event frames and frame image data to be processed. The input / output interface of the computer device is configured to exchange information between the processor and external devices. The communication interface of the computer device is configured to communicate with external terminals through a network connection. The computer program is executed by the processor to implement a low-latency target tracking method.

[0129] Those skilled in the art can understand that Figure 7 the structure shown in FIG. 1 is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0130] In an exemplary embodiment, a computer device is provided, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the above-mentioned method embodiments.

[0131] In an exemplary embodiment, a computer readable storage medium is provided, which stores a computer program. The computer program is executed by a processor to implement the steps in the above-mentioned method embodiments.

[0132] In an exemplary embodiment, a computer program product is provided, which includes a computer program. The computer program is executed by a processor to implement the steps in the above-mentioned method embodiments.

[0133] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0134] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to a memory, a database or other medium used in the embodiments provided in the present application can include at least one of a non-volatile and a volatile memory. The non-volatile memory can include a read-only memory (ROM), a magnetic tape, a floppy disk, a flash memory, an optical storage, a high-density embedded non-volatile memory, a resistive random access memory (ReRAM), a magnetoresistive random access memory (MRAM), a ferroelectric random access memory (FRAM), a phase change memory (PCM), a graphene memory, etc. The volatile memory can include a random access memory (RAM) or an external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as a static random access memory (SRAM) or a dynamic random access memory (DRAM), etc.

[0135] The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a blockchain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0136] The technical features of the above embodiments can be combined in any manner. To make the description concise, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combinations of the technical features do not exist contradictions, they should be considered as the scope of the present application.

[0137] The principles and implementation manners of the present application are described by using specific examples in the present application, and the above-mentioned embodiments are only used to help understand the method and its core idea of the present application; meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manners and application ranges can be changed. In conclusion, the content of the present application should not be understood as a limitation.

Claims

1. A low-latency target tracking method, characterized in that, include: Acquire frame images of the target to be tracked and a set of events within a preset time period; Based on the event set, an event frame is generated; The event frames include high-frequency event frames and low-frequency event frames; Based on the low-frequency event frames and the frame images, a multimodal fusion target tracking model is used to determine the target tracking box; Based on the target tracking box, the latest template is determined using the Laplace operator of Gauss and the sliding window method; Based on the high-frequency event frame, the target tracking box, and the latest template, a lightweight image matching network is used to determine the target pixel position at the current moment. The lightweight image matching network includes a feature extraction module, a feature fusion module, and a coordinate regression module; the feature extraction module includes two encoder-decoder structures; the encoder-decoder structure includes a 4-layer encoder network and a 4-layer decoder network; the feature fusion module includes a concatenation operation, a convolutional long short-term memory neural network, and a fusion unit; the coordinate regression module includes a multilayer perceptron; Based on the target tracking box and the target pixel position at the current time, the current target state is estimated using Kalman filtering or Kalman filtering based on enhanced state. The target state includes the target pixel position and pixel velocity.

2. The low-latency target tracking method according to claim 1, characterized in that, Based on the event set, an event frame is generated, specifically including: Using formula Generate event frames; where, For event frames; The coordinates of one pixel in the event frame; The timestamp of the event frame. for The timestamp of the most recent event. This is the time decay parameter.

3. The low-latency target tracking method according to claim 1, characterized in that, The target tracking bounding box is represented as follows: ;in, The coordinates of the center position of the target to be tracked in the current frame image; The length of the bounding box; The width of the target bounding box.

4. The low-latency target tracking method according to claim 1, characterized in that, Based on the target tracking bounding box, the latest template is determined using the Laplacian of Gaussian operator and the sliding window method, specifically including: Determine whether the length or width of the target tracking box is greater than the side length of the image block; If so, then the gradient map is extracted based on the frame image using the Laplacian of Gaussian operator; On the target tracking bounding box, a new template is determined using the sliding window method; the new template is the region with the highest texture richness on the target tracking bounding box; the texture richness is determined based on the pixel coordinates of the top left corner of the image patch, the side length of the image patch, and the gradient map; If not, use the target tracking box as the latest template.

5. The low-latency target tracking method according to claim 4, characterized in that, Texture richness is determined based on the top-left pixel coordinates of the image patch, the side length of the image patch, and the gradient map, specifically including: Using formula Determine texture richness; among which, The coordinates of the top-left pixel of the image block; k Representing the k One candidate image patch; The side length of the image block; For gradient plots; For pixels The L1 norm of the image gradient extracted from the Laplacian Gaussian operator at the given location.

6. The low-latency target tracking method according to claim 1, characterized in that, Based on the target tracking box and the target pixel position at the current time, the current target state is estimated using Kalman filtering or Kalman filtering based on the enhanced state, specifically including: When the target pixel position at the current moment is received, the current target state is determined using Kalman filtering; When the target tracking box is received, the current target state is determined using an enhanced state-based Kalman filter.

7. The low-latency target tracking method according to claim 6, characterized in that, When the target tracking box is received, the current target state is determined using an enhanced state-based Kalman filter, specifically including: Build an enhanced state model; Based on the enhanced state model, the target tracking box at the previous target estimation time is corrected using Kalman filtering to obtain the current target state.

8. A computer device, comprising: The memory and processor contain a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the low-latency target tracking method according to any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the low-latency target tracking method according to any one of claims 1-7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the low-latency target tracking method according to any one of claims 1-7.

Images