Event tracking method based on multi-scale knowledge distillation

Through multi-scale knowledge distillation and Transformer network, the problems of data redundancy and spatiotemporal attribute loss in event stream tracking are solved, and high-performance event stream tracking is achieved, which improves robustness and feature expression.

CN118898705APending Publication Date: 2024-11-05ANHUI UNIV

Patent Information

Application Number
CN202311596776.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-11-28
Publication Date
2024-11-05

AI Technical Summary

Technical Problem

The existing event stream tracking methods will introduce redundant information or lose spatiotemporal attributes in the data expression form, and it is difficult to effectively utilize event stream data. Convolutional neural networks lack global information modeling, and pulsed neural networks have low performance.

Method used

Using a multi-scale knowledge distillation method, the teacher network is trained through dual-modal data, and three-layer knowledge distillation (similarity matrix, feature representation, response mapping) is used to guide students' network learning, and features are extracted and timing information is fused with Transformer backbone network. Implement robust tracking of event flow.

Benefits of technology

It improves the robustness and performance of the student network, can improve the tracking effect at high-speed inference speed, deeply explore the deep spatial and temporal information of video frames, and obtain better feature expression and tracking results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118898705A_ABST
    Figure CN118898705A_ABST
Patent Text Reader

Abstract

The invention discloses an event tracking method based on multi-scale knowledge distillation, and belongs to the technical field of event cameras, and the method comprises the following steps: S1, inputting bimodal data, and training a teacher network; wherein the bimodal data comprises an RGB frame and an event stream; s2, single-mode event flow data are input, a student network is trained, and the single-mode event flow data comprise event flows; and through three-layer knowledge distillation, network learning of students is supervised and guided by using parameters of a teacher network. Through the above mode, multi-layer supervision is carried out on the student network by using a multi-layer knowledge distillation mode, and the robustness of the target student network can be improved. The input of the student network is single-mode event flow data, and the performance is improved under the condition of high reasoning speed through the guidance of the teacher network. A Transform backbone network is utilized to establish a long-distance dependency relationship of time sequence information, deep spatio-temporal information of video frames is deeply mined, and better feature expression is obtained.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of event cameras, and in particular to an event tracking method based on multi-scale knowledge distillation. Background Art

[0002] Compared with traditional visible light cameras (RGB cameras), event cameras have the advantages of low power consumption, low latency, and high dynamic range, and have opened up broad prospects in the field of computer vision. Visual tracking based on event camera data is a fundamental problem in this field. In order to track event stream data, it is first necessary to obtain a feature representation based on the original event stream, and then use the tracking model to locate the target position.

[0003] Existing event data mainly has two forms:

[0004] The first is to treat event data as a graph or point cloud, maintaining the sparsity of temporal data, but the event data is large in scale and may introduce too much redundant information;

[0005] The second method is to represent the event stream in the form of event frames, projecting the events onto a two-dimensional plane at certain time intervals. This retains the spatial properties of the event stream but sacrifices the natural temporal properties of the event stream.

[0006] Traditional methods usually use convolutional neural networks (CNN) or spiking neural networks (SNN) to achieve event stream-based tracking; however, convolutional neural networks lack modeling of global information and cannot capture the global timing information of event streams; spiking neural networks have the disadvantage of low overall performance.

[0007] In summary, existing event stream tracking methods have the following problems:

[0008] (1) Existing event data representations may introduce redundant information or lose the temporal and spatial attributes of event stream data;

[0009] (2) Existing event stream tracking methods find it difficult to make good use of event stream data.

[0010] Based on this, the present invention designs an event tracking method based on multi-scale knowledge distillation to solve the above problems. Summary of the invention

[0011] In view of the above-mentioned shortcomings of the prior art, the present invention provides an event tracking method based on multi-scale knowledge distillation.

[0012] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0013] An event tracking method based on multi-scale knowledge distillation includes the following steps:

[0014] S1. Input bimodal data to train the teacher network; wherein the bimodal data includes RGB frames and event streams;

[0015] S2. Input unimodal event stream data to train the student network, wherein the unimodal event stream data includes event streams; utilize the parameters of the teacher network through three-layer knowledge distillation to supervise and guide the learning of the student network that only inputs event stream data; specifically, the three-layer knowledge distillation includes knowledge distillation of similarity matrix, knowledge distillation of feature representation, and knowledge distillation of response mapping.

[0016] Furthermore, step S1 specifically includes the following steps:

[0017] a. The bimodal data is cut out from the initial frame and the subsequent frames to obtain the template and search area patches, and the projection layer is used to convert them into Token representations; these are then input into the Transformer backbone network;

[0018] b. Use multi-layer Transformer blocks as a unified Transformer backbone network to extract features and fuse the above tokens to obtain a trained teacher network;

[0019] c. The output of the Transformer backbone network is fed into the tracking head, which locates the bounding box of the target location and outputs the tracking result.

[0020] Furthermore, step S1 also includes a bimodal data fusion process, and the specific method is: along the number dimension of Token, RGB data and event stream data are spliced; and then input into the Transformer backbone network.

[0021] Furthermore, the event streams in step S1 and step S2 both include a preprocessing process, i.e., a process of dividing the event stream data, and the specific steps include: the source data stream size is T×X×Y×P;

[0022] Where T represents time, X and Y represent the spatial coordinates, and P represents the polarity of the event. The spatial position with an event is represented as 1, and the position without an event is marked as 0. The data is divided according to time t, where t represents the time interval, to form segments of size t×x×y, with a total of T / t segments.

[0023] Furthermore, the event stream in step S2 may be in the following forms: event frame, voxel grid, Time-Surface or reconstructed grayscale image.

[0024] Furthermore, the formula used in step b is:

[0025] MSA=concat(head1...headh ) 0 (1)

[0026] Among them, MSA represents the multi-head self-attention mechanism, head i represents the i-th head in the self-attention mechanism of the Transformer backbone network, w 0 represents the weight parameter;

[0027] head i =Attention(Q; K; V) (2)

[0028] Among them, Attention represents the self-attention mechanism in the Transformer backbone network, Q, K, and V represent the query, key, and value in the self-attention mechanism respectively;

[0029]

[0030] Among them, d k represents the feature dimension, and softmax represents the classification function.

[0031] Furthermore, in step S2, training the student network specifically includes the following steps:

[0032] a. The single-modal event stream data is cut out from the initial frame and subsequent frames to obtain the template and search area patches, and the projection layer is used to convert them into Token representations. These are then input into the Transformer backbone network in turn.

[0033] b. The Transformer backbone network extracts features and fuses the above tokens; the multi-head self-attention mechanism of the Transformer backbone network fully extracts the information in the frame;

[0034] c. The output of the Transformer backbone network is fed into the tracking head, which uses the tracking head to locate the bounding box of the target position and output the tracking result.

[0035] Furthermore, the loss function used for the distillation of the similarity matrix is ​​L2 loss, and the formula is:

[0036]

[0037] in, represents the similarity matrix of the jth layer of the student network, represents the similarity matrix of the i-th layer of the teacher network, represents the similarity distillation loss.

[0038] Furthermore, the loss function used for the distillation of feature representation is MSE loss, and the formula is:

[0039]

[0040] Among them, F t represents the feature representation of the teacher network, F s represents the feature representation of the student network, represents the feature distillation loss;

[0041] The loss function used for the distillation of the response map is Focal loss, and the formula is:

[0042]

[0043] Among them, P xy is the probability predicted by the model, represents the probability of the true value, Represents the response distillation loss.

[0044] Furthermore, the size of the number of tokens of the bimodal data in step S1 is twice the size of the number of tokens of the unimodal event stream data in step S2. Therefore, the Repeat method is used for the unimodal event stream data to align it with the bimodal data in terms of dimension.

[0045] Beneficial Effects

[0046] The present invention is based on a multi-level knowledge distillation strategy, including similarity matrix, feature representation and knowledge distillation based on response mapping, and guides the learning of the student Transformer network through a robust bimodal teacher Transformer network. Compared with the prior art, the present invention has the following advantages:

[0047] (1) The present invention uses multi-layer knowledge distillation to perform multi-layer supervision on the student network, which can improve the robustness of the target student network.

[0048] (2) The input of the student network of the present invention is unimodal event stream data, and through the guidance of the teacher network, performance improvement is achieved at a high reasoning speed.

[0049] (3) The present invention uses multi-layer Transformer blocks as a unified Transformer backbone network to establish long-range dependencies of temporal information, deeply mine the deep spatiotemporal information of video frames, and obtain better feature expression. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0051] Figure 1 is a flow chart of the present invention;

[0052] Figure 2 It is a schematic diagram of the model framework of the present invention; DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] The present invention will be further described below in conjunction with the embodiments.

[0055] Example 1

[0056] Please refer to the instruction manual Figure 1-2 , an event tracking method based on multi-scale knowledge distillation, comprising the following steps:

[0057] S1. Input bimodal data to train the teacher network; wherein the bimodal data includes RGB (red, green, and blue images captured by a traditional visible light camera) frames and event streams (asynchronous event information captured by an event camera); specifically, the following steps are included:

[0058] a. The bimodal data is cut out from the initial frame and subsequent frames to obtain the template patch blocks (image blocks of the same size divided from a picture) and the search area, and converted into Token (special marker of vector) representation using the projection layer; and then input into the Transformer backbone network;

[0059] b. Use multi-layer Transformer blocks as a unified Transformer backbone network to extract features and fuse the above tokens to obtain a trained teacher network;

[0060] c. The output of the Transformer backbone network is fed into the tracking head, which locates the bounding box of the target location and outputs the tracking result.

[0061] After the first stage of teacher network training is completed, the parameters learned by the teacher network are frozen and step S2 can begin.

[0062] S2, input unimodal event stream data, train the student network, and optimize it based on knowledge distillation;

[0063] Among them, the unimodal event stream data includes event stream (asynchronous event information captured by the event camera); it can be in various forms, such as: event frame, voxel grid, Time-Surface (a representation of event data stream, representing the surface composed of the latest event timestamp occurring on each pixel) or reconstructed grayscale image;

[0064] Among them, training the student network specifically includes the following steps:

[0065] a. The single-modal event stream data is cut out from the initial frame and subsequent frames to obtain the template and search area patches, and the projection layer is used to convert them into Token representations. These are then input into the Transformer backbone network in turn.

[0066] Since the size of the number of tokens of the bimodal data in step S1 is twice the size of the number of tokens of the unimodal event stream data in step S2, the Repeat method is used for the unimodal event stream data to align it with the bimodal data in terms of dimension.

[0067] b. The Transformer backbone network extracts features and fuses the above tokens; the multi-head self-attention mechanism of the Transformer backbone network is used to fully extract the information in the frame; as shown in formulas (1), (2) and (3):

[0068] MSA=concat(head1...head h ) 0 (1)

[0069] Among them, MSA represents the multi-head self-attention mechanism, head i represents the i-th head in the self-attention mechanism of the Transformer backbone network, w 0 represents the weight parameter;

[0070] head i =Attention(Q; K; V) (2)

[0071] Among them, Attention represents the self-attention mechanism in the Transformer backbone network, Q, K, and V represent the query, key, and value in the self-attention mechanism respectively;

[0072]

[0073] Among them, d k represents the feature dimension, and softmax represents the classification function;

[0074] c. The output of the Transformer backbone network is fed into the tracking head, which uses the tracking head to locate the bounding box of the target position and output the tracking result;

[0075] Due to the challenging factors of event-based tracking, such as sparse event points and cluttered background, the parameters of the teacher network are utilized through three layers of knowledge distillation to supervise and guide the learning of the student network which only inputs event stream data;

[0076] Specifically, the three-layer knowledge distillation includes knowledge distillation of similarity matrix, knowledge distillation of feature representation, and knowledge distillation of response mapping;

[0077] Specifically, in the cross-modal knowledge transfer of knowledge distillation, the knowledge distillation of the similarity matrix realizes the knowledge transfer from the similarity matrix learned by the teacher network to the student network. The loss function used in the distillation of the similarity matrix is ​​L2 (Euclidean distance) loss (Formula 4);

[0078]

[0079] in, represents the similarity matrix of the jth layer of the student network, represents the similarity matrix of the i-th layer of the teacher network, represents the similarity distillation loss;

[0080] Specifically, in the cross-modal knowledge transfer of knowledge distillation, the knowledge distillation of feature representation is to extract features from a powerful teacher network for feature representation distillation. The loss function used for feature representation distillation is MSE (mean square error) loss (Formula 5);

[0081]

[0082] Among them, F t represents the feature representation of the teacher network, F s represents the feature representation of the student network, represents the feature distillation loss;

[0083] Specifically, the tracking result is output by the tracking head, and the output response map is used to locate the target. Obviously, if the student network can directly simulate the response map output by the teacher network, the result will be better. Therefore, the distillation of the response map is used to achieve this. The loss function used for the distillation of the response map is the Focal loss (Formula 6);

[0084]

[0085] Among them, P xy is the probability predicted by the model, represents the probability of the true value, Represents the response distillation loss.

[0086] The present invention is based on a multi-level knowledge distillation strategy, including similarity matrix, feature representation and knowledge distillation based on response mapping, and guides the learning of the student Transformer network through a robust bimodal teacher Transformer network. Compared with the prior art, the present invention has the following advantages:

[0087] (1) The present invention uses multi-layer knowledge distillation to perform multi-layer supervision on the student network, which can improve the robustness of the target student network.

[0088] (2) The input of the student network of the present invention is unimodal event stream data, and through the guidance of the teacher network, performance improvement is achieved at a high reasoning speed.

[0089] (3) The present invention uses multi-layer Transformer blocks as a unified Transformer backbone network to establish long-range dependencies of temporal information, deeply mine the deep spatiotemporal information of video frames, and obtain better feature expression.

[0090] Example 2

[0091] On the basis of Example 1, step S1 also includes a bimodal data fusion process, and the specific method is: along the number dimension of Token, RGB data and event stream data are spliced; and then input into the Transformer backbone network.

[0092] Example 3

[0093] On the basis of Example 2, the event streams in step S1 and step S2 both include a preprocessing process, that is, a process of dividing the event stream data, and the specific steps include: the size of the source data stream is T×X×Y×P;

[0094] Where T represents time, X and Y represent spatial coordinates, and P represents the polarity of the event. The spatial position with an event is represented as 1, and the position without an event is marked as 0. The segment is divided according to time t (t represents the time interval) to form segments of size t×x×y, with a total of T / t segments.

[0095] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An event tracking method based on multi-scale knowledge distillation, characterized in that: The following steps are involved: S1. Input bimodal data to train the teacher network; wherein the bimodal data includes RGB frames and event streams; S2. Input unimodal event stream data to train the student network, wherein the unimodal event stream data includes event streams; utilize the parameters of the teacher network through three-layer knowledge distillation to supervise and guide the learning of the student network that only inputs event stream data; specifically, the three-layer knowledge distillation includes knowledge distillation of similarity matrix, knowledge distillation of feature representation, and knowledge distillation of response mapping.

2. The event tracking method based on multi-scale knowledge distillation according to claim 1 is characterized in that: Step S1 specifically includes the following steps: a. The bimodal data is cut out from the initial frame and the subsequent frames to obtain the template and search area patches, and the projection layer is used to convert them into Token representations; these are then input into the Transformer backbone network; b. Use multi-layer Transformer blocks as a unified Transformer backbone network to extract features and fuse the above tokens to obtain a trained teacher network; c. The output of the Transformer backbone network is fed into the tracking head, which locates the bounding box of the target location and outputs the tracking result.

3. The event tracking method based on multi-scale knowledge distillation according to claim 2 is characterized in that: Step S1 also includes a bimodal data fusion process, the specific method is: along the number dimension of Token, RGB data and event stream data are spliced; and then input into the Transformer backbone network.

4. The event tracking method based on multi-scale knowledge distillation according to any one of claims 1 to 3, characterized in that: The event streams in step S1 and step S2 both include a preprocessing process, i.e., a process of dividing the event stream data, and the specific steps include: the source data stream size is T×X×Y×P; Where T represents time, X and Y represent the spatial coordinates, and P represents the polarity of the event. The spatial position with an event is represented as 1, and the position without an event is marked as 0. The data is divided according to time t, where t represents the time interval, to form segments of size t×x×y, with a total of T / t segments.

5. The event tracking method based on multi-scale knowledge distillation according to claim 4 is characterized in that: The event stream in step S2 may be in the following forms: event frame, voxel grid, Time-Surface or reconstructed grayscale image.

6. The event tracking method based on multi-scale knowledge distillation according to claim 4, characterized in that: The formula used in step b is: MSA=concat(head1...head h )w 0 (1) Among them, MSA represents the multi-head self-attention mechanism, head i represents the i-th head in the self-attention mechanism of the Transformer backbone network, w 0 represents the weight parameter; head i =Attention(Q;K;V) (2) Among them, Attention represents the self-attention mechanism in the Transformer backbone network, Q, K, and V represent the query, key, and value in the self-attention mechanism respectively; Among them, d k represents the feature dimension, and softmax represents the classification function.

7. The event tracking method based on multi-scale knowledge distillation according to claim 6, characterized in that: In step S2, training the student network specifically includes the following steps: a. The single-modal event stream data is cut out from the initial frame and subsequent frames to obtain the template and search area patches, and the projection layer is used to convert them into Token representations. These are then input into the Transformer backbone network in turn. b. The Transformer backbone network extracts features and fuses the above tokens; the multi-head self-attention mechanism of the Transformer backbone network fully extracts the information in the frame; c. The output of the Transformer backbone network is fed into the tracking head, which uses the tracking head to locate the bounding box of the target position and output the tracking result.

8. The event tracking method based on multi-scale knowledge distillation according to claim 7, characterized in that: The loss function used for the distillation of the similarity matrix is ​​L2 loss, and the formula is: in, represents the similarity matrix of the jth layer of the student network, represents the similarity matrix of the i-th layer of the teacher network, represents the similarity distillation loss.

9. The event tracking method based on multi-scale knowledge distillation according to claim 8, characterized in that: The loss function used for the distillation of feature representation is MSE loss, and the formula is: Among them, F t represents the feature representation of the teacher network, F s represents the feature representation of the student network, represents the feature distillation loss; The loss function used for the distillation of the response map is Focal loss, and the formula is: Among them, P xy is the probability predicted by the model, represents the probability of the true value, Represents the response distillation loss.

10. The event tracking method based on multi-scale knowledge distillation according to claim 3, characterized in that: The size of the number of tokens in the bimodal data in step S1 is twice the size of the number of tokens in the unimodal event stream data in step S2, so the Repeat method is used for the unimodal event stream data to align it with the bimodal data in terms of dimension.

Citation Information

Patent Citations

  • Teacher and student network target tracking method and system based on double knowledge distillation

    CN116883459A

Cited By

  • Knowledge distillation method, electronic equipment and computer readable storage medium

    CN120611768A