A multi-target tracking method and system based on XFormer neural network

Abstract

This invention relates to the field of target tracking technology and provides a multi-target tracking method and system based on an XFormer neural network. The method includes the following steps: extracting frame-level features from the image input to the XFormer neural network using a CNN neural network, performing dimensionality reduction using one-dimensional convolution, and fusing these features with the corresponding spatial location encoding. The XFormer neural network consists of a CNN neural network and a TransFormer, with the TransFormer comprising an Encoder and a Decoder. The frame-level features extracted by the CNN neural network and the spatial location encoding are fed into the multi-layer Encoder part of the TransFormer, compressing the input sequence into a vector of a specified length, and then feeding it into the Decoder part. Candidate tracking categories and location information are obtained through multi-layer Decoder decoding. The final category and location information are obtained through two fully connected layers. The XFormer neural network in this invention can be deployed in portable devices with good real-time performance and can perform multi-target tracking.

Description

Technical Field

[0001] This invention relates to the field of target tracking technology, specifically to a multi-target tracking method and system based on the XFormer neural network. Background Technology

[0002] Multi-object tracking (MOT) aims to correlate target objects across video frames to obtain their entire motion trajectory. With the development of deep neural networks and the increasing demand for intelligent video analysis, MOT has received widespread attention in computer vision research and industry, and has become a core technology in many commercial applications. For example, in public places such as train stations, banks, shopping malls, and hotels, intelligent monitoring systems using MOT technology can analyze abnormal pedestrian behavior and calculate pedestrian flow, providing excellent early warning in emergency situations. In the field of smart city transportation, analyzing and processing online surveillance videos to obtain traffic density, congestion rates, etc., can improve effective traffic scheduling schemes. At the same time, law enforcement officers can use the behavioral trajectories of vehicle driving violations and pedestrian traffic violations as evidence of illegal behavior to reduce traffic violations. In addition, MOT has wide applications in military fields, drone patrols, autonomous driving, and live sports broadcasting.

[0003] Multi-object tracking is a challenging task, requiring simultaneous trajectory initialization, object recognition, and spatiotemporal trajectory inference. To date, TrackFormer, a trainable multi-object tracking method based on the encoder-decoder Transformer architecture, has achieved state-of-the-art performance on the MOT17 and MOTS20 datasets. However, TrackFormer's massive computational cost makes it unsuitable for edge deployment and difficult to implement in portable devices requiring real-time performance.

[0004] Therefore, there is a need to provide a multi-target tracking method and system based on XFormer neural networks, which aims to solve the above problems. Summary of the Invention

[0005] In view of the shortcomings of the existing technology, the purpose of this invention is to provide a multi-target tracking method and system based on XFormer neural network to solve the problems existing in the background technology.

[0006] This invention is implemented as follows: a multi-target tracking method based on an XFormer neural network, the method comprising the following steps:

[0007] The XFormer neural network extracts frame-level features from the image input to the XFormer neural network using a CNN neural network, performs dimensionality reduction using one-dimensional convolution, and fuses these features with the spatial location encoding corresponding to the image. The XFormer neural network consists of a CNN neural network and a TransFormer, with the TransFormer consisting of an Encoder and a Decoder.

[0008] The frame-level features and spatial location codes extracted by the CNN neural network are fed into the multi-layer Encoder part of TransFormer to compress the input sequence into a vector of a specified length. This vector serves as the semantics of the input sequence and is then fed into the Decoder part.

[0009] Candidate tracking categories and location information are obtained through multi-layer decoder decoding; the final category and location information are obtained through two fully connected layers.

[0010] As a further aspect of the present invention: during the input to the Decoder part, a track query is introduced to track the target in an autoregressive manner. The track query is generated by the DETR detector and integrates the position information of the corresponding target over time.

[0011] As a further aspect of the present invention: the Decoder part of the Transformer adjusts the track query between frames to follow the changes in the target position, so that TrackFormer achieves a seamless inter-frame data association in the paradigm of tracking-by-attention. The attention mechanism ensures that the model considers the recognition features of position, occlusion and target at the same time.

[0012] As a further aspect of the present invention: the method also provides an XFA operator, which calculates the attention mapping along the feature dimension D by constructing an intermediate query context and feature scores to reduce computational overhead; the XFA is adjusted and normalized using a learnable parameter, and the normalization operation limits the XFA value to a certain range.

[0013] As a further aspect of this invention: when using XFA, Q and K on the feature dimension D are first L2 standardized, and the standardization formula is: Construct two intermediate scores for K: and Using two convolution kernels and Calculate along the token dimension N and feature dimension D and Will and Represented as: and Define XFA as: Here, λ is a temperature parameter that can dynamically adjust the scaling factor of different TransFormer layers to improve training stability.

[0014] Another object of the present invention is to provide a multi-target tracking system based on the XFormer neural network, the system comprising:

[0015] The frame-level feature extraction module is used to extract frame-level features from the image input to the XFormer neural network through the CNN neural network, perform dimensionality reduction using one-dimensional convolution, and fuse them with the spatial location code corresponding to the image. The XFormer neural network consists of a CNN neural network and a TransFormer, and the TransFormer consists of two parts: an Encoder and a Decoder.

[0016] The encoding and decoding processing module is used to input the frame-level features and spatial location codes extracted by the CNN neural network into the multi-layer encoder part of TransFormer, compress the input sequence into a vector of a specified length, the vector serving as the semantics of the input sequence, and input it into the decoder part;

[0017] The target location tracking module is used to obtain candidate tracking categories and location information through multi-layer decoder decoding processing; and to obtain the final category and location information through two fully connected layers.

[0018] As a further aspect of the present invention: the system further includes a video input module, an on-chip SoC, a storage module, an XFormer IP, a VDMA module, and a serial port module. The storage module communicates with the FPGA for high-throughput, high-speed data communication via the AXI protocol. The VDMA module buffers the data stream, and the extracted image data, features, and weight data from the storage module are fed into the XFormer neural network for calculation. After the calculation is completed, the result is returned to the on-chip SoC. Finally, the serial port module communicates with the host computer.

[0019] Compared with the prior art, the beneficial effects of the present invention are:

[0020] This invention extracts frame-level features from images input to an XFormer neural network using a CNN neural network, performs dimensionality reduction using one-dimensional convolution, and fuses these features with the corresponding spatial location codes. The frame-level features extracted by the CNN and the spatial location codes are then fed into the multi-layer encoder of a TransFormer, compressing the input sequence into a vector of a specified length. This vector serves as the semantics of the input sequence and is then fed into the decoder. Next, a multi-layer decoder process is used to obtain candidate tracking categories and location information. Finally, two fully connected layers are used to obtain the final category and location information. The XFormer neural network can be deployed in portable devices with good real-time performance and can perform multi-target tracking. Attached Figure Description

[0021] Figure 1 This is a flowchart of a multi-target tracking method based on the XFormer neural network.

[0022] Figure 2 This is an architecture diagram of a multi-target tracking method based on the XFormer neural network.

[0023] Figure 3 This is an architecture diagram of Transformer in a multi-target tracking method based on XFormer neural network.

[0024] Figure 4 This is the original image used in a multi-target tracking method based on the XFormer neural network.

[0025] Figure 5 This is a tracking effect image from a multi-target tracking method based on the XFormer neural network. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0027] The specific implementation of the present invention will be described in detail below with reference to specific embodiments.

[0028] like Figure 1 and Figure 2 As shown in the figure, CNN represents a convolutional neural network, Encoder and Decoder represent the encoder and decoder respectively, and Positional encoding represents spatial encoding. This invention provides a multi-target tracking method based on the XFormer neural network, the method comprising the following steps:

[0029] S100: The image input to the XFormer neural network is extracted at the frame level through the CNN neural network, dimensionality is reduced by one-dimensional convolution, and fused with the spatial location code corresponding to the image. The XFormer neural network is composed of a CNN neural network and a TransFormer. The TransFormer is composed of two parts: an Encoder and a Decoder.

[0030] S200: The frame-level features and spatial location codes extracted by the CNN neural network are fed into the multi-layer Encoder part of TransFormer to compress the input sequence into a vector of a specified length. The vector serves as the semantics of the input sequence and is then fed into the Decoder part.

[0031] The S300 obtains candidate tracking categories and location information through multi-layer decoder decoding; and obtains the final category and location information through two fully connected layers.

[0032] It should be noted that, thanks to the deep mining of image features, the accuracy of deep learning-based image processing technology has been greatly improved. TrackFormer is a MOT work based on Transformer, achieving state-of-the-art performance on MOT17 and MOTS20. However, TrackFormer is not suitable for deployment at the edge due to its huge computational cost. This invention provides a novel, efficient, and lightweight hybrid neural network, XFormer.

[0033] XFormer neural networks can be deployed in portable devices with good real-time performance to track multiple targets. FPGAs, with their outstanding hardware acceleration and parallel processing capabilities, solve the problem of high-speed computation. For example, the ZYNQ7000 integrates an ARM processor and an FPGA system to build a PS+PL heterogeneous platform. By mapping the neural network into the logic resources of the FPGA through algorithm mapping, efficient hardware acceleration can be achieved.

[0034] In this embodiment of the invention, the XFormer neural network is composed of a CNN neural network and a TransFormer. The TransFormer consists of an Encoder and a Decoder. The CNN neural network extracts frame-level features from the image input to the XFormer neural network, performs dimensionality reduction using one-dimensional convolution, and fuses these features with the spatial location encoding corresponding to the image. Then, the frame-level features extracted by the CNN neural network and the spatial location encoding are fed into the multi-layer Encoder part of the TransFormer to compress the input sequence into a vector of a specified length. This vector serves as the semantics of the input sequence. This process is called encoding and is then fed into the Decoder part. During the process of feeding the input to the Decoder part, a track query is introduced to track the target in an autoregressive manner. The track query is generated by the DETR detector and integrates the position information of the corresponding target over time. The Decoder part of the Transformer adjusts the track query between frames to achieve the purpose of following the changes in the target position. This allows TrackFormer to achieve seamless inter-frame data association in a new tracking-by-attention paradigm. The attention mechanism ensures that the model considers the recognition features of position, occlusion, and target simultaneously.

[0035] like Figure 3 , Figure 4 and Figure 5 As shown in the preferred embodiment of the present invention, the method also provides XFA (cross-feature attention), which can effectively solve the quadratic complexity problem of self-attention. XFA calculates the attention mapping along the feature dimension D by constructing intermediate query context and feature scores to reduce computational overhead. The XFA uses a learnable parameter for adjustment and normalization, which limits the XFA value to a certain range. XFA uses a learnable parameter for adjustment and normalization and does not require a Softmax operation. Unlike the quadratic complexity of the original self-attention algorithm, XFA reduces the computational cost from O(N^2) to O(N^2). 2 D) decreased to O(ND) 2 ), while O(ND 2 It is only linearly related to N.

[0036] In this embodiment of the invention, when using XFA, Q and K on the feature dimension D are first L2 standardized, and the standardization formula is: QK in self-attention TDirect calculation would result in unnecessary redundancy and computational overhead. In this embodiment of the invention, two intermediate scores are constructed for K: and Using two convolution kernels and Calculate along the token dimension N and feature dimension D and By using convolutional kernels, the intermediate vectors can more compactly represent the computation of attention maps, while also reducing computational overhead. and Represented as: and Define XFA as: Here, λ is a temperature parameter that can dynamically adjust the scaling factor of different TransFormer layers to improve training stability.

[0037] In this embodiment of the invention, the process of profiling the XFA operator can be divided into three stages. In the first stage, image data and parameters for calculating the KQV matrix are cached on-chip using a row cache structure, and then parallel computation is achieved through multiple parallel multipliers and an accumulation tree structure. In the second stage, two convolutional kernels are retrieved from the on-chip cache. and Matrix operations are implemented using multiple multipliers and an accumulator tree structure. In the third stage, Q, K′, and V data are input. The Q and K′ data are first processed by a divider, and the results are then processed by a multiplier and λ. The V data is first stored in a FIFO. After the quotient of Q and K′ is calculated with λ, the read signal is obtained and then multiplied with the V data to obtain the final output result.

[0038] This invention also provides a multi-target tracking system based on the XFormer neural network, the system comprising:

[0039] The frame-level feature extraction module is used to extract frame-level features from the image input to the XFormer neural network through the CNN neural network, perform dimensionality reduction using one-dimensional convolution, and fuse them with the spatial location code corresponding to the image. The XFormer neural network consists of a CNN neural network and a TransFormer, and the TransFormer consists of two parts: an Encoder and a Decoder.

[0040] The encoding and decoding processing module is used to input the frame-level features and spatial location codes extracted by the CNN neural network into the multi-layer encoder part of TransFormer, compress the input sequence into a vector of a specified length, the vector serving as the semantics of the input sequence, and input it into the decoder part;

[0041] The target location tracking module is used to obtain candidate tracking categories and location information through multi-layer decoder decoding processing; and to obtain the final category and location information through two fully connected layers.

[0042] In this embodiment of the invention, the system further includes a video input module, an on-chip SoC, a storage module, an XFormer IP, a VDMA module, and a serial port module. The storage module communicates with the FPGA for high-throughput, high-speed data via the AXI protocol. The VDMA module buffers the data stream, and the extracted image data, features, and weight data from the storage module are fed into the XFormer neural network for calculation. After the calculation is completed, the result is returned to the on-chip SoC. Finally, the serial port module communicates with the host computer. This embodiment of the invention uses a multi-level pipelined circuit design to meet the requirements of real-time target tracking.

[0043] The above description only details the preferred embodiments of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0044] It should be understood that although the steps in the flowcharts of the various embodiments of the present invention are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the various embodiments may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0045] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0046] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the disclosure in the specification and embodiments. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

Claims

1. A multi-target tracking method based on XFormer neural network, characterized in that, The method includes the following steps: The XFormer neural network extracts frame-level features from the image input to the XFormer neural network using a CNN neural network, performs dimensionality reduction using one-dimensional convolution, and fuses these features with the spatial location encoding corresponding to the image. The XFormer neural network consists of a CNN neural network and a TransFormer, with the TransFormer consisting of an Encoder and a Decoder. The frame-level features and spatial location codes extracted by the CNN neural network are fed into the multi-layer Encoder part of TransFormer to compress the input sequence into a vector of a specified length. This vector serves as the semantics of the input sequence and is then fed into the Decoder part. Candidate tracking categories and location information are obtained through multi-layer decoder decoding; the final category and location information are obtained through two fully connected layers.

2. The multi-target tracking method based on XFormer neural network according to claim 1, characterized in that, During the input to the Decoder, a track query is introduced to track the target in an autoregressive manner. The track query is generated by the DETR detector and integrates the corresponding target's location information over time.

3. The multi-target tracking method based on XFormer neural network according to claim 2, characterized in that, The Decoder part of the Transformer adjusts the track query between frames to follow the changes in the target position. This enables TrackFormer to achieve seamless inter-frame data association using the tracking-by-attention paradigm. The attention mechanism ensures that the model considers the location, occlusion, and target recognition features simultaneously.

4. The multi-target tracking method based on XFormer neural network according to claim 1, characterized in that, The method also provides an XFA operator, which calculates the attention mapping along the feature dimension D by constructing an intermediate query context and feature scores to reduce computational overhead; the XFA is adjusted and normalized using a learnable parameter, and the normalization operation limits the XFA value to a certain range.

5. The multi-target tracking method based on XFormer neural network according to claim 4, characterized in that, When using XFA, Q and K on the feature dimension D are first L2 standardized. The standardization formula is as follows: Construct two intermediate scores for K: and Using two convolution kernels and Calculate along the token dimension N and feature dimension D and Will and Represented as: and Define XFA as: Here, λ is a temperature parameter that can dynamically adjust the scaling factor of different TransFormer layers to improve training stability.

6. A multi-target tracking system based on XFormer neural network, characterized in that, The system includes: The frame-level feature extraction module is used to extract frame-level features from the image input to the XFormer neural network through the CNN neural network, perform dimensionality reduction using one-dimensional convolution, and fuse them with the spatial location code corresponding to the image. The XFormer neural network consists of a CNN neural network and a TransFormer, and the TransFormer consists of two parts: an Encoder and a Decoder. The encoding and decoding processing module is used to input the frame-level features and spatial location codes extracted by the CNN neural network into the multi-layer encoder part of TransFormer, compress the input sequence into a vector of a specified length, the vector serving as the semantics of the input sequence, and input it into the decoder part; The target location tracking module is used to obtain candidate tracking categories and location information through multi-layer decoder decoding processing; and to obtain the final category and location information through two fully connected layers.

7. The multi-target tracking system based on the XFormer neural network according to claim 6, characterized in that, The system also includes a video input module, an on-chip SoC, a storage module, an XFormer IP, a VDMA module, and a serial port module. The storage module communicates with the FPGA for high-throughput, high-speed data via the AXI protocol. The VDMA module buffers the data stream, and the extracted image data, features, and weight data from the storage module are fed into the XFormer neural network for calculation. After the calculation is completed, the result is returned to the on-chip SoC. Finally, the serial port module communicates with the host computer.

Images