Television infrared composite image target tracking method and system based on FPGA (Field Programmable Gate Array)

Abstract

The invention discloses a television infrared composite image target tracking method and system based on an FPGA (Field Programmable Gate Array). The method comprises the following steps: acquiring television and infrared double-channel video stream images, cross-clock domain synchronous frame frequency and size in real time; hLS high-level synthesis is carried out, a television infrared multimode feature enhancement converter tracking model is realized, feature extraction, fusion and prediction of multimode images are completed, and a target tracking result is obtained; superposing a tracking frame, and outputting a video stream in real time; according to the invention, the problems of low tracking precision, poor robustness, slow operation speed and difficulty in real-time tracking of an edge device end target tracking algorithm in complex and diverse environments such as illumination variation, night environment and target shielding in the prior art can be solved.

Description

Technical Field

[0001] The invention belongs to the field of image target tracking, and in particular relates to a television infrared composite image target tracking method and system based on FPGA. Background Art

[0002] Target tracking is one of the main research directions in the field of computer vision. Its basic task is to give the initial information of the tracked target in a video image sequence and continue to track the target in subsequent images. Modern intelligent weapons and equipment such as unmanned reconnaissance aircraft, cruise missiles, and satellite detection devices have built-in image intelligent processing modules to track hostile targets. In daily civilian use, more and more imaging devices are embedded with image tracking algorithms to achieve monitoring and alarm purposes. As the focus and difficulty in the field of image processing, target tracking has great uses in daily production and life, and plays an important role in various occasions.

[0003] Most existing edge device target tracking methods use a single detection system. However, since the target tracking process is easily affected by lighting changes, nighttime environments, and occlusions, the tracking accuracy of the algorithm is significantly reduced. Accurate and robust tracking of targets in complex and diverse environments is still a major challenge. At the same time, most edge device algorithms are deployed on the CPU, making it difficult to achieve real-time recognition of targets in the video. Methods based on GPU (Graphics Processing Unit) acceleration can improve the recognition speed, but the high power consumption makes it difficult to apply on edge devices. Summary of the invention

[0004] In view of this, the present invention provides a TV infrared composite image target tracking method and system based on FPGA, which can solve the problems of low tracking accuracy, poor robustness and slow running speed of the edge device-side target tracking algorithm in complex and diverse environments such as lighting changes, night environments and target occlusion, making it difficult to track in real time.

[0005] The technical solution for implementing the present invention is as follows:

[0006] A television infrared composite image target tracking method based on FPGA, comprising:

[0007] Real-time acquisition of TV and infrared dual-channel video stream images, synchronizing frame rate and size across clock domains;

[0008] HLS high-level synthesis realizes the TV infrared multi-mode feature enhancement converter tracking model, completes the feature extraction, fusion and prediction of multi-mode images, and obtains the target tracking result;

[0009] Overlay the tracking frame and output the video stream in real time.

[0010] Furthermore, the HLS high-level synthesis realizes the TV infrared multi-mode feature enhancement converter tracking model, completes the feature extraction, fusion and prediction of the multi-mode image, and obtains the target tracking result, which specifically includes:

[0011] The TV infrared multi-mode feature enhancement converter tracking model is used to extract TV infrared features and fully explore the complementary information of TV images and infrared images; infrared images are less affected by lighting conditions and can capture target information in night environments and occlusions, while TV images are difficult to detect targets at night; TV images can provide information such as texture and color to help distinguish similar targets, while it is difficult to distinguish similar targets in infrared images.

[0012] Furthermore, the TV infrared multi-mode feature enhancement converter tracking model structure is divided into three parts: feature extraction network, multi-modal feature fusion network, and prediction network;

[0013] The input of the feature extraction network is the TV template image T tv , infrared template image T ir , TV search area image S tV and infrared search area image S ir The template image is the H image cropped with the target position of the first frame of the video clip as the center. t ×V t The image of the i-th frame search area is cropped at the i-th frame image with the target position of the i-1 frame as the center. s ×W s The image size is 20000; the fine-tuned ResNet50 is used as the feature extraction backbone network;

[0014] The multimodal feature fusion network first performs channel adjustment and reconstruction on the output results of the feature extraction network to obtain the corresponding visible light template vector V VT , infrared template vector V IT , visible light search area vector V VS and the infrared search area vector V IS ;

[0015] The two branch features are fused using a feature fusion network based on the Transformer converter structure. The correlation calculation operation is a local linear matching process that does not consider global information and is prone to fall into local optimality. Compared with the correlation calculation, the feature fusion network of the Transformer converter structure can establish global feature associations, adaptively focus on important information, and extract sufficient semantic information to improve the calculation accuracy of subsequent regression branches and classification branches, and output more accurate tracking results.

[0016] The feature fusion network structure mainly includes N feature fusion layers and a cross feature enhancement module. The feature fusion layer consists of a self-feature enhancement module and a cross feature enhancement module. The self-feature enhancement module X0 is expressed as:

[0017] X0=X+MultiHead(X+P,X+P,X)

[0018] Where X is the input vector and P is the corresponding spatial position code. The residual connection multi-head attention is used to integrate the information of different positions in the feature map, and different responses are given to different spatial positions, so that the network can distinguish the foreground from the background more effectively. The cross-feature enhancement module in the feature fusion layer takes the vectors of the template branch and the search area branch as input, where the main branch vector is used as the query vector and the other branch vector is used as the key vector and value vector. After calculation by N feature fusion layers, the vector of the search area branch is used as the query vector, and the template branch vector is used as the key vector and value vector to input into the final cross-feature enhancement module to obtain the final fusion vector d is the dimension of the fusion vector.

[0019] The prediction network consists of a regression branch and a classification branch, each of which is a multilayer perceptron with three linear transformation layers and one nonlinear activation layer. The fusion vector is mapped into a regression vector after being calculated by the regression branch and the classification branch. and classification vector The value of the first dimension of the regression vector corresponds to the coordinate information of the target bounding box, and the value of the first dimension of the classification vector is the foreground and background scores of the area corresponding to the vector.

[0020] Furthermore, the TV infrared multi-mode feature enhancement converter tracking model is implemented by HLS high-level synthesis method and integrated into FPGA in the form of IP; specifically, the TV infrared multi-mode feature enhancement converter tracking model IP consists of feature extraction module, multi-modal feature fusion module and prediction module. The AXI bus interface is used for data transmission between modules, so as to achieve data pipeline operation.

[0021] Among them, HLS is developed in C++, and high-level synthesis uses function parallelization optimization to reduce processing cycles; pipelines are used to implement internal parallel optimization of for loops to reduce processing cycles; dataflow optimization reduces the occupancy of variable FIFO to significantly reduce resources.

[0022] A television infrared composite image target tracking system based on FPGA, comprising: a television camera, an infrared camera, a video acquisition synchronization module, a hardware tracking acceleration module and a display module;

[0023] The television camera is used to collect visible light video image data in real time;

[0024] The infrared camera is used to collect infrared video image data in real time;

[0025] The video acquisition synchronization module is used to acquire the data transmitted by the TV camera and the infrared camera, and synchronize the frame rate and size;

[0026] The hardware tracking acceleration module runs a target tracking algorithm to obtain target coordinate information;

[0027] The display module is used to display the target tracking result.

[0028] Beneficial effects:

[0029] 1. In the scheme of the present invention, the complementary advantages of television and infrared image detection are utilized to solve the problem that the target tracking process is easily affected by illumination changes, night environment and occlusion.

[0030] 2. The present invention utilizes a multi-mode feature enhanced converter tracking model to fuse two modal data, effectively improving the joint characterization capability of multi-modal data and improving tracking accuracy.

[0031] 3. The present invention accelerates the entire algorithm process on FPGA to achieve real-time and robust tracking of targets in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 The figure is a flow chart of the method of the present invention.

[0033] Figure 2 It is a schematic diagram of the feature fusion process of the present invention.

[0034] Figure 3 Schematic diagram of the system of the present invention. DETAILED DESCRIPTION

[0035] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0036] The present invention aims to solve the problems in the prior art of edge device-side target tracking algorithms, such as low tracking accuracy, poor robustness, slow running speed, and difficulty in real-time tracking under complex and diverse environments such as illumination changes, night environments, and target occlusion. A television infrared composite image target tracking method based on FPGA is provided.

[0037] Embodiment 1

[0038] like Figure 1 As shown, the television infrared composite image target tracking method based on FPGA provided by the embodiment of the present invention includes:

[0039] S101, real-time acquisition of TV and infrared dual-channel video stream images, synchronization of frame rate and size across clock domains;

[0040] S102, HLS high-level synthesis, realizes the TV infrared multi-mode feature enhancement converter tracking model, completes the feature extraction, fusion and prediction of multi-mode images, and obtains the target tracking result;

[0041] S103, superimpose the tracking frame and output the video stream in real time.

[0042] The embodiment of the present invention collects TV and infrared dual-channel video stream images in real time, synchronizes frame rate and size across clock domains; implements HLS high-level synthesis to realize TV infrared multi-mode feature enhancement converter tracking model, completes feature extraction, fusion and prediction of multi-mode images, and obtains target tracking results; superimposes tracking frames and outputs video streams in real time. In this way, the complementary advantages of TV and infrared image detection are utilized to solve the problem that the target tracking process is easily affected by lighting changes, night environment and occlusion; the multi-mode feature enhancement converter tracking model is used to fuse the two modal data, effectively improving the joint characterization capability of multi-modal data and improving tracking accuracy; the entire algorithm process is accelerated on FPGA to achieve real-time robust tracking of targets in complex environments.

[0043] In order to better understand the television infrared composite image target tracking method based on FPGA according to the embodiment of the present invention, a detailed description is given. The method may specifically include:

[0044] S101, real-time acquisition of TV and infrared dual-channel video stream images, synchronization of frame rate and size across clock domains;

[0045] In this embodiment, the acquisition of television and infrared dual-channel video stream image data uses an asynchronous FIFO to synchronize with the bus clock, and the line valid and frame valid signals are used to mark the data position;

[0046] The cross-clock domain synchronization frame frequency introduces an external 100MHz clock into the clock management MMCM and multiplies it to 200MHz, dividing it into three related clock domains to drive the underlying data processing modules; the AXI4 bus and the top-level state machine are driven by a 100MHz clock provided by the on-chip ARM system, which uses a synchronous method for accessing the register table, and an asynchronous method for large data transmission;

[0047] The synchronization size uses bilinear interpolation to scale the television image to the same size as the infrared image in real time;

[0048] S102, HLS high-level synthesis, realizes the TV infrared multi-mode feature enhancement converter tracking model, completes the feature extraction, fusion and prediction of multi-mode images, and obtains the target tracking results, such as Figure 2 As shown;

[0049] In this embodiment, the TV infrared multi-mode feature enhancement converter tracking model structure is divided into three parts: feature extraction network, multi-modal feature fusion network, and prediction network;

[0050] The TV infrared multi-mode feature enhancement converter tracking model is used to extract TV infrared features and fully exploit the complementary information of TV images and infrared images. Infrared images are less affected by lighting conditions and can capture target information in night environments and occlusions, while TV images are difficult to detect targets at night; TV images can provide information such as texture and color to help distinguish similar targets, while infrared images are difficult to distinguish similar targets.

[0051] The input of the feature extraction network is the TV template image T tv , infrared template image T ir , TV search area image S tV and infrared search area image S ir The template image is the H image cropped with the target position of the first frame of the video clip as the center. t ×V t The image of the i-th frame search area is cropped at the i-th frame image with the target position of the i-1 frame as the center. s ×W s The fine-tuned ResNet50 is used as the feature extraction backbone network.

[0052] The multimodal feature fusion network first performs channel adjustment and reconstruction on the output results of the feature extraction network to obtain the corresponding visible light template vector V VT , infrared template vector V IT , visible light search area vector V VS and the infrared search area vector V IS .

[0053] The feature fusion network based on the Transformer converter structure is used to fuse the two branch features after fusion. The correlation calculation operation is a local linear matching process that does not consider global information and is prone to fall into local optimality. Compared with the correlation calculation, the feature fusion network of the Transformer converter structure can establish global feature associations and adaptively focus on important information, and extract sufficient semantic information to improve the calculation accuracy of subsequent regression branches and classification branches, and output more accurate tracking results.

[0054] The feature fusion network structure mainly includes N feature fusion layers and a cross feature enhancement module. The feature fusion layer consists of a self-feature enhancement module and a cross-feature enhancement module. The self-feature enhancement module can be expressed as:

[0055] X0=X+MultiHead(X+P,X+P,X)

[0056] Where X is the input vector and P is the corresponding spatial position code. The residual connection multi-head attention is used to integrate the information of different positions in the feature map, and different responses are given to different spatial positions, so that the network can distinguish the foreground from the background more effectively. The cross-feature enhancement module in the feature fusion layer takes the vectors of the template branch and the search area branch as input, where the main branch vector is used as the query vector and the other branch vector is used as the key vector and value vector. After calculation by N feature fusion layers, the vector of the search area branch is used as the query vector, and the template branch vector is used as the key vector and value vector to input into the final cross-feature enhancement module to obtain the final fusion vector d is the dimension of the fusion vector.

[0057] The prediction network consists of a regression branch and a classification branch, each of which is a multilayer perceptron with three linear transformation layers and one nonlinear activation layer. The fusion vector is mapped into a regression vector after being calculated by the regression branch and the classification branch. and classification vector The value of the first dimension of the regression vector corresponds to the coordinate information of the target bounding box, and the value of the first dimension of the classification vector is the foreground and background scores of the area corresponding to the vector.

[0058] The TV infrared multi-mode feature enhancement converter tracking model is implemented by HLS high-level synthesis method and integrated into FPGA in the form of IP; specifically, the TV infrared multi-mode feature enhancement converter tracking model IP consists of feature extraction module, multi-modal feature fusion module and prediction module. The AXI bus interface is used for data transmission between modules, so as to achieve data pipeline operation.

[0059] Among them, HLS is developed in C++, and high-level synthesis uses function parallelization optimization to reduce processing cycles; pipelines are used to implement internal parallel optimization of for loops to reduce processing cycles; dataflow optimization reduces the occupancy of variable FIFO to significantly reduce resources.

[0060] S103, superimpose the tracking frame and output the video stream in real time.

[0061] In this embodiment, a picture frame is superimposed on the input video stream according to the coordinates of the target bounding box, and the video stream is output in real time.

[0062] Embodiment 2

[0063] The embodiment of the present invention also provides a television infrared composite image target tracking system based on FPGA, such as Figure 3 As shown, it includes: a TV camera, an infrared camera, a hardware tracking acceleration module and a display module;

[0064] The television camera is used to collect visible light video image data in real time;

[0065] The infrared camera is used to collect infrared video image data in real time;

[0066] The video acquisition synchronization module is used to acquire the data transmitted by the television and the infrared camera, and synchronize the frame rate and size;

[0067] The hardware tracking acceleration module runs a target tracking algorithm to obtain target coordinate information;

[0068] The display module is used to display the target tracking result.

[0069] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A TV infrared composite image target tracking method based on FPGA, characterized in that: include: Real-time acquisition of TV and infrared dual-channel video stream images, synchronizing frame rate and size across clock domains; HLS high-level synthesis realizes the TV infrared multi-mode feature enhancement converter tracking model, completes the feature extraction, fusion and prediction of multi-mode images, and obtains the target tracking result; Overlay the tracking frame and output the video stream in real time.

2. The FPGA-based TV infrared composite image target tracking method according to claim 1, characterized in that: The HLS high-level synthesis implements the TV infrared multi-mode feature enhancement converter tracking model, completes the feature extraction, fusion and prediction of the multi-mode image, and obtains the target tracking result, specifically including: The TV infrared multi-mode feature enhancement converter tracking model is used to extract TV infrared features and fully explore the complementary information of TV images and infrared images; infrared images are less affected by lighting conditions and can capture target information in night environments and occlusions, while TV images are difficult to detect targets at night; TV images can provide information such as texture and color to help distinguish similar targets, while it is difficult to distinguish similar targets in infrared images.

3. The FPGA-based TV infrared composite image target tracking method according to claim 2, characterized in that: The TV infrared multi-mode feature enhancement converter tracking model structure is divided into three parts: feature extraction network, multi-modal feature fusion network and prediction network; The input of the feature extraction network is the TV template image T tv , infrared template image T ir , TV search area image S tv and infrared search area image S ir The template image is the H image cropped with the target position of the first frame of the video clip as the center. t ×V t The image of the i-th frame search area is cropped at the i-th frame image with the target position of the i-1 frame as the center. s ×W s The image size is 20000; the fine-tuned ResNet50 is used as the feature extraction backbone network; The multimodal feature fusion network first performs channel adjustment and reconstruction on the output results of the feature extraction network to obtain the corresponding visible light template vector V VT , infrared template vector V IT , visible light search area vector V VS and the infrared search area vector V IS ; A feature fusion network based on the Transformer converter structure is used to fuse the two branch features; The correlation calculation operation is a local linear matching process that does not consider global information and is prone to fall into local optimality. Compared with the correlation calculation, the feature fusion network of the Transformer converter structure can establish global feature associations, adaptively focus on important information, and extract sufficient semantic information to improve the calculation accuracy of subsequent regression branches and classification branches, and output more accurate tracking results. The multimodal feature fusion network structure includes N feature fusion layers and a cross feature enhancement module. The feature fusion layer consists of a self-feature enhancement module and a cross-feature enhancement module; the self-feature enhancement module X0 is expressed as: X0=X+MultiHead(X+P,X+P,X) In the formula, X is the input vector and P is the corresponding spatial position code; Through residual connection multi-head attention, the information of different positions in the feature map is integrated, and different spatial positions have different responses, making the network more effective in distinguishing foreground and background; The cross-feature enhancement module in the feature fusion layer takes the vectors of the template branch and the search area branch as input, where the main branch vector is used as the query vector and the other branch vector is used as the key vector and value vector; after calculation by N feature fusion layers, the vector of the search area branch is used as the query vector, and the template branch vector is used as the key vector and value vector to input into the final cross-feature enhancement module to obtain the final fusion vector d is the dimension of the fusion vector; The prediction network consists of a regression branch and a classification branch, each branch is a multilayer perceptron with three linear transformation layers and one nonlinear activation layer; the fusion vector is mapped into a regression vector after being calculated by the regression branch and the classification branch. and classification vector The value of the first dimension of the regression vector corresponds to the coordinate information of the target bounding box, and the value of the first dimension of the classification vector is the foreground and background scores of the area corresponding to the vector.

4. The FPGA-based TV infrared composite image target tracking method according to claim 2, characterized in that: The TV infrared multi-mode feature enhancement converter tracking model is implemented by HLS high-level synthesis method and integrated into FPGA in the form of IP; specifically, the TV infrared multi-mode feature enhancement converter tracking model IP consists of feature extraction module, multi-modal feature fusion module and prediction module; the AXI bus interface is used between modules for data transmission, so as to achieve data pipeline operation; Among them, HLS is developed in C++, and high-level synthesis uses function parallelization optimization to reduce processing cycles; pipelines are used to implement internal parallel optimization of for loops to reduce processing cycles; dataflow optimization reduces the occupancy of variable FIFO to significantly reduce resources.

5. A television infrared composite image target tracking system applied to any method of claims 1-4, characterized in that: include: TV camera, infrared camera, video acquisition synchronization module, hardware tracking acceleration module and display module; The television camera is used to collect visible light video image data in real time; The infrared camera is used to collect infrared video image data in real time; The video acquisition synchronization module is used to acquire the data transmitted by the TV camera and the infrared camera, and synchronize the frame rate and size; The hardware tracking acceleration module runs a target tracking algorithm to obtain target coordinate information; The display module is used to display the target tracking result.

Images