Target tracking method and device

Abstract

The present application provides a target tracking method and device. The method includes: obtaining three-color channel imaging data of a visible light image and imaging data of an infrared image; performing data dimension increase fitting processing on the obtained visible light imaging data and infrared imaging data through a data dimension increase fitting model; determining the target to be tracked based on the data obtained by the dimension increase fitting and through a target detection model; performing data dimension reduction fitting processing on the obtained visible light imaging data and infrared imaging data through a data dimension reduction fitting model; tracking the determined target to be tracked based on the data obtained by the dimension reduction fitting and through a target tracking model. In this way, the respective characteristics of the target in the visible light band and the infrared band are retained, the current tracking accuracy is improved, and the data calculation efficiency in the tracking algorithm can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a target tracking method and device. Background Art

[0002] Object tracking has long been a research hotspot in computer vision. Single-target tracking allows for the continuous identification and location of objects of interest within a video sequence based on their characteristics. It's well known that objects exhibit different imaging characteristics in different spectral bands. For example, in the visible light band, an object image can be captured in color; correspondingly, in the infrared band, the infrared radiation generated by the object itself can be converted into a black and white image, known as thermal imaging. Relying solely on a single optical image source, especially low-quality optical images acquired under conditions such as poor illumination, inclement weather, or fog, will result in limited image features. Infrared images, however, can be acquired in complete darkness and are less dependent on light sources, compensating for the disadvantages of optical images. Therefore, it's possible to leverage the combined image features of both visible and infrared dual-band imaging for object tracking.

[0003] In the process of implementing the prior art, the inventors found that:

[0004] Although using the image features of visible light and infrared dual-band imaging for target tracking can alleviate the shortcomings of a single optical imaging source, it requires extracting multi-scale image features for target recognition and tracking, which undoubtedly increases the amount of data required for calculation, especially in the process of real-time target tracking, which will directly affect the efficiency of target tracking.

[0005] Therefore, it is necessary to provide a target tracking method and device to solve the technical problem of large amount of computation required for target tracking using visible light and infrared dual-band imaging. Summary of the Invention

[0006] The embodiments of the present application provide a target tracking method and device to solve the technical problem of high computational complexity in target tracking using visible light and infrared dual-band imaging.

[0007] Specifically, a target tracking method includes the following steps:

[0008] Acquire three-color channel imaging data of visible light images and imaging data of infrared images;

[0009] Performing data dimensionality increase processing on the acquired visible light imaging data and infrared imaging data through a data dimensionality increase fitting model to obtain visible light-infrared four-dimensional imaging data containing visible light imaging data information and infrared imaging data information;

[0010] Determining a target to be tracked based on the obtained four-dimensional data and through a target detection model;

[0011] Performing data dimensionality reduction processing on the acquired visible light imaging data and infrared imaging data by using a data dimensionality reduction fitting model to obtain visible light-infrared imaging data containing visible light imaging data information and infrared imaging data information;

[0012] The determined target to be tracked is tracked based on the visible light-infrared imaging data obtained by the dimensionality reduction fitting and through a target tracking model.

[0013] Furthermore, the data dimension increasing fitting model is used to perform data dimension increasing processing on the acquired visible light imaging data and infrared imaging data to obtain visible light-infrared four-dimensional imaging data containing visible light imaging data information and infrared imaging data information, specifically including:

[0014] The infrared image data is scaled by a linear scaling model to obtain infrared image data with a resolution consistent with that of the visible light image.

[0015] The acquired visible light imaging data and the infrared image data obtained by the scaling process are subjected to data dimension increasing processing by means of a data dimension increasing fitting model to obtain visible light-infrared four-dimensional imaging data containing visible light imaging data information and infrared imaging data information.

[0016] Furthermore, the data dimension increase fitting model is used to perform data dimension increase processing on the acquired visible light imaging data and the infrared image data obtained by the scaling process to obtain visible light-infrared four-dimensional imaging data containing visible light imaging data information and infrared imaging data information, specifically including:

[0017] converting the infrared image data obtained by the scaling process into infrared image data registered with the visible light image through an image registration model;

[0018] The acquired visible light imaging data and the registered infrared image data are subjected to data dimension increasing processing by using a data dimension increasing fitting model to obtain visible light-infrared four-dimensional imaging data containing visible light imaging data information and infrared imaging data information.

[0019] Furthermore, tracking the determined target to be tracked based on the visible light-infrared imaging data and using a target tracking model specifically includes:

[0020] Acquire a first position parameter and a first size parameter of a target to be tracked in a first fitting image corresponding to the visible light-infrared four-dimensional imaging data;

[0021] screening tracking points of the determined target to be tracked according to the first position parameter and the first size parameter and using a target tracking model to obtain a target tracking point set consisting of a plurality of tracking points for tracking the determined target;

[0022] Calculating a second position parameter of the target to be tracked in the next frame of image based on the target tracking point set and using a target tracking model;

[0023] determining, based on the second position parameter and using a target tracking model, whether the target to be tracked exceeds a second fitting image range corresponding to the visible light-infrared imaging data obtained by the dimensionality reduction fitting in the next frame of image;

[0024] When it is determined that the target to be tracked is beyond the range of the second fitting image, the target to be tracked is re-determined based on the visible light-infrared four-dimensional imaging data and the target detection model.

[0025] Furthermore, the method of tracking the determined target to be tracked by using a target tracking model based on the visible light-infrared imaging data obtained by the dimensionality reduction fitting further includes:

[0026] When it is determined that the target to be tracked does not exceed the range of the second fitting image, calculating a second size parameter of the target to be tracked in the second fitting image based on the target tracking point set and using the target tracking model;

[0027] determining, based on the second size parameter and using a target tracking model, whether the first size parameter is consistent with the second size parameter;

[0028] When the first size parameter is inconsistent with the second size parameter, the target to be tracked is re-determined based on the visible light-infrared four-dimensional imaging data and the target detection model.

[0029] Furthermore, the data dimension reduction fitting model is used to perform data dimension reduction processing on the acquired visible light imaging data and infrared imaging data to obtain visible light-infrared imaging data containing visible light imaging data information and infrared imaging data information, specifically including:

[0030] The infrared image data is scaled by a linear scaling model to obtain infrared image data with a resolution consistent with that of the visible light image.

[0031] The acquired visible light imaging data and the infrared image data obtained by the scaling process are subjected to data dimensionality reduction processing by using a data dimensionality reduction fitting model to obtain visible light-infrared imaging data containing visible light imaging data information and infrared imaging data information.

[0032] Furthermore, the data dimension reduction fitting model is used to perform data dimension reduction processing on the acquired visible light imaging data and the infrared image data obtained by the scaling process to obtain visible light-infrared imaging data containing visible light imaging data information and infrared imaging data information, specifically including:

[0033] Converting the infrared image data obtained by the scaling process into infrared image data registered with the visible light image through an image registration model;

[0034] The acquired visible light imaging data and the registered infrared image data are subjected to data dimensionality reduction processing by using a data dimensionality reduction fitting model to obtain visible light-infrared imaging data containing visible light imaging data information and infrared imaging data information.

[0035] Furthermore, based on the obtained four-dimensional data and through the target detection model, the target to be tracked is determined, specifically including:

[0036] Based on the obtained four-dimensional data and the YOLO_V5s target detection model, the target to be tracked is determined.

[0037] Furthermore, tracking the determined target to be tracked is performed based on the visible light-infrared imaging data obtained by the dimensionality reduction fitting and using a target tracking model, specifically including:

[0038] According to the visible light-infrared imaging data obtained by the dimensionality reduction fitting, the determined target to be tracked is tracked through the Median-Flow target tracking model.

[0039] An embodiment of the present application also provides a target tracking device.

[0040] Specifically, a target tracking device includes:

[0041] An acquisition module, used to acquire three-color channel imaging data of a visible light image and imaging data of an infrared image;

[0042] a computing module for performing data dimensionality increase processing on the acquired visible light imaging data and infrared imaging data by using a data dimensionality increase fitting model to obtain visible light-infrared four-dimensional imaging data containing visible light imaging data information and infrared imaging data information; and for performing data dimensionality reduction processing on the acquired visible light imaging data and infrared imaging data by using a data dimensionality reduction fitting model to obtain visible light-infrared imaging data containing visible light imaging data information and infrared imaging data information;

[0043] A detection module, configured to determine a target to be tracked based on the obtained four-dimensional data and through a target detection model;

[0044] The tracking module is used to track the determined target to be tracked based on the visible light-infrared imaging data obtained by the dimensionality reduction fitting and through a target tracking model.

[0045] The technical solutions provided in the embodiments of the present application have at least the following beneficial effects:

[0046] By using the four-dimensional image information of the target imaged in the visible light and infrared dual bands for target recognition, and tracking the target in the compressed data of the visible light and infrared dual band images, the respective characteristics of the target in the dual band images are retained, and the amount of data calculation in the tracking algorithm is reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0048] Figure 1 A flowchart of a target tracking method provided in an embodiment of the present application.

[0049] Figure 2 A schematic diagram of a single target tracking process provided in an embodiment of the present application.

[0050] Figure 3 A schematic diagram of the structure of a target tracking device provided in an embodiment of the present application.

[0051] 100 Target Tracking Device

[0052] 11 Get Module

[0053] 12 Computing Module

[0054] 13 Detection Module

[0055] 14 Tracking Module DETAILED DESCRIPTION

[0056] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0057] Target tracking can be categorized into single-target tracking and multi-target tracking. Single-target tracking refers to tracking a single target, while multi-target tracking is the opposite. The target tracking method provided in this application is primarily applicable to tracking a single target. Therefore, the embodiments of this application also primarily focus on single-target tracking.

[0058] Single-target tracking continuously identifies and locates a target of interest within a video sequence based on its characteristics. Single-target tracking requires first identifying the target in the image, and then predicting its size and position in subsequent frames. To identify the target in the image, a detector must first acquire images of the relevant scene. For example, visible light detectors can be used to identify and track targets using visible light images. However, visible light detectors rely on sunlight and starlight for passive imaging. In particular, image quality is poor in conditions such as poor illumination, inclement weather, and fog, and image features are limited. This results in low target identification accuracy, hindering target tracking. Infrared detectors, on the other hand, detect and identify targets by receiving infrared light from the target itself, leveraging the temperature difference between the target and the background, as well as the differences in radiant energy from different parts of the target. Therefore, infrared detectors can observe targets in complete darkness without relying on any light source. This demonstrates the significant complementarity between visible light and infrared images. Using both visible light and infrared images for target identification and tracking yields a more feature-rich target, facilitating subsequent tracking.

[0059] It should be noted that identifying and tracking targets based on visible light images and infrared images often requires huge computations, which directly affects the efficiency of target tracking. In order to reduce the amount of computation in the target tracking process, the present application provides a target tracking method. For details, please refer to Figure 1 and Figure 2 The target tracking method provided in the embodiment of the present application includes the following steps:

[0060] S100: Acquire three-color channel imaging data of a visible light image and imaging data of an infrared image.

[0061] The three-color channel imaging data of the visible light image here can be understood as the three-color channel imaging data of the photographed object in the visible light band, which can be obtained through a visible light detector. Among them, the resolution of the visible light detector can be regarded as the first resolution, denoted as m0×n0; "three-color channel imaging data" can be understood as the brightness values corresponding to the red (R), green (G), and blue (B) channels of the visible light image in the RGB color mode. In the field of computer vision, this "three-color channel imaging data" can also be called "three-dimensional imaging data." That is, each color channel corresponds to a certain color dimension.

[0062] Here, infrared image data can be understood as imaging data of the object in the infrared band, which can be obtained by an infrared detector. The resolution of the infrared detector can be considered as the second resolution, denoted as m1×n1. Compared to the three-color channel (three-dimensional) imaging data of the visible light image, the infrared image data here can be understood as the single-color channel (one-dimensional) imaging data of the infrared image.

[0063] S200: performing data dimension increase processing on the acquired visible light imaging data and infrared imaging data through a data dimension increase fitting model to obtain visible light-infrared four-dimensional imaging data containing visible light imaging data information and infrared imaging data information.

[0064] It's understandable that infrared imaging data carries information about the infrared radiation energy of each region of the object being photographed, while visible light imaging data carries imaging information about the object's surface under the visible light band. However, infrared imaging data contains relatively little target feature information, making it difficult to identify and track targets. Visible light imaging data is also significantly affected by environmental factors, which can easily lead to target feature loss. Target tracking based on either visible light or infrared imaging data alone is detrimental to effective target identification. Therefore, the infrared and visible light imaging data are concatenated, performing a dimensionality-increasing fitting process. This allows the infrared imaging data to compensate for information lost in the visible light imaging data due to environmental factors such as poor illumination, inclement weather, and fog obstruction. The resulting four-dimensional data carries more comprehensive feature information. Specifically, the visible-infrared four-dimensional imaging data contains all target feature information from both the visible light and infrared bands, resulting in the most comprehensive target detection results. This effectively overcomes the drawbacks of target loss in the visible light band due to rain and fog, or the limited target feature information in the infrared band alone.

[0065] Furthermore, in a preferred embodiment provided herein, data dimension increase processing is performed on the acquired visible light imaging data and infrared imaging data through a data dimension increase fitting model to obtain visible light-infrared four-dimensional imaging data containing visible light imaging data information and infrared imaging data information, specifically including: performing scaling processing on the imaging data of the infrared image through a linear scaling model to obtain infrared image data having a resolution consistent with the resolution of the visible light image; performing data dimension increase processing on the acquired visible light imaging data and the infrared image data obtained through scaling processing through a data dimension increase fitting model to obtain visible light-infrared four-dimensional imaging data containing visible light imaging data information and infrared imaging data information.

[0066] It is understandable that visible light imaging data can be acquired using a visible light detector at a first resolution, while infrared imaging data can be acquired using a visible light detector at a second resolution. However, the resolution of an infrared detector is often lower than that of a visible light detector. That is, the second resolution is not consistent with the first resolution. Therefore, after splicing the visible light band data with the infrared band data, there is a possibility that the data length will change, hindering subsequent target detection and identification. In other words, there is a possibility that target feature information in the visible light band and the infrared band will be superimposed and confused in the four-dimensional data obtained by dimensionality increase fitting, reducing the credibility of the resulting four-dimensional data. Therefore, before performing data dimensionality increase fitting, the infrared band imaging data at the second resolution can be converted to infrared band data at the first resolution.

[0067] Specifically, the infrared band imaging data at the second resolution can be converted to the infrared band data at the first resolution by linear scaling. Assuming [x, y] is the coordinate of a certain point in the infrared band data before transformation, and [X, Y] is the coordinate of the same point after transformation, the linear scaling process of the infrared band imaging data at the second resolution can be expressed as:

[0068]

[0069] Where m0 and n0 are the resolutions of the visible light detector (first resolution), m1 and n1 are the resolutions of the infrared detector (second resolution), and the image pixel f(X, Y) corresponding to the [X, Y] coordinate can be determined using bilinear interpolation.

[0070] The conversion of infrared imaging data at different resolutions can bring the infrared and visible light imaging data into the same resolution dimension. In this case, dimensionality-increased fitting of the infrared and visible light imaging data, i.e., performing dimensionality-increased fitting on the infrared and visible light imaging data at the first resolution, increases the accuracy of target feature information in the visible-infrared four-dimensional imaging data.

[0071] Furthermore, in a preferred embodiment provided in the present application, data dimension increase processing is performed on the acquired visible light imaging data and the infrared image data obtained by scaling processing through a data dimension increase fitting model to obtain visible light-infrared four-dimensional imaging data containing visible light imaging data information and infrared imaging data information, specifically including: converting the infrared image data obtained by scaling processing into infrared image data registered with the visible light image through an image registration model; and performing data dimension increase processing on the acquired visible light imaging data and the infrared image data obtained by scaling processing through a data dimension increase fitting model to obtain visible light-infrared four-dimensional imaging data containing visible light imaging data information and infrared imaging data information.

[0072] Image registration here can be understood as registration based on the imaging coordinates of different detectors. Specifically, a mapping is established between visible light imaging and infrared imaging, allowing the spatial alignment of images captured by different sensors. This results in higher accuracy in the feature information in the visible-infrared 4D imaging data obtained through dimensionality-increment fitting.

[0073] The infrared image data obtained by the scaling process is converted into infrared image data that is registered with the visible light image through the image registration model. First, the image registration parameters need to be determined. In a specific embodiment provided in the present application, a sufficient number of same-name points (imaging points of the same target) can be found in the visible light image corresponding to the visible light imaging data and the infrared image corresponding to the infrared imaging data, and then a polynomial model is used for fitting and registration. Assuming that the same-name point position corresponding to any point [X, Y] in the visible light image is [X0, Y0] in the infrared wave band data, the following polynomial model is established:

[0074]

[0075] Where a0, a1, a2, a3, a4, a5 and b0, b1, b2, b3, b4, b5 are polynomial parameters. In practical applications, the values of parameters such as a0-a5 and b0-b5 can be determined by calibrating the spatial coordinates of visible light detectors and infrared detectors and performing data fitting.

[0076] Based on the determined image registration parameters, the infrared image at the first resolution can be registered with the visible light image at the first resolution. Specifically, the infrared image data obtained through the scaling process is converted into infrared image data that is registered with the visible light image. In this case, the input data for the dimensionality-increasing fitting model is the visible light imaging data at the first resolution and the infrared image data obtained through image registration that can be registered with the visible light image. This ensures that every point in the infrared and visible light images remains consistent, improving the accuracy of target feature information in the visible light-infrared four-dimensional imaging data obtained through dimensionality-increasing fitting, thereby enhancing target recognition accuracy.

[0077] S300: Determine the target to be tracked based on the obtained four-dimensional data and through the target detection model.

[0078] It can be understood that the four-dimensional data here contains both infrared and visible light imaging information, which greatly enriches the characteristic information of the target area, making the target detection more accurate. The target detection model here is mainly used to determine the target to be tracked in the fitted four-dimensional image.

[0079] Furthermore, in a preferred embodiment provided in the present application, the target to be tracked is determined based on the obtained four-dimensional data and through the target detection model, specifically including: determining the target to be tracked based on the obtained four-dimensional data and through the YOLO_V5s target detection model.

[0080] The YOLO_V5s target detection model here is a detection model obtained by neural network training. In a specific embodiment provided in this application, the training optimization of the YOLO_V5s target detection model can be performed by a negative feedback training method. Compared with other versions of the detection model, the data processing speed and accuracy of the YOLO_V5 detection model have been greatly improved. Among them, the YOLO_V5s version of the detection model has the smallest depth and the smallest width of the feature map. Therefore, in order to achieve both detection speed and accuracy, the YOLO_V5s model is selected here for target detection.

[0081] In a specific embodiment provided in this application, the target to be tracked is determined by the YOLO_V5s target detection model. It is necessary to first determine several targets of interest (suspected targets to be tracked) in the fitting image corresponding to the visible light-infrared four-dimensional imaging data, and then determine the target to be tracked from the determined set of targets of interest. Here, several targets of interest (suspected targets to be tracked) in the fitting image corresponding to the four-dimensional imaging data are represented by a set, which is recorded as:

[0082] ROI obj ={obj0,obj1,obj2...obj i};

[0083] Where, obj i Represents the i-th target of interest (suspected target to be tracked). It is understandable that each target detected by the YOLO_V5s target detection model can be marked by a target box. Therefore, for each target of interest (suspected target to be tracked) in the fitted image corresponding to the four-dimensional imaging data here, there is a corresponding target box, which contains at least five basic information (x, y, w, h, p); wherein x represents the x-coordinate of the upper left corner of the target of interest, y represents the y-coordinate of the upper left corner of the target of interest, w represents the image width of the target of interest, h represents the image height of the target of interest, and p represents the detection accuracy of the target of interest. It should be noted that x and y here are respectively used to represent the position information of the target box corresponding to the target of interest, and they can be uniformly corresponded to the same position in the same target box. It is understandable that the upper left corner of the target of interest here is only for ease of understanding, and its specific reference position in actual application obviously does not constitute a limitation on the scope of protection of this application.

[0084] To determine the single target to be tracked in the set of suspected targets, in addition to referring to the detection accuracy of each suspected target, it is also necessary to fully consider the position of each suspected target in the fitting image corresponding to the visible light-infrared four-dimensional imaging data.

[0085] In a specific embodiment provided in the present application, by presetting the detection accuracy threshold and taking the center point of the four-dimensional data corresponding fitting image as the reference point, the target set ROI is selected. obj Specifically, from the target data set ROI of interest, obj The suspected target whose detection accuracy is greater than the set threshold and is closest to the set reference point is selected as the target to be tracked, which can be expressed as:

[0086]

[0087] In the formula, min() represents the minimum value, obj i .p represents the detection accuracy of the i-th target of interest, Thr represents the preset detection accuracy threshold, obj i .x represents the x coordinate of the upper left corner of the i-th object of interest, obj i .y represents the y coordinate of the upper left corner of the i-th object of interest, obj i .w represents the image width of the i-th object of interest, obj i .h represents the image height of the i-th target of interest, C x Represents the x-coordinate of the center point of the fitted image corresponding to the four-dimensional data, C y It is understood that the specific value of the target detection accuracy threshold preset here does not limit the scope of protection of this application.

[0088] It's understandable that the visible-infrared 4D data obtained through dimensionality-increment fitting contains all target feature information from both the visible and infrared bands. Therefore, using this 4D visible-infrared data as input to the target detection model yields a more comprehensive representation of the target being tracked, effectively overcoming the drawbacks of target loss in rainy and foggy environments with a single visible light band, or the limited target feature information in a single infrared band.

[0089] S400: performing data dimensionality reduction processing on the acquired visible light imaging data and infrared imaging data by using a data dimensionality reduction fitting model to obtain visible light-infrared imaging data containing visible light imaging data information and infrared imaging data information.

[0090] The data dimensionality reduction and fitting process here can be understood as fusing the visible light and infrared imaging data. Specifically, the visible light and infrared images are compressed, which can be understood as blurring the images. The resulting visible light-infrared imaging data is then used as input for the subsequent tracking algorithm.

[0091] It is understandable that the identified target to be tracked carries both its characteristic information in the visible light band and its characteristic information in the infrared band. When tracking the identified target to be tracked, only the visible light image corresponding to the visible light detector is tracked. If the target is obscured in the visible light image, the target cannot be accurately identified in subsequent image frames, resulting in target tracking failure. Similarly, if only the infrared image corresponding to the infrared detector is tracked, due to the low resolution and poor detail resolution of the infrared image, the target cannot be accurately identified in subsequent images, which will also lead to target tracking failure. Therefore, the visible light-infrared imaging data containing visible light imaging data information and infrared imaging data information is used as the reference image for subsequent target tracking to increase the accuracy of target tracking.

[0092] Given the rich feature information of the identified target to be tracked, and to reduce the computational complexity of the subsequent target tracking model, we perform dimensionality reduction fitting on the imaging data from the visible light detector and the infrared detector, effectively reducing the amount of data required for the tracking algorithm. Furthermore, the data obtained through dimensionality reduction fitting contains feature information from both the visible light and infrared bands, ensuring accurate target tracking.

[0093] In one embodiment provided herein, the acquired three-color channel (three-dimensional) imaging data of the visible light image and the infrared image data are reduced in dimension and fitted into one-dimensional (monochromatic) visible-light-infrared imaging data, which is then used as input for the subsequent target tracking model. This effectively reduces the dimensionality of the input data for the target tracking model, thereby reducing the computational complexity and improving target tracking efficiency.

[0094] Specifically, the acquired three-color channel imaging data of the visible light image and the imaging data of the infrared image are subjected to data fusion and dimensionality reduction processing, and a weighted operation is used, which is expressed as:

[0095] F(i,j)=0.6(0.299·R(i,j)+0.587·G(i,j)+0.114·B(i,j))+0.4·I(i,j)

[0096] Where F(i,j) represents the one-dimensional (monochrome channel) data after image fusion at the corresponding (i,j) coordinate, R(i,j), G(i,j), and B(i,j) represent the three-color channel data of the three-color channel imaging data of the visible light image at the corresponding (i,j) coordinate, and I(i,j) represents the monochrome channel data of the imaging data of the infrared image at the corresponding (i,j) coordinate.

[0097] Furthermore, in a preferred embodiment provided in the present application, data dimension reduction processing is performed on the acquired visible light imaging data and infrared imaging data through a data dimension reduction fitting model to obtain visible light-infrared imaging data containing visible light imaging data information and infrared imaging data information, specifically including: scaling processing is performed on the imaging data of the infrared image through a linear scaling model to obtain infrared image data having a resolution consistent with the resolution of the visible light image; data dimension reduction processing is performed on the acquired visible light imaging data and the infrared image data obtained by the scaling processing through a data dimension reduction fitting model to obtain visible light-infrared imaging data containing visible light imaging data information and infrared imaging data information.

[0098] It is understandable that visible light imaging data can be acquired using a visible light detector at a first resolution, while infrared imaging data can be acquired using a visible light detector at a second resolution. However, the resolution of an infrared detector is often lower than that of a visible light detector. That is, the second resolution is not consistent with the first resolution. Therefore, after splicing the visible light band data with the infrared band data, there is a possibility that the data length will change, hindering subsequent target detection and identification. Specifically, there is a possibility that target feature information in the visible light band and the infrared band will be superimposed and confused during the dimensionality reduction and fitting process, reducing the credibility of the resulting data. Therefore, before performing data dimensionality increase and fitting, the infrared band imaging data at the second resolution can be converted to infrared band data at the first resolution.

[0099] Specifically, the infrared band imaging data at the second resolution is converted into infrared band data at the first resolution by linear scaling. The detailed linear scaling process has been described in the previous data dimensionality increase fitting process and will not be repeated here.

[0100] The conversion of infrared imaging data at different resolutions can bring the infrared and visible light imaging data into the same resolution dimension. In this case, dimensionality reduction fitting of the infrared and visible light imaging data, i.e., performing dimensionality reduction fitting on the infrared and visible light imaging data at a first resolution, increases the accuracy of target feature information in the visible light-infrared imaging data obtained by dimensionality reduction.

[0101] Furthermore, in a preferred embodiment provided in the present application, data dimension reduction processing is performed on the acquired visible light imaging data and the infrared image data obtained by scaling processing through a data dimension reduction fitting model to obtain visible light-infrared imaging data containing visible light imaging data information and infrared imaging data information, specifically including: converting the infrared image data obtained by scaling processing into infrared image data that is aligned with the visible light image through an image registration model; performing data dimension reduction processing on the acquired visible light imaging data and the infrared image data obtained by scaling processing through a data dimension reduction fitting model to obtain visible light-infrared imaging data that contains visible light imaging data information and infrared imaging data information.

[0102] Image registration here can be understood as registration based on the imaging coordinates of different detectors. Specifically, a mapping is established between visible light and infrared imaging, allowing the spatial alignment of images captured by different sensors. This results in more accurate regional feature information within the visible-infrared imaging data obtained through dimensionality reduction fitting.

[0103] The image registration model converts the scaled infrared image data into infrared image data aligned with the visible light image. This detailed process has been described in the data dimensionality increase and fitting process above and will not be repeated here. After image registration, the input data for the dimensionality reduction fitting model is the visible light imaging data at the first resolution and the infrared image data aligned with the visible light image obtained through image registration. This ensures that every point in the infrared and visible light images remains consistent, improving the accuracy of the feature information of each region in the visible light-infrared imaging data obtained through dimensionality reduction fitting, thereby enhancing target tracking accuracy.

[0104] S500: Tracking the determined target to be tracked based on the visible light-infrared imaging data obtained by dimensionality reduction fitting and using a target tracking model.

[0105] It is understandable that using the visible-infrared imaging data obtained through dimensionality reduction fitting as the input to the target tracking model not only reduces the amount of computational data and improves target tracking speed, but also appropriately preserves the image features of both the infrared and visible light bands. It is worth noting that the imaging angle can directly affect the accuracy of target tracking. For example, when the imaging angle is offset, the appearance of the target in the image will change or be occluded, resulting in target drift or loss. During the target tracking process, target occlusion, deformation, resizing, and out-of-field conditions often occur, which directly affect the accuracy of target tracking. Therefore, the training of the target tracking model needs to comprehensively consider the target occlusion, deformation, resizing, and out-of-field conditions that may occur during the target tracking process in order to accurately track the target.

[0106] Furthermore, in a preferred embodiment provided in the present application, tracking of a determined target to be tracked is carried out based on the visible light-infrared imaging data obtained by dimensionality reduction fitting and through a target tracking model, specifically including: obtaining a first position parameter and a first size parameter of the determined target to be tracked in a first fitting image corresponding to the visible light-infrared four-dimensional imaging data; screening tracking points of the determined target to be tracked based on the first position parameter and the first size parameter and through the target tracking model to obtain a target tracking point set consisting of a number of tracking points for tracking the determined target; calculating a second position parameter of the target to be tracked in the next frame image based on the target tracking point set and through the target tracking model; judging whether the determined target to be tracked in the next frame image exceeds the range of the second fitting image corresponding to the visible light-infrared imaging data obtained by dimensionality reduction fitting based on the second position parameter and through the target tracking model; when the determined target to be tracked exceeds the range of the second fitting image, re-determining the target to be tracked based on the visible light-infrared four-dimensional imaging data and through the target detection model.

[0107] It is understood that in the fitting image corresponding to the visible light-infrared 4D imaging data (the first fitting image), there will be a target frame corresponding to the determined target, and it will contain at least five basic information items: (x, y, w, h, p). Among them, (x, y) is used to represent the position of the target frame, and (w, h) is used to represent the size of the target frame. The first position parameter and the first size parameter here can be understood as the position parameters (x, y) and size parameters (w, h) of the target frame corresponding to the fitting image corresponding to the visible light-infrared 4D imaging data to be tracked, which can be directly obtained after the target to be tracked is determined.

[0108] The tracking points here can be understood as positioning points that represent the same position of the target to be tracked in each frame. Tracking point screening can be achieved by uniformly generating points within the determined target size range. These points are then tracked forward to frame t+1 using the Lucas-Kanade tracker and then reversely tracked to frame t. The tracking point error between the two frames is calculated, and the half of the points with the smallest error are selected as the optimal tracking points. Based on the changes in the coordinates and distances of these optimal tracking points, the position and size of the target to be tracked in the next frame can be calculated. The position of the target to be tracked in the next frame is prioritized to determine whether the image exceeds the range of the reference image. If it does, the tracker fails. In this case, there is no need to calculate the position of the target to be tracked in the next frame, thereby speeding up the processing of the target tracking model. The position parameter corresponding to the position of the target to be tracked in the next frame is referred to as the second position parameter.

[0109] Specifically, the fitting image (second fitting image) corresponding to the visible light-infrared data obtained by dimensionality reduction fitting is used as the reference image. By comparing the second position parameter with the reference image, when it exceeds the range of the reference image, it means that the target to be tracked is lost. At this time, it is necessary to redetermine the target to be tracked. The originally determined target to be tracked is referred to as the first target, and the target obtained by re-detection is referred to as the second target. The second target can be re-determined by intercepting an image of a certain range around the first target and performing target detection using a target detection model. After the second target is determined, the position parameters and size parameters corresponding to the second target will become the relevant parameters of the subsequent target to be tracked. That is, the update of the target to be tracked is completed, and continuous tracking is carried out based on the data obtained by dimensionality reduction fitting of the next frame of image.

[0110] By comparing the second position parameter with the reference image, if the second position parameter does not exceed the reference image range, the corresponding size parameter of the target to be tracked in the next frame of image should be referred to to prevent target tracking failure caused by target size changes due to factors such as changes in shooting angle.

[0111] Furthermore, in a preferred embodiment provided in the present application, tracking of the determined target to be tracked is carried out based on the visible light-infrared imaging data obtained by dimensionality reduction fitting and through the target tracking model, and also includes: when it is determined that the target to be tracked does not exceed the range of the second fitting image, calculating the second size parameter of the target to be tracked in the second fitting image based on the target tracking point set and through the target tracking model; judging whether the first size parameter is consistent with the second size parameter based on the second size parameter and through the target tracking model; when the first size parameter is inconsistent with the second size parameter, re-determining the target to be tracked based on the visible light-infrared four-dimensional imaging data and through the target detection model.

[0112] The second size parameter here can be understood as the size parameter corresponding to the target to be tracked in the next frame image calculated based on the target tracking point. Although the target to be tracked does not exceed the range of the reference image (second fitting image), there is a possibility that the second size parameter is inconsistent with the first size parameter, which will directly affect the accuracy of target tracking. The inconsistency between the second size parameter and the first size parameter here can be understood as the different target frame sizes corresponding to the target to be tracked. When the second size parameter is inconsistent with the first size parameter, it means that the size of the target to be tracked has changed. At this time, it is necessary to redetermine the target to be tracked. Similarly, the originally determined target to be tracked is regarded as the first target, and the re-detected target is regarded as the second target. The redetermination of the second target can be achieved by intercepting an image within a certain range around the first target and performing target detection using the target detection model. After the second target is determined, the corresponding position parameters and size parameters of the second target are generated.

[0113] It's worth noting that this doesn't simply replace the second target with the new one to be tracked. Instead, the target tracking model first determines whether the size parameters corresponding to the second target and the second size parameters corresponding to the original first target meet the preset target tracking parameter replacement conditions. Specifically, the target tracking model determines whether the size corresponding to the second target and the second size corresponding to the original first target intersect in space and meet specific criteria.

[0114] Specifically, assume that the second size of the first target (the original target to be tracked) is represented by a rectangular box [x1, y1, x2, y2], (x1, y1) represents the upper left corner of the first target, and (x2, y2) represents the lower right corner of the first target; the size of the second target (the re-detected target) is represented by a rectangular box [x3, y3, x4, y4], (x3, y3) represents the upper left corner of the second target, and (x4, y4) represents the lower right corner of the second target. The target tracking model judgment criteria are as follows:

[0115]

[0116] Where area_inter represents the area of the overlapped portion of the first and second targets at the second size, area_T represents the area of the second size, min() represents the minimum value, and max() represents the maximum value.

[0117] After being processed by the target tracking model, when the size corresponding to the second target and the second size corresponding to the first target intersect in space and meet the judgment criteria, the size of the first target is replaced with the size parameters corresponding to the second target, and the position parameters of the first target remain unchanged and remain the second position parameters. That is, the size parameters of the target to be tracked that are re-determined are the size parameters corresponding to the re-detected target (second target); the position parameters are the position parameters (second position parameters) of the original tracked target calculated based on the tracking points. Afterwards, the size parameters and position parameters input by the target tracking model are respectively updated to the size parameters and second position parameters corresponding to the re-detected target, and tracking is carried out based on the data obtained by dimensionality reduction fitting of the next frame of image. In this way, target tracking failures caused by target deformation and size transformation are effectively avoided.

[0118] Furthermore, in a preferred embodiment provided in the present application, the determined target to be tracked is tracked based on the visible light-infrared imaging data obtained by dimensionality reduction fitting and through the target tracking model, specifically including: tracking the determined target to be tracked based on the visible light-infrared imaging data obtained by dimensionality reduction fitting and through the Median-Flow target tracking model.

[0119] The Median-Flow target tracking model is based on the Median-Flow tracking algorithm. This algorithm can track the size changes of the target frame, thereby reducing the impact of target frame size changes on target accuracy.

[0120] Please refer to Figure 3 The present application further provides a target tracking device 100 for tracking a target according to the target tracking method. Specifically, the target tracking device 100 includes:

[0121] An acquisition module 11 is used to acquire three-color channel imaging data of a visible light image and imaging data of an infrared image;

[0122] The computing module 12 is configured to perform data dimensionality increase processing on the acquired visible light imaging data and infrared imaging data by using a data dimensionality increase fitting model to obtain visible light-infrared four-dimensional imaging data containing visible light imaging data information and infrared imaging data information; and is further configured to perform data dimensionality reduction processing on the acquired visible light imaging data and infrared imaging data by using a data dimensionality reduction fitting model to obtain visible light-infrared imaging data containing visible light imaging data information and infrared imaging data information;

[0123] A detection module 13 is configured to determine a target to be tracked based on the obtained four-dimensional data and using a target detection model;

[0124] The tracking module 14 is configured to track the determined target to be tracked based on the visible light-infrared imaging data obtained by the dimensionality reduction fitting and through a target tracking model.

[0125] The acquisition module 11 is used to acquire the three-color channel imaging data of the visible light image and the imaging data of the infrared image. The three-color channel imaging data of the visible light image here can be understood as the three-color channel imaging data of the photographed object in the visible light band, which can be obtained through a visible light detector. Among them, the resolution of the visible light detector can be regarded as the first resolution, recorded as m0×n0; "three-color channel imaging data" can be understood as the brightness values corresponding to the red (R), green (G), and blue (B) channels of the visible light image in the RGB color mode. In the field of computer vision, the "three-color channel imaging data" here can also be called "three-dimensional imaging data". That is, each color channel corresponds to a certain color dimension.

[0126] Here, infrared image data can be understood as imaging data of the object in the infrared band, which can be obtained by an infrared detector. The resolution of the infrared detector can be considered as the second resolution, denoted as m1×n1. Compared to the three-color channel (three-dimensional) imaging data of the visible light image, the infrared image data here can be understood as the single-color channel (one-dimensional) imaging data of the infrared image.

[0127] The computing module 12 is configured to perform data dimensionality increase processing on the acquired visible light imaging data and infrared imaging data using a data dimensionality increase fitting model, thereby obtaining visible light-infrared four-dimensional imaging data containing both visible light imaging data and infrared imaging data information. It is understood that infrared imaging data carries information about the infrared radiation energy of each region of the photographed object, while visible light imaging data carries imaging information about the object's surface in the visible light band. However, infrared imaging data contains relatively little target feature information, making it difficult to identify and track targets. Visible light imaging data is also significantly affected by environmental factors, which can easily lead to loss of target features. Target tracking based on either visible light imaging data or infrared imaging data alone is detrimental to effective target identification. Therefore, the infrared imaging data and visible light imaging data are concatenated, i.e., data dimensionality increase fitting is performed. This allows the infrared imaging data to compensate for information missing from the visible light imaging data due to environmental factors such as poor illumination, inclement weather, and fog obstruction, resulting in four-dimensional data containing more comprehensive feature information. That is, the visible light-infrared four-dimensional imaging data contains all the target feature information of both the visible light band and the infrared band, making the subsequent target detection results the most comprehensive, effectively overcoming the defects of a single visible light band losing the target due to the rain and fog environment, or a single infrared band having little target feature information.

[0128] Furthermore, in a preferred embodiment provided herein, the computing module 12 is used to perform data dimensionality increase processing on the acquired visible light imaging data and infrared imaging data through a data dimensionality increase fitting model to obtain visible light-infrared four-dimensional imaging data containing visible light imaging data information and infrared imaging data information. Specifically, it is used to: perform data dimensionality increase processing on the imaging data of the infrared image through a linear scaling model to obtain infrared image data having a resolution consistent with the resolution of the visible light image; perform data dimensionality increase processing on the acquired visible light imaging data and the infrared image data obtained through scaling processing through a data dimensionality increase fitting model to obtain visible light-infrared four-dimensional imaging data containing visible light imaging data information and infrared imaging data information.

[0129] It is understandable that visible light imaging data can be acquired using a visible light detector at a first resolution, while infrared imaging data can be acquired using a visible light detector at a second resolution. However, the resolution of an infrared detector is often lower than that of a visible light detector. That is, the second resolution is not consistent with the first resolution. Therefore, after splicing the visible light band data with the infrared band data, there is a possibility that the data length will change, hindering subsequent target detection and identification. In other words, there is a possibility that target feature information in the visible light band and the infrared band will be superimposed and confused in the four-dimensional data obtained by dimensionality increase fitting, reducing the credibility of the resulting four-dimensional data. Therefore, before performing data dimensionality increase fitting, the infrared band imaging data at the second resolution can be converted to infrared band data at the first resolution.

[0130] Specifically, the infrared band imaging data at the second resolution can be converted to the infrared band data at the first resolution by linear scaling. Assuming [x, y] is the coordinate of a certain point in the infrared band data before transformation, and [X, Y] is the coordinate of the same point after transformation, the linear scaling process of the infrared band imaging data at the second resolution can be expressed as:

[0131]

[0132] Where m0 and n0 are the resolutions of the visible light detector (first resolution), m1 and n1 are the resolutions of the infrared detector (second resolution), and the image pixel f(X, Y) corresponding to the [X, Y] coordinate can be determined using bilinear interpolation.

[0133] The conversion of infrared imaging data at different resolutions can bring the infrared and visible light imaging data into the same resolution dimension. In this case, dimensionality-increased fitting of the infrared and visible light imaging data, i.e., performing dimensionality-increased fitting on the infrared and visible light imaging data at the first resolution, increases the accuracy of target feature information in the visible-infrared four-dimensional imaging data.

[0134] Furthermore, in a preferred embodiment provided in the present application, the computing module 12 is used to perform data dimensionality increase processing on the acquired visible light imaging data and the infrared image data obtained by scaling processing through a data dimensionality increase fitting model to obtain visible light-infrared four-dimensional imaging data containing visible light imaging data information and infrared imaging data information. Specifically, it is used to: convert the infrared image data obtained by scaling processing into infrared image data registered with the visible light image through an image registration model; perform data dimensionality increase processing on the acquired visible light imaging data and the infrared image data obtained by scaling processing through a data dimensionality increase fitting model to obtain visible light-infrared four-dimensional imaging data containing visible light imaging data information and infrared imaging data information.

[0135] Image registration here can be understood as registration based on the imaging coordinates of different detectors. Specifically, a mapping is established between visible light imaging and infrared imaging, allowing the spatial alignment of images captured by different sensors. This results in higher accuracy in the feature information in the visible-infrared 4D imaging data obtained through dimensionality-increment fitting.

[0136] The infrared image data obtained by the scaling process is converted into infrared image data that is registered with the visible light image through the image registration model. First, the image registration parameters need to be determined. In a specific embodiment provided in the present application, a sufficient number of same-name points (imaging points of the same target) can be found in the visible light image corresponding to the visible light imaging data and the infrared image corresponding to the infrared imaging data, and then a polynomial model is used for fitting and registration. Assuming that the same-name point position corresponding to any point [X, Y] in the visible light image is [X0, Y0] in the infrared wave band data, the following polynomial model is established:

[0137]

[0138] Where a0, a1, a2, a3, a4, a5 and b0, b1, b2, b3, b4, b5 are polynomial parameters. In practical applications, the values of parameters such as a0-a5 and b0-b5 can be determined by calibrating the spatial coordinates of visible light detectors and infrared detectors and performing data fitting.

[0139] Based on the determined image registration parameters, the infrared image at the first resolution can be registered with the visible light image at the first resolution. Specifically, the infrared image data obtained through the scaling process is converted into infrared image data that is registered with the visible light image. In this case, the input data for the dimensionality-increasing fitting model is the visible light imaging data at the first resolution and the infrared image data obtained through image registration that can be registered with the visible light image. This ensures that every point in the infrared and visible light images remains consistent, improving the accuracy of target feature information in the visible light-infrared four-dimensional imaging data obtained through dimensionality-increasing fitting, thereby enhancing target recognition accuracy.

[0140] Detection module 13 is configured to determine the target to be tracked based on the obtained four-dimensional data and using a target detection model. It will be appreciated that the four-dimensional data herein includes both infrared and visible light imaging information, significantly enriching the characteristic information of the target area, thereby making target detection more accurate. The target detection model herein is primarily used to determine the target to be tracked within the fitted four-dimensional image.

[0141] Furthermore, in a preferred embodiment provided in the present application, the detection module 13 is used to determine the target to be tracked based on the obtained four-dimensional data and through the target detection model, and is specifically used to determine the target to be tracked based on the obtained four-dimensional data and through the YOLO_V5s target detection model.

[0142] The YOLO_V5s target detection model here is a detection model obtained by neural network training. In a specific embodiment provided in this application, the training optimization of the YOLO_V5s target detection model can be performed by a negative feedback training method. Compared with other versions of the detection model, the data processing speed and accuracy of the YOLO_V5 detection model have been greatly improved. Among them, the YOLO_V5s version of the detection model has the smallest depth and the smallest width of the feature map. Therefore, in order to achieve both detection speed and accuracy, the YOLO_V5s model is selected here for target detection.

[0143] In a specific embodiment provided in this application, the target to be tracked is determined by the YOLO_V5s target detection model. It is necessary to first determine several targets of interest (suspected targets to be tracked) in the fitting image corresponding to the visible light-infrared four-dimensional imaging data, and then determine the target to be tracked from the determined set of targets of interest. Here, several targets of interest (suspected targets to be tracked) in the fitting image corresponding to the four-dimensional imaging data are represented by a set, which is recorded as:

[0144] ROI obj ={obj0,obj1,obj2...obji};

[0145] Wherein, obji represents the i-th target of interest (suspected target to be tracked). It is understandable that each target detected by the YOLO_V5s target detection model can be marked by a target box. Therefore, for each target of interest (suspected target to be tracked) in the fitted image corresponding to the four-dimensional imaging data, there is a corresponding target box, which contains at least five basic information items (x, y, w, h, p); wherein x represents the x-coordinate of the upper left corner of the target of interest, y represents the y-coordinate of the upper left corner of the target of interest, w represents the image width of the target of interest, h represents the image height of the target of interest, and p represents the detection accuracy of the target of interest. It should be noted that x and y here are respectively used to represent the position information of the target box corresponding to the target of interest, and they can be uniformly corresponded to the same position in the same target box. It is understandable that the upper left corner of the target of interest here is only for ease of understanding, and its specific reference position in actual application obviously does not constitute a limitation on the scope of protection of this application.

[0146] To determine the single target to be tracked in the set of suspected targets, in addition to referring to the detection accuracy of each suspected target, it is also necessary to fully consider the position of each suspected target in the fitting image corresponding to the visible light-infrared four-dimensional imaging data.

[0147] In a specific embodiment provided in the present application, by presetting the detection accuracy threshold and taking the center point of the four-dimensional data corresponding fitting image as the reference point, the target set ROI is selected. obj Specifically, from the target data set ROI of interest, obj The suspected target whose detection accuracy is greater than the set threshold and is closest to the set reference point is selected as the target to be tracked, which can be expressed as:

[0148]

[0149] In the formula, min() represents the minimum value, obj i .p represents the detection accuracy of the i-th target of interest, Thr represents the preset detection accuracy threshold, obj i .x represents the x coordinate of the upper left corner of the i-th object of interest, obj i .y represents the y coordinate of the upper left corner of the i-th object of interest, obj i .w represents the image width of the i-th object of interest, obj i .h represents the image height of the i-th target of interest, C x Represents the x-coordinate of the center point of the fitted image corresponding to the four-dimensional data, C y It is understood that the specific value of the target detection accuracy threshold preset here does not limit the scope of protection of this application.

[0150] It can be understood that the visible light-infrared four-dimensional data obtained by dimensionality increase fitting contains all target feature information in both the visible light band and the infrared dual band. Therefore, using the visible light-infrared four-dimensional data as input to the target detection model can obtain more comprehensive feature information of the target to be tracked, effectively overcoming the shortcomings of a single visible light band losing the target due to rain and fog, or a single infrared band having less target feature information. Therefore, the calculation module 12 is also used to perform data dimensionality reduction processing on the acquired visible light imaging data and infrared imaging data through a data dimensionality reduction fitting model, thereby obtaining visible light-infrared imaging data containing visible light imaging data information and infrared imaging data information.

[0151] The data dimensionality reduction and fitting process here can be understood as fusing the visible light and infrared imaging data. Specifically, the visible light and infrared images are compressed, which can be understood as blurring the images. The resulting visible light-infrared imaging data is then used as input for the subsequent tracking algorithm.

[0152] It is understandable that the identified target to be tracked carries both its characteristic information in the visible light band and its characteristic information in the infrared band. When tracking the identified target to be tracked, only the visible light image corresponding to the visible light detector is tracked. If the target is obscured in the visible light image, the target cannot be accurately identified in subsequent image frames, resulting in target tracking failure. Similarly, if only the infrared image corresponding to the infrared detector is tracked, due to the low resolution and poor detail resolution of the infrared image, the target cannot be accurately identified in subsequent images, which will also lead to target tracking failure. Therefore, the visible light-infrared imaging data containing visible light imaging data information and infrared imaging data information is used as the reference image for subsequent target tracking to increase the accuracy of target tracking.

[0153] Given the rich feature information of the identified target to be tracked, and to reduce the computational complexity of the subsequent target tracking model, we perform dimensionality reduction fitting on the imaging data from the visible light detector and the infrared detector, effectively reducing the amount of data required for the tracking algorithm. Furthermore, the data obtained through dimensionality reduction fitting contains feature information from both the visible light and infrared bands, ensuring accurate target tracking.

[0154] In one embodiment provided herein, the acquired three-color channel (three-dimensional) imaging data of the visible light image and the infrared image data are reduced in dimension and fitted into one-dimensional (monochromatic) visible-light-infrared imaging data, which is then used as input for the subsequent target tracking model. This effectively reduces the dimensionality of the input data for the target tracking model, thereby reducing the computational complexity and improving target tracking efficiency.

[0155] Specifically, the acquired three-color channel imaging data of the visible light image and the imaging data of the infrared image are subjected to data fusion and dimensionality reduction processing, and a weighted operation is used, which is expressed as:

[0156] F(i,j)=0.6(0.299·R(i,j)+0.587·G(i,j)+0.114·B(i,j))+0.4·I(i,j)

[0157] Where F(i,j) represents the one-dimensional (monochrome channel) data after image fusion at the corresponding (i,j) coordinate, R(i,j), G(i,j), and B(i,j) represent the three-color channel data of the three-color channel imaging data of the visible light image at the corresponding (i,j) coordinate, and I(i,j) represents the monochrome channel data of the imaging data of the infrared image at the corresponding (i,j) coordinate.

[0158] Furthermore, in a preferred embodiment provided in the present application, the computing module 12 is used to perform data dimensionality reduction processing on the acquired visible light imaging data and infrared imaging data through a data dimensionality reduction fitting model to obtain visible light-infrared imaging data containing visible light imaging data information and infrared imaging data information. Specifically, it is used to: perform scaling processing on the imaging data of the infrared image through a linear scaling model to obtain infrared image data having a resolution consistent with the resolution of the visible light image; perform data dimensionality reduction processing on the acquired visible light imaging data and the infrared image data obtained by the scaling processing through a data dimensionality reduction fitting model to obtain visible light-infrared imaging data containing visible light imaging data information and infrared imaging data information.

[0159] It is understandable that visible light imaging data can be acquired using a visible light detector at a first resolution, while infrared imaging data can be acquired using a visible light detector at a second resolution. However, the resolution of an infrared detector is often lower than that of a visible light detector. That is, the second resolution is not consistent with the first resolution. Therefore, after splicing the visible light band data with the infrared band data, there is a possibility that the data length will change, hindering subsequent target detection and identification. Specifically, there is a possibility that target feature information in the visible light band and the infrared band will be superimposed and confused during the dimensionality reduction and fitting process, reducing the credibility of the resulting data. Therefore, before performing data dimensionality increase and fitting, the infrared band imaging data at the second resolution can be converted to infrared band data at the first resolution.

[0160] Specifically, the infrared band imaging data at the second resolution is converted into infrared band data at the first resolution by linear scaling. The detailed linear scaling process has been described in the previous data dimensionality increase fitting process and will not be repeated here.

[0161] The conversion of infrared imaging data at different resolutions can bring the infrared and visible light imaging data into the same resolution dimension. In this case, dimensionality reduction fitting of the infrared and visible light imaging data, i.e., performing dimensionality reduction fitting on the infrared and visible light imaging data at a first resolution, increases the accuracy of target feature information in the visible light-infrared imaging data obtained by dimensionality reduction.

[0162] Furthermore, in a preferred embodiment provided in the present application, the computing module 12 is used to perform data dimensionality reduction processing on the acquired visible light imaging data and the infrared image data obtained by scaling processing through a data dimensionality reduction fitting model to obtain visible light-infrared imaging data containing visible light imaging data information and infrared imaging data information. Specifically, it is used to: convert the infrared image data obtained by scaling processing into infrared image data that is aligned with the visible light image through an image registration model; perform data dimensionality reduction processing on the acquired visible light imaging data and the infrared image data obtained by scaling processing through a data dimensionality reduction fitting model to obtain visible light-infrared imaging data that contains visible light imaging data information and infrared imaging data information.

[0163] Image registration here can be understood as registration based on the imaging coordinates of different detectors. Specifically, a mapping is established between visible light and infrared imaging, allowing the spatial alignment of images captured by different sensors. This results in more accurate regional feature information within the visible-infrared imaging data obtained through dimensionality reduction fitting.

[0164] The image registration model converts the scaled infrared image data into infrared image data aligned with the visible light image. This detailed process has been described in the data dimensionality increase and fitting process above and will not be repeated here. After image registration, the input data for the dimensionality reduction fitting model is the visible light imaging data at the first resolution and the infrared image data aligned with the visible light image obtained through image registration. This ensures that every point in the infrared and visible light images remains consistent, improving the accuracy of the feature information of each region in the visible light-infrared imaging data obtained through dimensionality reduction fitting, thereby enhancing target tracking accuracy.

[0165] Tracking module 14 is configured to track the identified target to be tracked based on the visible light-infrared imaging data obtained through dimensionality reduction fitting and using a target tracking model. It is understood that using the visible light-infrared imaging data obtained through dimensionality reduction fitting as input to the target tracking model not only reduces the amount of computational data and improves target tracking speed, but also appropriately preserves the image features of both infrared and visible light bands. It is worth noting that the imaging angle can directly affect the accuracy of target tracking. For example, when the imaging angle is offset, the appearance of the target in the image changes or is obscured, resulting in target drift or loss. During the target tracking process, target occlusion, deformation, resizing, and out-of-field conditions often occur, directly impacting target tracking accuracy. Therefore, when training the target tracking model, it is necessary to comprehensively consider target occlusion, deformation, resizing, and out-of-field conditions that may occur during the target tracking process in order to accurately track the target.

[0166] Furthermore, in a preferred embodiment provided in the present application, the tracking module 14 is used to track the determined target to be tracked based on the visible light-infrared imaging data obtained by dimensionality reduction fitting and through the target tracking model, and is specifically used to: obtain the first position parameter and the first size parameter of the determined target to be tracked in the first fitting image corresponding to the visible light-infrared four-dimensional imaging data; based on the first position parameter and the first size parameter and through the target tracking model, screen the tracking points of the determined target to be tracked to obtain a target tracking point set consisting of a number of tracking points for tracking the determined target; based on the target tracking point set and through the target tracking model, calculate the second position parameter of the target to be tracked in the next frame image; based on the second position parameter and through the target tracking model, determine whether the determined target to be tracked in the next frame image exceeds the second fitting image range corresponding to the visible light-infrared imaging data obtained by dimensionality reduction fitting; when the determined target to be tracked exceeds the second fitting image range, re-determine the target to be tracked based on the visible light-infrared four-dimensional imaging data and through the target detection model.

[0167] It is understood that in the fitting image corresponding to the visible light-infrared 4D imaging data (the first fitting image), there will be a target frame corresponding to the determined target, and it will contain at least five basic information items: (x, y, w, h, p). Among them, (x, y) is used to represent the position of the target frame, and (w, h) is used to represent the size of the target frame. The first position parameter and the first size parameter here can be understood as the position parameters (x, y) and size parameters (w, h) of the target frame corresponding to the fitting image corresponding to the visible light-infrared 4D imaging data to be tracked, which can be directly obtained after the target to be tracked is determined.

[0168] The tracking points here can be understood as positioning points that represent the same position of the target to be tracked in each frame. Tracking point screening can be achieved by uniformly generating points within the determined target size range. These points are then tracked forward to frame t+1 using the Lucas-Kanade tracker and then reversely tracked to frame t. The tracking point error between the two frames is calculated, and the half of the points with the smallest error are selected as the optimal tracking points. Based on the changes in the coordinates and distances of these optimal tracking points, the position and size of the target to be tracked in the next frame can be calculated. The position of the target to be tracked in the next frame is prioritized to determine whether the image exceeds the range of the reference image. If it does, the tracker fails. In this case, there is no need to calculate the position of the target to be tracked in the next frame, thereby speeding up the processing of the target tracking model. The position parameter corresponding to the position of the target to be tracked in the next frame is referred to as the second position parameter.

[0169] Specifically, the fitting image (second fitting image) corresponding to the visible light-infrared data obtained by dimensionality reduction fitting is used as the reference image. By comparing the second position parameter with the reference image, when it exceeds the range of the reference image, it means that the target to be tracked is lost. At this time, it is necessary to redetermine the target to be tracked. The originally determined target to be tracked is referred to as the first target, and the target obtained by re-detection is referred to as the second target. The second target can be re-determined by intercepting an image of a certain range around the first target and performing target detection using a target detection model. After the second target is determined, the position parameters and size parameters corresponding to the second target will become the relevant parameters of the subsequent target to be tracked. That is, the update of the target to be tracked is completed, and continuous tracking is carried out based on the data obtained by dimensionality reduction fitting of the next frame of image.

[0170] By comparing the second position parameter with the reference image, if the second position parameter does not exceed the reference image range, the corresponding size parameter of the target to be tracked in the next frame of image should be referred to to prevent target tracking failure caused by target size changes due to factors such as changes in shooting angle.

[0171] Furthermore, in a preferred embodiment provided in the present application, the tracking module 14 is used to track the determined target to be tracked based on the visible light-infrared imaging data obtained by dimensionality reduction fitting and through the target tracking model, and is also used to: when it is determined that the target to be tracked does not exceed the range of the second fitting image, calculate the second size parameter of the target to be tracked in the second fitting image according to the target tracking point set and through the target tracking model; based on the second size parameter and through the target tracking model, determine whether the first size parameter is consistent with the second size parameter; when the first size parameter is inconsistent with the second size parameter, re-determine the target to be tracked based on the visible light-infrared four-dimensional imaging data and through the target detection model.

[0172] The second size parameter here can be understood as the size parameter corresponding to the target to be tracked in the next frame image calculated based on the target tracking point. Although the target to be tracked does not exceed the range of the reference image (second fitting image), there is a possibility that the second size parameter is inconsistent with the first size parameter, which will directly affect the accuracy of target tracking. The inconsistency between the second size parameter and the first size parameter here can be understood as the different target frame sizes corresponding to the target to be tracked. When the second size parameter is inconsistent with the first size parameter, it means that the size of the target to be tracked has changed. At this time, it is necessary to redetermine the target to be tracked. Similarly, the originally determined target to be tracked is regarded as the first target, and the re-detected target is regarded as the second target. The redetermination of the second target can be achieved by intercepting an image within a certain range around the first target and performing target detection using the target detection model. After the second target is determined, the corresponding position parameters and size parameters of the second target are generated.

[0173] It's worth noting that this doesn't simply replace the second target with the new one to be tracked. Instead, the target tracking model first determines whether the size parameters corresponding to the second target and the second size parameters corresponding to the original first target meet the preset target tracking parameter replacement conditions. Specifically, the target tracking model determines whether the size corresponding to the second target and the second size corresponding to the original first target intersect in space and meet specific criteria.

[0174] Specifically, assume that the second size of the first target (the original target to be tracked) is represented by a rectangular box [x1, y1, x2, y2], (x1, y1) represents the upper left corner of the first target, and (x2, y2) represents the lower right corner of the first target; the size of the second target (the re-detected target) is represented by a rectangular box [x3, y3, x4, y4], (x3, y3) represents the upper left corner of the second target, and (x4, y4) represents the lower right corner of the second target. The target tracking model judgment criteria are as follows:

[0175]

[0176] Where area_inter represents the area of the overlapped portion of the first and second targets at the second size, area_T represents the area of the second size, min() represents the minimum value, and max() represents the maximum value.

[0177] After being processed by the target tracking model, when the size corresponding to the second target and the second size corresponding to the first target intersect in space and meet the judgment criteria, the size of the first target is replaced with the size parameters corresponding to the second target, and the position parameters of the first target remain unchanged and remain the second position parameters. That is, the size parameters of the target to be tracked that are re-determined are the size parameters corresponding to the re-detected target (second target); the position parameters are the position parameters (second position parameters) of the original tracked target calculated based on the tracking points. Afterwards, the size parameters and position parameters input by the target tracking model are respectively updated to the size parameters and second position parameters corresponding to the re-detected target, and tracking is carried out based on the data obtained by dimensionality reduction fitting of the next frame of image. In this way, target tracking failures caused by target deformation and size transformation are effectively avoided.

[0178] Furthermore, in a preferred embodiment provided in the present application, the tracking module 14 is used to track the determined target to be tracked based on the visible light-infrared imaging data obtained by dimensionality reduction fitting and through the target tracking model, and is specifically used to track the determined target to be tracked based on the visible light-infrared imaging data obtained by dimensionality reduction fitting and through the Median-Flow target tracking model.

[0179] The Median-Flow target tracking model is based on the Median-Flow tracking algorithm. This algorithm can track the size changes of the target frame, thereby reducing the impact of target frame size changes on target accuracy.

[0180] It should be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, product, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, product, or apparatus. In the absence of further limitations, the phrase "comprises a..." to define an element does not preclude the presence of other identical elements in the process, method, product, or apparatus that includes the element.

[0181] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A target tracking method, characterized in that: The following steps are involved: Acquire three-color channel imaging data of visible light images and imaging data of infrared images; Performing data dimensionality increase processing on the acquired visible light imaging data and infrared imaging data through a data dimensionality increase fitting model to obtain visible light-infrared four-dimensional imaging data containing visible light imaging data information and infrared imaging data information; Determining a target to be tracked based on the obtained four-dimensional data and through a target detection model; Performing data dimensionality reduction processing on the acquired visible light imaging data and infrared imaging data by using a data dimensionality reduction fitting model to obtain visible light-infrared imaging data containing visible light imaging data information and infrared imaging data information; Tracking the determined target to be tracked based on the visible light-infrared imaging data obtained by the dimensionality reduction fitting and using a target tracking model; The data dimension increasing fitting model is used to perform data dimension increasing processing on the obtained visible light imaging data and infrared imaging data to obtain visible light-infrared four-dimensional imaging data containing visible light imaging data information and infrared imaging data information, specifically including: The infrared image data is scaled by a linear scaling model to obtain infrared image data with a resolution consistent with that of the visible light image. Performing data dimensionality increase processing on the acquired visible light imaging data and the infrared image data obtained by the scaling process through a data dimensionality increase fitting model to obtain visible light-infrared four-dimensional imaging data containing visible light imaging data information and infrared imaging data information; The data dimension increasing fitting model is used to perform data dimension increasing processing on the acquired visible light imaging data and the infrared image data obtained by the scaling process to obtain visible light-infrared four-dimensional imaging data containing visible light imaging data information and infrared imaging data information, specifically including: Converting the infrared image data obtained by the scaling process into infrared image data registered with the visible light image through an image registration model; The acquired visible light imaging data and the registered infrared image data are subjected to data dimension increasing processing by using a data dimension increasing fitting model to obtain visible light-infrared four-dimensional imaging data containing visible light imaging data information and infrared imaging data information.

2. The target tracking method according to claim 1, wherein: Tracking the determined target to be tracked based on the visible light-infrared imaging data obtained by the dimensionality reduction fitting and using a target tracking model specifically includes: Obtaining first position parameters and first size parameters of a target to be tracked in a first fitting image corresponding to the visible light-infrared four-dimensional imaging data; screening tracking points of the determined target to be tracked according to the first position parameter and the first size parameter and using a target tracking model to obtain a target tracking point set consisting of a plurality of tracking points for tracking the determined target; Calculating a second position parameter of the target to be tracked in the next frame of image based on the target tracking point set and using a target tracking model; determining, based on the second position parameter and using a target tracking model, whether the target to be tracked exceeds a second fitting image range corresponding to the visible light-infrared imaging data obtained by the dimensionality reduction fitting in the next frame of image; When it is determined that the target to be tracked is beyond the range of the second fitting image, the target to be tracked is re-determined based on the visible light-infrared four-dimensional imaging data and the target detection model.

3. The target tracking method according to claim 2, wherein: The method further includes tracking the determined target to be tracked based on the visible light-infrared imaging data obtained by the dimensionality reduction fitting and using a target tracking model: When it is determined that the target to be tracked does not exceed the range of the second fitting image, calculating a second size parameter of the target to be tracked in the second fitting image based on the target tracking point set and using the target tracking model; determining, based on the second size parameter and using a target tracking model, whether the first size parameter is consistent with the second size parameter; When the first size parameter is inconsistent with the second size parameter, the target to be tracked is re-determined based on the visible light-infrared four-dimensional imaging data and the target detection model.

4. The target tracking method according to claim 3, wherein: The data dimensionality reduction fitting model is used to perform data dimensionality reduction processing on the acquired visible light imaging data and infrared imaging data to obtain visible light-infrared imaging data containing visible light imaging data information and infrared imaging data information, specifically including: The infrared image data is scaled by a linear scaling model to obtain infrared image data with a resolution consistent with that of the visible light image. The acquired visible light imaging data and the infrared image data obtained by the scaling process are subjected to data dimensionality reduction processing by using a data dimensionality reduction fitting model to obtain visible light-infrared imaging data containing visible light imaging data information and infrared imaging data information.

5. The target tracking method according to claim 4, wherein: The data dimension reduction fitting model is used to perform data dimension reduction processing on the acquired visible light imaging data and the infrared image data obtained by the scaling process to obtain visible light-infrared imaging data containing visible light imaging data information and infrared imaging data information, specifically including: Converting the infrared image data obtained by the scaling process into infrared image data registered with the visible light image through an image registration model; The acquired visible light imaging data and the registered infrared image data are subjected to data dimensionality reduction processing by using a data dimensionality reduction fitting model to obtain visible light-infrared imaging data containing visible light imaging data information and infrared imaging data information.

6. The target tracking method according to claim 5, wherein: Based on the obtained four-dimensional data, and through the target detection model, the target to be tracked is determined, specifically including: Based on the obtained four-dimensional data and the YOLO_V5s target detection model, the target to be tracked is determined.

7. The target tracking method according to claim 6, wherein: Tracking the determined target to be tracked based on the visible light-infrared imaging data obtained by the dimensionality reduction fitting and using a target tracking model, specifically includes: According to the visible light-infrared imaging data obtained by the dimensionality reduction fitting, the determined target to be tracked is tracked through the Median-Flow target tracking model.

8. A target tracking device, characterized in that: include: An acquisition module, used to acquire three-color channel imaging data of a visible light image and imaging data of an infrared image; a computing module for performing data dimensionality increase processing on the acquired visible light imaging data and infrared imaging data by using a data dimensionality increase fitting model to obtain visible light-infrared four-dimensional imaging data containing visible light imaging data information and infrared imaging data information; and for performing data dimensionality reduction processing on the acquired visible light imaging data and infrared imaging data by using a data dimensionality reduction fitting model to obtain visible light-infrared imaging data containing visible light imaging data information and infrared imaging data information; The data dimension increasing fitting model is used to perform data dimension increasing processing on the obtained visible light imaging data and infrared imaging data to obtain visible light-infrared four-dimensional imaging data containing visible light imaging data information and infrared imaging data information, specifically including: The infrared image data is scaled by a linear scaling model to obtain infrared image data with a resolution consistent with that of the visible light image. Performing data dimensionality increase processing on the acquired visible light imaging data and the infrared image data obtained by the scaling process through a data dimensionality increase fitting model to obtain visible light-infrared four-dimensional imaging data containing visible light imaging data information and infrared imaging data information; The data dimension increasing fitting model is used to perform data dimension increasing processing on the acquired visible light imaging data and the infrared image data obtained by the scaling process to obtain visible light-infrared four-dimensional imaging data containing visible light imaging data information and infrared imaging data information, specifically including: Converting the infrared image data obtained by the scaling process into infrared image data registered with the visible light image through an image registration model; Performing data dimensionality increase processing on the acquired visible light imaging data and the registered infrared image data through a data dimensionality increase fitting model to obtain visible light-infrared four-dimensional imaging data containing visible light imaging data information and infrared imaging data information; A detection module, configured to determine a target to be tracked based on the obtained four-dimensional data and through a target detection model; The tracking module is used to track the determined target to be tracked based on the visible light-infrared imaging data obtained by the dimensionality reduction fitting and through a target tracking model.

Images