An 8K intelligent target tracking method and system

Abstract

The present invention discloses an 8K intelligent target tracking method and system, the method comprising: obtaining a first infrared image taken by an infrared camera and a first 8K image taken by an 8K camera; determining a common center point of the first infrared image and the first 8K image by image registration; selecting a target on the center point and identifying the target; tracking the identified target by a target tracking algorithm and predicting a motion path of the identified target; adjusting the angle of the pan / tilt according to the predicted motion path; obtaining a second infrared image taken by the infrared camera after adjusting the pan / tilt angle and a second 8K image taken by the 8K camera; obtaining a second infrared feature of the second infrared image and a second 8K feature of the center point of the second 8K image; verifying whether the second infrared feature and the second 8K feature are consistent by multi-spectral target. The invention has the beneficial effect of solving the problem of high-efficiency and high-accuracy tracking when tracking a target.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an 8K intelligent target tracking method and system. Background Art

[0002] The tracking systems currently used are usually equipped with broadcast-grade HD cameras and lenses with high-power zoom and automatic dimming functions. These devices can automatically adjust the lens's focal length and aperture size according to the target's real-time position and speed to ensure that the captured images are clear and stable.

[0003] However, there are still some shortcomings in the actual application process. The first and foremost is the insufficient tracking accuracy. For example, when the target moves quickly or performs complex movements, the device may experience tracking delays or misjudgments, causing the captured image to appear jittery or lose key movements.

[0004] Based on this, the present invention proposes an 8K intelligent target tracking method and system, which solves the problem of quickly tracking the target and acquiring the target image. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the prior art and provide an 8K intelligent target tracking method and system that is easy to operate, thereby solving the problem of quickly tracking and acquiring target images.

[0006] The purpose of the present invention is achieved through the following technical solutions: an 8K intelligent target tracking method, comprising:

[0007] S1. Acquire a first infrared image taken by an infrared camera and a first 8K image taken by an 8K camera, where the infrared camera and the 8K camera are located on the same pan / tilt;

[0008] S2. Determine a common center point of the first infrared image and the first 8K image through image registration; and adjust the imaging center of the infrared camera and the imaging center of the 8K camera to the center point according to the center point;

[0009] S3, select the target at the center point and mark the target;

[0010] S4, tracking the identified target and predicting the movement path of the identified target through a target tracking algorithm;

[0011] S5, adjusting the angle of the gimbal according to the predicted motion path;

[0012] S6, obtaining a second infrared image taken by the infrared camera after adjusting the pan / tilt angle and a second 8K image taken by the 8K camera;

[0013] S7, acquiring a second infrared feature on the second infrared image and a second 8K feature on the second 8K image;

[0014] S8. Verify whether the second infrared feature and the second 8K feature are consistent through a multi-spectral target;

[0015] If the second infrared feature and the second 8K feature are consistent, the multi-spectral target verification is passed and they are determined to be the same target, that is, the same identification target, and the second 8K image is output to the display screen and steps S4 to S8 are repeated;

[0016] If the second infrared feature is inconsistent with the second 8K feature and the multi-spectral target verification fails, it is determined that the target is lost, and the process returns to step S2.

[0017] Furthermore, the image registration determines a common center point of the first infrared image and the first 8K image, including:

[0018] S201, acquiring a first infrared feature of a first infrared image and a first 8K feature of a first 8K image;

[0019] S202, obtaining a matching feature point pair of the first infrared feature and the second 8K feature through a feature matching algorithm;

[0020] S203, calculating a transformation matrix based on the matched feature point pairs through epipolar geometry;

[0021] S204, obtaining a common center point through transformation matrix calculation.

[0022] Furthermore, the loss function in the target tracking algorithm includes a mean square error and a structural similarity index.

[0023] Further, verifying whether the second infrared feature and the second 8K feature are consistent through the multi-spectral target includes:

[0024] S801, obtaining a central area feature in a second infrared feature and marking it as a first preset target;

[0025] S802, obtaining a central area feature in the second 8K feature and marking it as a second preset target;

[0026] S803, comparing the contour features of the first preset target and the contour features of the second preset target to see whether they are consistent;

[0027] If the profile features are consistent, the multispectral target verification passes;

[0028] If the profile features are inconsistent, the multispectral target verification fails.

[0029] The present invention also provides an 8K intelligent target tracking system, which is characterized by:

[0030] The system includes: an 8K imaging module, a multispectral imaging module, an image registration module, a target tracking module and a multispectral target verification module;

[0031] The 8K imaging module is used to obtain a first 8K image and a second 8K image of the area to be detected;

[0032] The multispectral imaging module is used to obtain a first infrared image and a second infrared image of the area to be detected;

[0033] The image registration module is used to obtain a common center point of the first infrared image and the first 8K image, and adjust the imaging center of the infrared camera and the imaging center of the 8K camera to the common center point;

[0034] The target tracking module tracks the identified target and predicts the movement path of the identified target through a target tracking algorithm, and adjusts the pan / tilt rotation to track the target;

[0035] The multi-spectral target verification module determines whether the target is lost by verifying the second infrared feature of the second image and the second 8K feature of the second 8K image.

[0036] Furthermore, the loss function in the target tracking module includes average error and structural similarity.

[0037] Furthermore, the image registration module is constructed using a HomographyNe regression network, and the HomographyNe regression network includes:

[0038] A feature extraction layer, used to obtain feature points of the first 8K image or the first infrared image;

[0039] Feature description layer, obtaining feature point descriptors through feature points;

[0040] Feature point matching layer, which obtains matching feature point pairs through feature points and feature point descriptors;

[0041] The geometry acquisition layer obtains the transformation matrix through feature point pairs;

[0042] The center point acquisition layer is used to obtain the common center point according to the transformation matrix.

[0043] The present invention has the following advantages:

[0044] (1) The present invention constructs an image configuration module through a HomographyNe regression network and adopts a five-layer neural network architecture to reduce the number of layers of the neural network, thereby achieving lightweight processing of the module; when performing data processing, the amount of calculation is reduced and the calculation speed is improved.

[0045] (2) The present invention introduces a loss function, which is composed of an average error and a structural similarity index, and the tracking accuracy can reach 99.517%. The center positioning of the tracked target is used as an index. After adopting the loss function, the center positioning error is reduced by 94.6% compared with when only the average error is used to construct the loss function. Therefore, the tracking accuracy and tracking efficiency are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a diagram of the method steps of the present invention;

[0047] Figure 2 It is the center positioning error graph of various loss functions;

[0048] Table 1 is a comparative experimental data table;

[0049] Figure 3 It is a schematic diagram of the tracking window overlap rate;

[0050] Figure 4 It is a schematic diagram of the accuracy curve;

[0051] Figure 5 It is a success rate diagram;

[0052] Figure 6 It is a schematic diagram of missed detection rate;

[0053] Figure 7 It is a schematic diagram of the structure of the present invention. DETAILED DESCRIPTION

[0054] The present invention is further described below in conjunction with the accompanying drawings, but the protection scope of the present invention is not limited to the following description.

[0055] It should be noted that the directions or positional relationships indicated by "left", "right", etc. are based on the directions or positional relationships shown in the accompanying drawings, or are the directions or positional relationships in which the inventive product is usually placed when in use, or are the directions or positional relationships commonly understood by those skilled in the art. Such terms are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as a limitation on the present invention.

[0056] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features and technical solutions in the embodiments may be combined with each other.

[0057] First aspect

[0058] An 8K intelligent target tracking method, comprising:

[0059] S1. Acquire a first infrared image taken by an infrared camera and a first 8K image taken by an 8K camera, where the infrared camera and the 8K camera are located on the same pan / tilt;

[0060] S2. Determine a common center point of the first infrared image and the first 8K image through image registration; and adjust the imaging center of the infrared camera and the imaging center of the 8K camera to the center point according to the center point;

[0061] S3, select the target at the center point and mark the target;

[0062] S4, tracking the identified target and predicting the movement path of the identified target through a target tracking algorithm;

[0063] S5, adjusting the angle of the gimbal according to the predicted motion path;

[0064] S6, obtaining a second infrared image taken by the infrared camera after adjusting the pan / tilt angle and a second 8K image taken by the 8K camera;

[0065] S7, acquiring a second infrared feature on the second infrared image and a second 8K feature on the second 8K image;

[0066] S8. Verify whether the second infrared feature and the second 8K feature are consistent through a multi-spectral target;

[0067] If the second infrared feature and the second 8K feature are consistent, the multi-spectral target verification is passed and they are determined to be the same target, that is, the same identification target, and the second 8K image is output to the display screen and steps S4 to S8 are repeated;

[0068] If the second infrared feature is inconsistent with the second 8K feature and the multi-spectral target verification fails, it is determined that the target is lost, and the process returns to step S2.

[0069] The present invention obtains infrared images through an infrared camera and obtains 8K images through an 8K camera. The infrared camera and the 8K camera are installed on the same pan / tilt and the lenses are set on the same horizontal line.

[0070] The ultra-high-definition 8K camera head has a resolution of up to 7680×4320, and each frame of the image contains up to 33 million pixels; this makes the color of the 8K image more realistic and the audio clearer; the first infrared image and the first 8K are configured to obtain a common center point, and the imaging center of the infrared camera and the imaging center of the 8K camera are adjusted to the center point according to the center point to ensure that the center point and frame of the imaging picture are consistent;

[0071] According to the characteristics of the image, through the single target tracking algorithm in the target tracking algorithm, similar feature search is performed in the infrared image subsequently acquired by the infrared camera according to the first infrared feature to gradually iterate and realize target positioning, and the future motion trajectory of the identified target is predicted according to the motion trajectory of the identified target that has been acquired, and the motion direction of the identified target at the next moment is obtained, and then the pan-tilt rotation is adjusted according to the predicted motion direction of the target at the next moment, and the infrared camera and the 8K camera are aimed at the target to prevent the target from being lost; the second infrared image and the second 8K image are obtained again through the infrared camera and the 8K image, and the second infrared feature of the second infrared image and the second 8K feature of the second 8K image are verified through the multi-spectral target to determine whether the target is lost; if the target is not lost, the motion trajectory of the target is continued to be predicted to obtain the latest image, and if the target is lost, the center point is recalibrated; the accuracy of the target tracking of the present invention is further guaranteed;

[0072] The present invention introduces an infrared camera to obtain the characteristics of the infrared band of the target, and tracks and locates the target through the infrared band; infrared imaging is not affected by factors such as visible light and shadows, and has strong anti-interference ability; and infrared imaging obtains the first infrared characteristic or the second infrared characteristic through the infrared heat of the target, so as to achieve sub-pixel high-precision positioning.

[0073] Furthermore, the image registration determines a common center point of the first infrared image and the first 8K image, including:

[0074] S201, acquiring a first infrared feature of a first infrared image and a first 8K feature of a first 8K image;

[0075] S202, obtaining a matching feature point pair of the first infrared feature and the second 8K feature through a feature matching algorithm;

[0076] S203, calculating a transformation matrix based on the matched feature point pairs through epipolar geometry;

[0077] S204, obtaining a common center point through transformation matrix calculation.

[0078] Furthermore, the loss function in the target tracking algorithm includes a mean square error and a structural similarity index.

[0079] like Figure 2As shown, the center positioning of the target is selected as an indicator, and the mean value of the center positioning error is calculated to be: 0.027125 (m); by constructing a variety of loss functions, it can be seen that when the mean square error MSE and the structural similarity index SSIM are selected by the present invention, the error is reduced by 94.6% compared with the 0.50625 (m) error when only the mean error MSE is used to construct the loss function; it is reduced by 93.2% compared with the 0.399375 (m) when the quantile loss QL (any other loss function) + the structural similarity index SSIM is used; it is reduced by 80.3% compared with the 0.13775 (m) when the mean error MSE + the quantile loss QL (any other loss function) is used; therefore, after introducing the loss function of the present invention, the center positioning error after target tracking is significantly reduced.

[0080] Referring to Table 1, further verification experiments were conducted to consider the improved target tracking from multiple dimensions including tracking frame overlap rate, accuracy rate, success rate, missed detection rate and real-time performance. The real-time performance showed little significant difference, so no further analysis was performed.

[0081] See also Figure 3 It can be concluded that after using the mean square error + structural similarity index (MSE+SSIM), the tracking window overlap rate difference has been significantly improved, and the average tracking window overlap rate is: 93%, which is 41% higher than the 52% when only the mean error is used to construct the loss function MSE; 33% higher than the 60% when the mean error + logarithmic loss function (MSE+LogLoss) is used; 19% higher than the 74% when the logarithm + structural similarity index (LogLoss+SSIM) is used;

[0082] See Figure 4 It can be concluded that after the present invention adopts the mean square error + structural similarity index (MSE + SSIM), the accuracy of the target tracking algorithm has been significantly improved, and the average value of the accuracy curve is: 96.58%, which is 33.03% higher than 63.55% when only the mean error is used to construct the loss function MSE; 15.86% higher than 80.73% when the mean error + logarithmic loss function (MSE + LogLoss) is used; 12.87% higher than 83.72% when the logarithm + structural similarity index (LogLoss + SSIM) is used;

[0083] See Figure 5After using the mean square error + structural similarity index (MSE+SSIM), the accuracy of the target tracking algorithm has been significantly improved. The average value of the accuracy curve is: 94.24%, which is 23.71% higher than the 70.53% when only the mean error is used to construct the loss function MSE; 21.39% higher than the 72.86% when the mean error + logarithmic loss function (MSE+LogLoss) is used; 11.07% higher than the 83.18% when the logarithm + structural similarity index (LogLoss+SSIM) is used;

[0084] See Figure 6 It can be seen that after using the mean square error + structural similarity index (MSE+SSIM), the missed detection rate of the target tracking algorithm has been significantly reduced, and the average missed detection rate is: 1.25%, which is 2.88% lower than 4.13% when only the mean error is used to construct the loss function MSE; it is 3.16% lower than 3.16% when the mean error + logarithmic loss function (MSE+LogLoss) is used; it is 0.21% lower than 1.46% when the logarithm + structural similarity index (LogLoss+SSIM) is used.

[0085] Further, verifying whether the second infrared feature and the second 8K feature are consistent through the multi-spectral target includes:

[0086] S801, obtaining a central area feature in a second infrared feature and marking it as a first preset target;

[0087] S802, obtaining a central area feature in the second 8K feature and marking it as a second preset target;

[0088] S803, comparing the contour features of the first preset target and the contour features of the second preset target to see whether they are consistent;

[0089] If the profile features are consistent, the multispectral target verification passes;

[0090] If the profile features are inconsistent, the multispectral target verification fails;

[0091] By verifying the characteristics of infrared light and visible light, i.e. the 8K characteristics of 8K images, accurate target detection can be performed.

[0092] Second aspect

[0093] The present invention also provides an 8K intelligent target tracking system, which is characterized by:

[0094] The system includes: an 8K imaging module, a multispectral imaging module, an image registration module, a target tracking module and a multispectral target verification module;

[0095] The 8K imaging module is used to obtain a first 8K image and a second 8K image of the area to be detected;

[0096] The multispectral imaging module is used to obtain a first infrared image and a second infrared image of the area to be detected;

[0097] The image registration module is used to obtain a common center point of the first infrared image and the first 8K image, and adjust the imaging center of the infrared camera and the imaging center of the 8K camera to the common center point;

[0098] The target tracking module tracks the identified target and predicts the movement path of the identified target through a target tracking algorithm, and adjusts the pan / tilt rotation to track the target;

[0099] The multi-spectral target verification module determines whether the target is lost by verifying the second infrared feature of the second image and the second 8K feature of the second 8K image.

[0100] Furthermore, the loss function in the target tracking module includes average error and structural similarity.

[0101] Furthermore, the image registration module is constructed using a HomographyNe regression network, and the HomographyNe regression network includes:

[0102] A feature extraction layer, used to obtain feature points of the first 8K image or the first infrared image;

[0103] Feature description layer, obtaining feature point descriptors through feature points;

[0104] Feature point matching layer, which obtains matching feature point pairs through feature points and feature point descriptors;

[0105] The geometry acquisition layer obtains the transformation matrix through feature point pairs;

[0106] The center point acquisition layer is used to obtain the common center point according to the transformation matrix.

[0107] The present invention constructs an image configuration module through a HomographyNe regression network, adopts a five-layer neural network architecture, reduces the number of layers of the neural network, and achieves lightweight processing of the module; when performing data processing, it also reduces the amount of calculation and improves the calculation speed.

[0108] The above embodiments only express preferred implementation modes, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present invention. It should be pointed out that, for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention.

Claims

1. An 8K intelligent target tracking method, characterized in that: include: S1. Acquire a first infrared image taken by an infrared camera and a first 8K image taken by an 8K camera, where the infrared camera and the 8K camera are located on the same pan / tilt; S2. Determine a common center point of the first infrared image and the first 8K image through image registration; and adjust the imaging center of the infrared camera and the imaging center of the 8K camera to the center point according to the center point; S3, select the target at the center point and mark the target; S4, tracking the identified target and predicting the movement path of the identified target through a target tracking algorithm; S5, adjusting the angle of the gimbal according to the predicted motion path; S6, obtaining a second infrared image taken by the infrared camera after adjusting the pan / tilt angle and a second 8K image taken by the 8K camera; S7, acquiring a second infrared feature on the second infrared image and a second 8K feature on the second 8K image; S8. Verify whether the second infrared feature and the second 8K feature are consistent through a multi-spectral target; If the second infrared feature and the second 8K feature are consistent, the multi-spectral target verification is passed and they are determined to be the same target, that is, the same identification target, and the second 8K image is output to the display screen and steps S4 to S8 are repeated; If the second infrared feature and the second 8K feature are inconsistent, the multi-spectral target verification fails and the target is determined to be lost, and the process returns to step S2; Among them, the loss function in the target tracking algorithm includes mean square error and structural similarity index.

2. The 8K intelligent target tracking method according to claim 1, characterized in that: The image registration determines a common center point of the first infrared image and the first 8K image, including: S201, acquiring a first infrared feature of a first infrared image and a first 8K feature of a first 8K image; S202, obtaining a matching feature point pair of the first infrared feature and the second 8K feature through a feature matching algorithm; S203, calculating a transformation matrix based on the matched feature point pairs through epipolar geometry; S204, obtaining a common center point through transformation matrix calculation.

3. The 8K intelligent target tracking method according to claim 2, characterized in that: Verifying whether the second infrared feature and the second 8K feature are consistent through the multi-spectral target includes: S801, obtaining a central area feature in a second infrared feature and marking it as a first preset target; S802, obtaining a central area feature in the second 8K feature and marking it as a second preset target; S803, comparing the contour features of the first preset target and the contour features of the second preset target to see whether they are consistent; If the profile features are consistent, the multispectral target verification passes; If the profile features are inconsistent, the multispectral target verification fails.

4. An 8K intelligent target tracking system, characterized by: The system includes: an 8K imaging module, a multispectral imaging module, an image registration module, a target tracking module and a multispectral target verification module; The 8K imaging module is used to obtain a first 8K image and a second 8K image of the area to be detected; The multispectral imaging module is used to obtain a first infrared image and a second infrared image of the area to be detected; The image registration module is used to obtain a common center point of the first infrared image and the first 8K image, and adjust the imaging center of the infrared camera and the imaging center of the 8K camera to the common center point; The target tracking module tracks the identified target and predicts the movement path of the identified target through a target tracking algorithm, and adjusts the pan / tilt rotation to track the target; The multi-spectral target verification module determines whether the target is lost by verifying the second infrared feature of the second image and the second 8K feature of the second 8K image; Among them, the loss function in the target tracking module includes average error and structural similarity.

5. The 8K intelligent target tracking system according to claim 4, characterized in that: The image registration module is constructed using a HomographyNe regression network, and the HomographyNe regression network includes: A feature extraction layer, used to obtain feature points of the first 8K image or the first infrared image; Feature description layer, obtaining feature point descriptors through feature points; Feature point matching layer, which obtains matching feature point pairs through feature points and feature point descriptors; The geometry acquisition layer obtains the transformation matrix through feature point pairs; The center point acquisition layer is used to obtain the common center point according to the transformation matrix.

Images