Target tracking method and device based on visible light and ultraviolet light

Abstract

The invention provides a target tracking method and device based on visible light and ultraviolet light. The method comprises the following steps: acquiring a visible light modal image and an ultraviolet light modal image in real time; extracting a first template image covering a to-be-tracked target from the mth frame of visible light modal image, and extracting a second template image covering the to-be-tracked target from the mth frame of ultraviolet ray modal image; and inputting the first template image, the second template image, the nth frame of visible light modal image and the nth frame of ultraviolet ray modal image into a target tracking model for target tracking so as to obtain an nth frame of prediction result image, and marking the prediction position of the target to be tracked on the prediction result image. In the scheme, the target tracking model is used for processing the visible light modal image and the ultraviolet light modal image to track the to-be-tracked target, and even if the to-be-tracked target is mixed with the background color, the visible light and the ultraviolet light can be used for tracking the to-be-tracked target, so that the target tracking effect is improved, and the limitation of target tracking is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of target tracking, and in particular to a target tracking method and device based on visible light and ultraviolet light. Background Art

[0002] With the rapid development of visual target tracking, more and more target tracking algorithms have emerged. The current target tracking algorithms are mainly based on visible light to achieve target tracking.

[0003] However, in the process of tracking the target, if the target and the background colors are confused, the existing target tracking algorithm cannot distinguish the target and the background based on visible light, which leads to the inability to continue target tracking, and the target tracking effect is poor and has great limitations. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a target tracking method and device based on visible light and ultraviolet light to solve the problems of poor target tracking effect and large limitations existing in the current target tracking algorithm.

[0005] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0006] A first aspect of an embodiment of the present invention discloses a target tracking method based on visible light and ultraviolet light, the method comprising:

[0007] Real-time acquisition of visible light modality images and ultraviolet modality images;

[0008] Extracting a first template image covering the target to be tracked from the m-th frame of the visible light modality image, and extracting a second template image covering the target to be tracked from the m-th frame of the ultraviolet modality image;

[0009] The first template image, the second template image, the n-th frame of the visible light modal image and the n-th frame of the ultraviolet modal image are input into a preset target tracking model for target tracking to obtain an n-th frame of prediction result image, wherein n is greater than m, and the prediction result image is annotated with the predicted position of the target to be tracked. The target tracking model is obtained by training a specified neural network model based on a specific data set, and the specified neural network model at least includes a specified deep learning network and an adaptive feature selection mechanism.

[0010] Preferably, real-time acquisition of visible light modality images and ultraviolet modality images includes:

[0011] The visible light lens is called to collect visible light modal images in real time, and the ultraviolet light lens is called to collect ultraviolet modal images in real time.

[0012] Preferably, extracting a first template image covering the target to be tracked from the m-th frame of the visible light modality image, and extracting a second template image covering the target to be tracked from the m-th frame of the ultraviolet modality image, comprises:

[0013] In response to a labeling operation triggered by a user, an image area covering the target to be tracked is extracted from the m-th frame of the visible light modality image to obtain a first template image, and an image area covering the target to be tracked is extracted from the m-th frame of the ultraviolet modality image to obtain a second template image.

[0014] Preferably, the process of training a specified neural network model based on a specific data set to obtain a target tracking model includes:

[0015] Acquire a specific data set, wherein the specific data set includes a plurality of image sequences carrying real annotations, any of the image sequences in the specific data set is set with a challenge attribute, and any of the image sequences includes a plurality of frames of continuous sample visible light modality images or sample ultraviolet modality images;

[0016] The specified neural network model is trained using the specific data set until the specified neural network model converges to obtain a target tracking model.

[0017] Preferably, the prediction result image of the nth frame is: the visible light modality image of the nth frame marked with the predicted position of the target to be tracked, or the ultraviolet modality image of the nth frame marked with the predicted position of the target to be tracked.

[0018] Preferably, after obtaining the n-th frame prediction result image, the method further includes:

[0019] The output displays the predicted result image of the nth frame.

[0020] A second aspect of an embodiment of the present invention discloses a target tracking device based on visible light and ultraviolet light, the device comprising:

[0021] An acquisition unit, used for acquiring visible light modality images and ultraviolet light modality images in real time;

[0022] an extraction unit, configured to extract a first template image covering the target to be tracked from the m-th frame of the visible light modality image, and to extract a second template image covering the target to be tracked from the m-th frame of the ultraviolet modality image;

[0023] The tracking unit is used to input the first template image, the second template image, the n-th frame of the visible light modality image and the n-th frame of the ultraviolet modality image into a preset target tracking model for target tracking to obtain an n-th frame of prediction result image, wherein n is greater than m, and the prediction result image is annotated with the predicted position of the target to be tracked. The target tracking model is obtained by training a specified neural network model based on a specific data set, and the specified neural network model at least includes a specified deep learning network and an adaptive feature selection mechanism.

[0024] Preferably, the acquisition unit is specifically used to: call a visible light lens to collect visible light modal images in real time, and call an ultraviolet light lens to collect ultraviolet modal images in real time.

[0025] Preferably, the extraction unit is specifically used to: in response to a labeling operation triggered by a user, extract an image area covering the target to be tracked from the visible light modal image of the mth frame to obtain a first template image, and extract an image area covering the target to be tracked from the ultraviolet modal image of the mth frame to obtain a second template image.

[0026] Preferably, the tracking unit comprises:

[0027] An acquisition subunit is used to acquire a specific data set, wherein the specific data set includes a plurality of image sequences carrying real annotations, any of the image sequences in the specific data set is set with a challenge attribute, and any of the image sequences includes a plurality of frames of continuous sample visible light modality images or sample ultraviolet modality images;

[0028] The training subunit is used to train the specified neural network model using the specific data set until the specified neural network model converges to obtain a target tracking model.

[0029] Based on the above-mentioned embodiment of the present invention, a target tracking method and device based on visible light and ultraviolet light are provided. The method is as follows: real-time acquisition of visible light modal images and ultraviolet light modal images; extracting a first template image covering the target to be tracked from the m-th frame of the visible light modal image, and extracting a second template image covering the target to be tracked from the m-th frame of the ultraviolet light modal image; inputting the first template image, the second template image, the n-th frame of the visible light modal image and the n-th frame of the ultraviolet light modal image into the target tracking model for target tracking to obtain the n-th frame of the prediction result image, and the prediction result image is annotated with the predicted position of the target to be tracked. In this solution, the target tracking model is used to process the visible light modal image and the ultraviolet light modal image to track the target to be tracked. Even if the target to be tracked is confused with the background color, visible light and ultraviolet light can be used to track the target to be tracked, thereby improving the effect of target tracking and reducing the limitations of target tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0031] Figure 1 A comparative example diagram of visible light imaging and ultraviolet imaging provided by an embodiment of the present invention;

[0032] Figure 2 A flow chart of a target tracking method based on visible light and ultraviolet light provided by an embodiment of the present invention;

[0033] Figure 3 A flowchart of a target tracking model obtained by training provided by an embodiment of the present invention;

[0034] Figure 4 An example diagram of some image pairs in a specific data set provided by an embodiment of the present invention;

[0035] Figure 5 An example diagram of the architecture of a specified neural network model provided by an embodiment of the present invention;

[0036] Figure 6 An example diagram of the architecture of the adaptive feature selection mechanism provided by an embodiment of the present invention;

[0037] Figure 7 A structural block diagram of a target tracking device based on visible light and ultraviolet light provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] In this application, the terms "comprises", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element.

[0040] With the rapid development of visual target tracking, more and more target tracking algorithms have emerged. The current target tracking algorithms are mainly based on visible light to track the target. For example, the visible light tracker tracks the target based on visible light.

[0041] However, in the process of tracking the target, if the target is confused with the background color, the existing target tracking algorithm cannot distinguish the target from the background based on visible light, and thus the target cannot be tracked further. For example, in a scene where the target to be tracked is camouflaged in color, the target is confused with the background color, resulting in limited information provided by visible light, and the target cannot be clearly distinguished from the background, so the target cannot be tracked further. Therefore, the existing target tracking algorithm has poor target tracking effect and great limitations.

[0042] The inventors have found through research that UV sensors can be sensitive to some camouflage materials (or UV imaging is more sensitive to some camouflage materials). More specifically, UV imaging can directly generate clear images of objects using only weak UV light sources. UV imaging can capture the UV spectrum with a wavelength range of 10 nanometers to 400 nanometers (the region between the visible spectrum and X-rays).

[0043] like Figure 1 The comparison example diagram of visible light imaging and ultraviolet imaging is shown in FIG. Figure 1 Each column of "image pairs" in contains visible light imaging and ultraviolet imaging of the same scene. In each column of "image pairs", the upper image is the visible light imaging, and the lower image is the ultraviolet imaging.

[0044] from Figure 1 It can be seen that when the target to be tracked ( Figure 1 When the object (the object framed by the red frame) is color camouflaged, the effect of ultraviolet imaging is significantly better than that of visible light imaging; in order to improve the effect of target tracking (equivalent to improving target tracking performance), ultraviolet light can be used as supplementary information of visible light to track the target.

[0045] It should be noted that in ultraviolet imaging, any object reflects ultraviolet light to varying degrees. Since the reflectivity of camouflage objects cannot be consistent with that of all natural environments, the difference between the ultraviolet light reflected by the camouflage objects and the natural environment can be used to achieve ultraviolet-sensitive imaging of the camouflage objects. Ultraviolet imaging of some camouflage objects is more effective than visible light imaging, and visible light and ultraviolet data can be used to complement each other, thereby achieving more robust target tracking effects in most scenarios.

[0046] Therefore, this scheme proposes a target tracking method and device based on visible light and ultraviolet light, which uses a target tracking model to process visible light modal images and ultraviolet modal images to track the target to be tracked. Even if the target to be tracked is confused with the background color, visible light and ultraviolet light can be used to track the target to be tracked, thereby improving the target tracking effect and reducing the limitations of target tracking.

[0047] It should be noted that in order to better understand the subsequent contents of this plan, some of the terms involved in this plan are explained here.

[0048] ultraviolet(UV): ultraviolet;

[0049] visible (RGB): visible light;

[0050] RGB-UV object tracking: visible light and ultraviolet object tracking;

[0051] Transformer: deep learning network structure;

[0052] ViT (Vision Transformer): The specified deep learning network in this solution. ViT is a deep learning network for image feature extraction.

[0053] VUOT dataset (visible-ultraviolet object tracking): a specific dataset used in this solution to train the target tracking model;

[0054] VUTrack algorithm (visible and ultraviolet object tracking): target tracking algorithm based on visible light and ultraviolet.

[0055] See also Figure 2 , shows a flow chart of a target tracking method based on visible light and ultraviolet light provided by an embodiment of the present invention, the target tracking method comprising:

[0056] Step S201: collecting visible light modality images and ultraviolet modality images in real time.

[0057] In the specific implementation of step S201, the visible light lens is called to collect visible light modal images (such as RGB images) in real time, and the ultraviolet light lens is called to collect ultraviolet modal images (referred to as UV images) in real time.

[0058] Specifically, when tracking a target, a visible light lens is used to collect multiple continuous frames of visible light modal images in real time, and an ultraviolet lens is used to collect multiple continuous frames of ultraviolet modal images in real time.

[0059] Step S202: extracting a first template image covering the target to be tracked from the m-th visible light modality image frame, and extracting a second template image covering the target to be tracked from the m-th ultraviolet modality image frame.

[0060] In the process of specifically implementing step S202, in the process of acquiring visible light modal images and ultraviolet modal images for target tracking, in response to a labeling operation triggered by a user, an image area covering the target to be tracked is extracted from the m-th frame of the visible light modal image to obtain a first template image, and an image area covering the target to be tracked is extracted from the m-th frame of the ultraviolet modal image to obtain a second template image, wherein the m-th frame is equivalent to an initial frame.

[0061] Specifically, during the target tracking process, the user marks the image area covering the target to be tracked in the m-th frame visible light modal image and the m-th frame ultraviolet modal image, respectively. In response to the marking operation triggered by the user, the background area except the "image area covering the target to be tracked" in the m-th frame visible light modal image is eliminated to obtain a first template image, and the background area except the "image area covering the target to be tracked" in the m-th frame ultraviolet modal image is eliminated to obtain a second template image.

[0062] For example: in response to a labeling operation triggered by a user, an image area covering the target to be tracked is extracted from the first frame (m=1) of the visible light modality image to obtain a first template image, and an image area covering the target to be tracked is extracted from the first frame of the ultraviolet modality image to obtain a second template image.

[0063] Step S203: inputting the first template image, the second template image, the n-th visible light modality image and the n-th ultraviolet modality image into a preset target tracking model for target tracking to obtain the n-th prediction result image.

[0064] It should be noted that n is greater than m, and the target tracking model is obtained by training a specified neural network model based on a specific data set. The specified neural network model at least includes a specified deep learning network and an adaptive feature selection mechanism.

[0065] In the specific implementation of step S203, after extracting the first template image and the second template image, the first template image, the second template image, the n-th visible light modality image, and the n-th ultraviolet modality image are input into the target tracking model for target tracking, and the n-th prediction result image output by the target tracking model is obtained, and the prediction result image is annotated with the predicted position of the target to be tracked;

[0066] Specifically, the n-th frame prediction result image is: the n-th frame visible light modality image marked with the predicted position of the target to be tracked, or the n-th frame ultraviolet modality image marked with the predicted position of the target to be tracked.

[0067] In some embodiments, after the first template image, the second template image, the n-th visible light modal image frame and the n-th ultraviolet modal image frame are input into the target tracking model, the target tracking model performs feature matching in the search area (equivalent to the entire n-th visible light modal image frame) shown in the n-th visible light modal image based on the first template image, thereby predicting the position of the target to be tracked in the n-th visible light modal image frame (referred to as the predicted position) and marking it; at the same time, the target tracking model performs feature matching in the search area shown in the n-th ultraviolet modal image frame based on the second template image, thereby predicting the position of the target to be tracked in the n-th ultraviolet modal image frame (referred to as the predicted position) and marking it.

[0068] The target tracking model outputs "the nth frame of visible light modal image marked with the predicted position of the target to be tracked" or "the nth frame of ultraviolet modal image marked with the predicted position of the target to be tracked" as the nth frame prediction result image according to the confidence of the predicted position of the target to be tracked in the nth frame of visible light modal image and the confidence of the predicted position of the target to be tracked in the nth frame of ultraviolet modal image.

[0069] In other words, the predicted position of the target to be tracked in the n-th frame visible light modal image and the predicted position of the target to be tracked in the n-th frame ultraviolet modal image are equivalent to two prediction results; the target tracking model outputs any one prediction result (usually the prediction result with the highest confidence) as the prediction result image of the n-th frame based on the confidence of the two prediction results.

[0070] In practical applications, one of the application scenarios of this solution is:

[0071] Extracting an image region covering the target to be tracked from the first frame (m=1) of visible light modality image acquired in real time to obtain a first template image, and extracting an image region covering the target to be tracked from the first frame of ultraviolet modality image to obtain a second template image;

[0072] Inputting the first template image, the second template image, each frame (nth frame, n is greater than 1) of subsequently acquired visible light modality image and ultraviolet modality image into the target tracking model for target tracking, and obtaining a multi-frame prediction result image;

[0073] By inputting a frame of visible light modality image and ultraviolet modality image into the target tracking model, a frame of prediction result image can be obtained;

[0074] That is, after extracting the first template image and the second template image from the first frame of visible light modal image and the first frame of ultraviolet modal image respectively, each subsequently acquired frame (nth frame, n is greater than 1) of visible light modal image and ultraviolet modal image can be predicted to obtain a frame of prediction result image (equivalent to each subsequent frame of image being able to predict the predicted position of the target to be tracked).

[0075] In some embodiments, after obtaining the n-th frame prediction result image, the n-th frame prediction result image is output and displayed, and the n-th frame prediction result image is marked with the predicted position of the target to be tracked;

[0076] It should be noted that the n-th frame prediction result image can be marked with the predicted position of the target to be tracked in the form of a rectangular frame.

[0077] In an embodiment of the present invention, a target tracking model is used to process visible light modal images and ultraviolet light modal images to track a target. Even if the target is confused with the background color, visible light and ultraviolet light can be used to track the target, thereby improving the effect of target tracking and reducing the limitations of target tracking.

[0078] For the above-mentioned embodiment of the present invention Figure 2 The target tracking model mentioned in step S203 is shown in Figure 3 , shows a flowchart of training a target tracking model, including the following steps:

[0079] Step S301: Acquire a specific data set.

[0080] In the specific implementation process of step S301, a specific data set (such as a VUOT data set) is obtained, which includes multiple image sequences carrying real annotations, and any image sequence in the specific data set is set with a challenge attribute, and any image sequence includes multiple frames of continuous sample visible light modal images or sample ultraviolet modal images.

[0081] It should be noted that when collecting specific data sets, a multi-sensor lens platform with two degrees of freedom for rotation is used to collect images. The ultraviolet lens (also called UV lens) in the multi-sensor lens platform can capture the ultraviolet spectrum with a wavelength range of 20nm-400nm. The resolution of the UV lens can be 1280*720 (for example only); the resolution of the visible light lens in the multi-sensor lens platform can be 1920*1080 (for example only).

[0082] The process of using the multi-sensor lens platform to collect images is as follows: the distance between the multi-sensor lens platform and the target to be tracked (called a sample target) is controlled between 10 meters and 30 meters (the distance is only an example) so that a suitable target size can be captured;

[0083] A specific data set acquired by a multi-sensor lens platform includes multiple image sequences, which in total include multiple image pairs, and these image pairs are stored in a specified resolution and format by scaling;

[0084] It should be noted that the image sequence is composed of continuous image frames; an image pair refers to: a sample visible light modality image and a sample ultraviolet modality image captured in the same scene.

[0085] For example, a specific dataset collects 226 image sequences, which contain 75,370 image pairs in total. These image pairs are scaled and stored in 960*720 resolution and jpg format with a sampling rate of 30fps. The total data volume of the specific dataset is 31.5GB.

[0086] In the subsequent process of training the specified neural network model using the specific data set, the specific data set is divided into a training set, a validation set, and a test set, and real annotations are added to each image sequence. The real annotations are the label boxes of the real positions of the sample targets.

[0087] For example: assuming that a specific dataset collects 226 image sequences, 160 of them are used as training sets, 33 of them are used as validation sets, and the remaining 33 are used as test sets; precise frame-by-frame target annotations are provided for all image sequences, and real annotations (labeled boxes of the real locations of sample targets) are given, and a total of 150,740 frame-level real annotations are added.

[0088] It should be noted that in the process of collecting specific data sets, since the multi-sensor devices in the multi-sensor lens platform cannot be aligned, view differences will occur in the light card aggregation process. Therefore, in specific data sets (such as the VUOT data set), it is necessary to align the visible light modality and the ultraviolet modality of the initial frame of each collected video, and then check and align the video frame by frame. The aligned video is then divided into image sequences to obtain a well-aligned image sequence.

[0089] In practical applications, in order to enable the trained target tracking model to complete target tracking in the case where the target foreground is confused with the background, one of the collection scenes of a specific data set is a natural environment;

[0090] like Figure 4 As can be seen from the example diagram of some image pairs in the specific dataset, one of the collected scenes of the specific dataset includes grass and bushes of different colors. Figure 4 Each column of “image pairs” in contains a sample visible light modality image and a sample UV modality image acquired for the same scene. Figure 4The red box in the figure is the “real annotation”, that is, the red box is the annotation box of the real position of the sample target (the target to be tracked or tracked);

[0091] Figure 4 The sample targets in the dataset are divided into three categories: "green camouflage net", "yellow camouflage net" and "people wearing camouflage clothes". For fixed sample targets, the camera is moved to collect videos, and then a specific data set is collected. Figure 4 The sample targets in the image pairs from left to right are "yellow camouflage net", "green camouflage net" and "person wearing camouflage clothes", that is, the sample target in image pair a is "yellow camouflage net", the sample target in image pair b is "green camouflage net", and the sample target in image pair c is "person wearing camouflage clothes".

[0092] It should be noted that in the same scene, the ultraviolet modality and the visible light modality have different imaging effects on the sample target (the imaging effects of the sample visible light modality image and the sample ultraviolet modality image are inconsistent). Therefore, in order to facilitate the evaluation of the visible light modality and the ultraviolet modality, the contrast effects of the visible light modality and the ultraviolet modality are divided into three levels: A, B, and C;

[0093] If the imaging effect of the sample ultraviolet modal image is better than that of the sample visible light modal image, it is rated as grade A; if the imaging effect of the sample ultraviolet modal image is equivalent to that of the sample visible light modal image, it is rated as grade B; if the imaging effect of the sample ultraviolet modal image is worse than that of the sample visible light modal image, it is rated as grade C; through the above-mentioned grade division, images with different imaging effects in a specific data set can be more evenly distributed in the training set and the test set, thereby achieving data balance during the training process and the testing process.

[0094] In addition, in the process of training a target tracking model using a specific data set, in order to increase the difficulty of target tracking, this scheme also sets challenge attributes (also called sequence attributes) for any image sequence in the specific data set, and the set challenge attributes are partial occlusion (PO), out of sight (OV) and deformation (DEF); by setting challenge attributes for the image sequence, the adaptability of the trained target tracking model can be improved, so that the target tracking model can adapt to more complex tracking scenarios, and enhance the robustness, stability and generalization ability of the target tracking model.

[0095] For example, suppose a specific data set collects 226 image sequences, and the specific information of the specific data set is shown in Table 1.

[0096] Table 1:

[0097]

[0098] It should be noted that the numbers in Table 1 represent the number of image sequences.

[0099] In the process of training the target tracking model using a specific data set, this scheme evaluates the tracking effect through three evaluation indicators: precision rate (PR), normalized precision rate (NPR) and success rate (SR).

[0100] Precision (PR): Precision refers to the percentage of frames where the distance between the predicted position and the ground truth is within a certain threshold range; due to the large number of close-range scenes and relatively large sample objects in a specific dataset, this scheme sets the threshold to 10 pixels when calculating the PR score.

[0101] Normalized Precision (NPR): Since the accuracy metric is easily affected by image resolution and bounding box size, NPR is used as the second evaluation metric.

[0102] Success Rate (SR): The success rate is the ratio of successful frames whose overlap between the predicted bounding box and the ground truth is greater than a certain threshold. This scheme uses the area under the curve to calculate the SR score.

[0103] Step S302: Train the specified neural network model using a specific data set until the specified neural network model converges to obtain a target tracking model.

[0104] In the specific implementation of step S302, for multiple frames of continuous sample visible light modality images in a specific data set, a first sample template image (called a target area) is extracted from the first frame of the sample visible light modality image; for multiple frames of continuous sample ultraviolet modality images in a specific data set, a second sample template image (called a target area) is extracted from the first frame of the sample ultraviolet modality image;

[0105] Inputting the first sample template image, the second sample template image, each subsequent frame (each frame after the first frame) of the sample visible light modality image, and the sample ultraviolet modality image into the specified neural network model for training, thereby training to obtain a target tracking model;

[0106] Among them, each frame of sample visible light modality image and sample ultraviolet modality image after the first frame input into the specified neural network model is called a search area.

[0107] To better explain the working principle of the specified neural network model, the following Figure 5 The example diagram of the architecture of the specified neural network model is shown for illustration.

[0108] like Figure 5As shown in the figure, the specified neural network model at least includes a specified deep learning network (ViT), an adaptive feature selection mechanism, a connection layer and a prediction head. ViT (Vision Transformer) is the transformer area as the backbone network. The transformer area includes standardization, multi-head attention layer, feature selection, and multi-layer perceptron head.

[0109] The search area and target area are input into the specified neural network model, where: Figure 5 The target area corresponding to "visible light" is the first sample template image extracted, and the search area corresponding to "visible light" is the sample visible light modality image of the subsequent frame (the image frame after the first sample template image); Figure 5 The target area corresponding to “ultraviolet” in is the extracted second sample template image, and the search area corresponding to “ultraviolet” is the sample ultraviolet modal image of the subsequent frame (the image frame after the second sample template image).

[0110] Specify the embedding layer of the neural network model ( Figure 5 The segmentation and embedding part in the image processing method is as follows: the input RGB search area (sample visible light modal image), UV search area (sample ultraviolet modal image), first sample template image, and second sample template image are segmented and flattened into sequence patches (equivalent to feature sequence blocks), and the sequence patches are marked (that is, numbered) so as to restore the feature positions after subsequent feature extraction; these sequence patches are then input into a series of transformer areas for feature extraction and "search-template matching" under each modality, wherein the "search-template matching" under each modality specifically refers to: matching the sample visible light modal image with the first sample template image to predict the predicted position of the sample target in the sample visible light modal image, and matching the sample ultraviolet modal image with the second sample template image to predict the predicted position of the sample target in the sample ultraviolet modal image.

[0111] The adaptive feature selection mechanism of the specified neural network model is inserted between the transformer regions. The adaptive feature selection mechanism is used to select modal information that is more suitable for the current tracking scenario and discard more inefficient modal information.

[0112] Specify the final prediction head (or tracking head) of the neural network model to obtain the features of the connected RGB search area and UV search area from the backbone network to obtain the predicted position of the sample target;

[0113] in, Figure 5In the two images output by the prediction head, the upper image is a sample visible light modal image with the predicted position of the sample target marked, and the lower image is a sample ultraviolet modal image with the predicted position of the sample target (marked with a red frame) marked; according to the confidence of the aforementioned two sample visible light modal images and sample ultraviolet modal images with the predicted position of the sample target marked (equivalent to the confidence of the two predicted positions), the neural network model is specified to output any one predicted position (usually the predicted position with the highest confidence), that is, the prediction result finally output by the neural network model is specified as: the sample visible light modal image with the predicted position of the sample target marked, or the sample ultraviolet modal image with the predicted position of the sample target marked.

[0114] The architecture of the above-specified neural network model is the overall architecture of the VUTrack algorithm. In general, the target area and the search area are segmented, flattened, and linearly projected through a double embedding layer, and then input into the transformer area for "feature extraction" and "search-template matching"; the adaptive feature selection mechanism is responsible for retaining the features of the modalities with higher reliability and reducing the interference of inefficient modalities; finally, the features of the RGB search area and the UV search area are connected and input into the prediction head to predict the predicted position of the sample target.

[0115] Regarding the adaptive feature selection mechanism mentioned above, this adaptive feature selection mechanism can help ViT select more features of the search area of ​​the efficient mode that is suitable for the current tracking scene in the early stage, and remove more background features of the search area of ​​the inefficient mode, thereby reducing the computational burden and reducing the negative impact of background area noise caused by the inefficient mode on feature learning;

[0116] An adaptive feature selection mechanism is inserted between the transformer regions of the ViT backbone network, and the feature selection modules of the two modalities are controlled by calculating the reliability parameters of the modalities.

[0117] The architecture of the adaptive feature selection mechanism is as follows Figure 6 As shown in Figure 1, the adaptive feature selection mechanism consists of feature retention rate, reliable evaluation, feature selection, softmax, etc. Figure 6 R rgb is the reliability score of the visible light modality, R uv is the reliability score of the UV mode, λ rgb is the adaptive weight of the visible light modality, λ uv is the adaptive weight of the UV mode.

[0118] It should be noted that for R rgb and R uv After processing, we can get λrgb and λ uv , specifically, λ rgb and λ uv The specific calculation process of λ is: rgb ,λ uv =softmax(R rgb ,R uv ).

[0119] The above embodiments are detailed descriptions of this solution. Figures 1 to 6 As can be seen from the content, this scheme constructs a specific dataset (VOUT) for visible light and ultraviolet target tracking. This specific dataset has images collected under various "target foreground-background confusion" conditions, and takes into account data attributes such as target occlusion and target loss;

[0120] Taking advantage of the characteristics of ultraviolet imaging, ultraviolet light is used as a supplement to visible light for target tracking, which solves the tracking problem of confusion between the target and the background color caused by color camouflage. In addition, this scheme proposes the VUTrack algorithm, which introduces an adaptive feature selection mechanism based on modal reliability to ensure efficient inference speed and improve tracking performance. According to the research results, the ultraviolet modality can provide information supplement to visible light in complex field environments, has a wide range of application scenarios, and can play a key role in specific fields.

[0121] In order to better reflect the beneficial effects of this solution, the following contents are used for explanation:

[0122] This solution first uses a specific data set to train the specified neural network model mentioned above to obtain a target tracking model, and then evaluates the precision rate (PR), normalized precision rate (NPR) and success rate (SR) of the target tracking model;

[0123] Select 4 existing target trackers (referred to as target tracker 1 to target tracker 4), train and test target tracker 1 to target tracker 4 respectively, and evaluate the PR, NPR and SR of target tracker 1 to target tracker 4;

[0124] From the experimental results, after the introduction of the ultraviolet modality in this scheme, compared with the performance of target trackers 1 to 4, the SR of the target tracking model trained by this scheme is improved by 4.85%, the PR is improved by 7.99%, and the NPR is improved by 5.75%. Therefore, the introduction of the ultraviolet modality can effectively improve the performance of target tracking.

[0125] During the training process, the same model of GPU is used to train the target tracking model, target tracker 1, target tracker 2, target tracker 3 and target tracker 4; from the experimental results, compared with the performance and speed of target trackers 1 to 4, the performance and inference speed of the target tracking model trained by this scheme are significantly improved.

[0126] It should be noted that an adaptive feature selection mechanism is set in the target tracking model trained by this scheme, and the adaptive feature selection mechanism set can effectively improve the SR, PR and NPR of the target tracking model.

[0127] Corresponding to the target tracking method based on visible light and ultraviolet light provided in the above embodiment of the present invention, see Figure 7 , the embodiment of the present invention also provides a structural block diagram of a target tracking device based on visible light and ultraviolet light, the target tracking device includes: a collection unit 701, an extraction unit 702 and a tracking unit 703;

[0128] The acquisition unit 701 is used to acquire visible light modality images and ultraviolet modality images in real time.

[0129] In a specific implementation, the acquisition unit 701 is specifically used to: call the visible light lens to collect visible light modal images in real time, and call the ultraviolet lens to collect ultraviolet modal images in real time.

[0130] The extraction unit 702 is used to extract a first template image covering the target to be tracked from the m-th frame of the visible light modality image, and to extract a second template image covering the target to be tracked from the m-th frame of the ultraviolet modality image.

[0131] In a specific implementation, the extraction unit 702 is specifically used to: in response to a labeling operation triggered by a user, extract an image area covering the target to be tracked from the mth frame of the visible light modal image to obtain a first template image, and extract an image area covering the target to be tracked from the mth frame of the ultraviolet modal image to obtain a second template image.

[0132] The tracking unit 703 is used to input the first template image, the second template image, the n-th visible light modality image and the n-th ultraviolet modality image into a preset target tracking model for target tracking to obtain an n-th prediction result image, wherein n is greater than m, and the prediction result image is annotated with a predicted position of a target to be tracked. The target tracking model is obtained by training a specified neural network model based on a specific data set, and the specified neural network model at least includes a specified deep learning network and an adaptive feature selection mechanism.

[0133] In some embodiments, the nth frame prediction result image is: the nth frame visible light modality image marked with the predicted position of the target to be tracked, or the nth frame ultraviolet modality image marked with the predicted position of the target to be tracked.

[0134] In an embodiment of the present invention, a target tracking model is used to process visible light modal images and ultraviolet light modal images to track a target. Even if the target is confused with the background color, visible light and ultraviolet light can be used to track the target, thereby improving the effect of target tracking and reducing the limitations of target tracking.

[0135] Preferably, combined Figure 7 As shown in the figure, the tracking unit 703 includes an acquisition subunit and a training subunit, and the execution principle of each subunit is as follows:

[0136] The acquisition subunit is used to acquire a specific data set, wherein the specific data set includes multiple image sequences carrying real annotations, any image sequence in the specific data set is set with a challenge attribute, and any image sequence includes multiple frames of continuous sample visible light modal images or sample ultraviolet modal images.

[0137] The training subunit is used to train the specified neural network model using a specific data set until the specified neural network model converges to obtain a target tracking model.

[0138] Preferably, combined Figure 7 The target tracking device further comprises:

[0139] The display unit is used to output and display the prediction result image of the nth frame.

[0140] In summary, the embodiments of the present invention provide a target tracking method and device based on visible light and ultraviolet light, which utilizes a target tracking model to process visible light modal images and ultraviolet light modal images to track a target to be tracked. Even if the target to be tracked is confused with the background color, visible light and ultraviolet light can be used to track the target to be tracked, thereby improving the effect of target tracking and reducing the limitations of target tracking.

[0141] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0142] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0143] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A target tracking method based on visible light and ultraviolet light, characterized in that: The method comprises: Real-time acquisition of visible light modality images and ultraviolet modality images; Extracting a first template image covering the target to be tracked from the m-th frame of the visible light modality image, and extracting a second template image covering the target to be tracked from the m-th frame of the ultraviolet modality image; The first template image, the second template image, the n-th frame of the visible light modal image and the n-th frame of the ultraviolet modal image are input into a preset target tracking model for target tracking to obtain an n-th frame of prediction result image, wherein n is greater than m, and the prediction result image is annotated with the predicted position of the target to be tracked. The target tracking model is obtained by training a specified neural network model based on a specific data set, and the specified neural network model at least includes a specified deep learning network and an adaptive feature selection mechanism.

2. The method according to claim 1, characterized in that Real-time acquisition of visible light modality images and ultraviolet modality images, including: The visible light lens is called to collect visible light modal images in real time, and the ultraviolet light lens is called to collect ultraviolet modal images in real time.

3. The method according to claim 1, characterized in that Extracting a first template image covering the target to be tracked from the m-th frame of the visible light modality image, and extracting a second template image covering the target to be tracked from the m-th frame of the ultraviolet modality image, including: In response to a labeling operation triggered by a user, an image area covering the target to be tracked is extracted from the m-th frame of the visible light modality image to obtain a first template image, and an image area covering the target to be tracked is extracted from the m-th frame of the ultraviolet modality image to obtain a second template image.

4. The method according to claim 1, characterized in that The process of training a specified neural network model based on a specific data set to obtain a target tracking model includes: Acquire a specific data set, wherein the specific data set includes a plurality of image sequences carrying real annotations, any of the image sequences in the specific data set is set with a challenge attribute, and any of the image sequences includes a plurality of frames of continuous sample visible light modality images or sample ultraviolet modality images; The specified neural network model is trained using the specific data set until the specified neural network model converges to obtain a target tracking model.

5. The method according to any one of claims 1 to 4, characterized in that: The prediction result image of the nth frame is: the visible light modality image of the nth frame marked with the predicted position of the target to be tracked, or the ultraviolet modality image of the nth frame marked with the predicted position of the target to be tracked.

6. The method according to any one of claims 1 to 4, characterized in that: After obtaining the n-th frame prediction result image, it also includes: The output displays the predicted result image of the nth frame.

7. A target tracking device based on visible light and ultraviolet light, characterized in that: The device comprises: An acquisition unit, used for acquiring visible light modality images and ultraviolet light modality images in real time; an extraction unit, configured to extract a first template image covering the target to be tracked from the m-th frame of the visible light modality image, and to extract a second template image covering the target to be tracked from the m-th frame of the ultraviolet modality image; The tracking unit is used to input the first template image, the second template image, the n-th frame of the visible light modality image and the n-th frame of the ultraviolet modality image into a preset target tracking model for target tracking to obtain an n-th frame of prediction result image, wherein n is greater than m, and the prediction result image is annotated with the predicted position of the target to be tracked. The target tracking model is obtained by training a specified neural network model based on a specific data set, and the specified neural network model at least includes a specified deep learning network and an adaptive feature selection mechanism.

8. The device according to claim 7, characterized in that The acquisition unit is specifically used to: call the visible light lens to collect visible light modal images in real time, and call the ultraviolet lens to collect ultraviolet modal images in real time.

9. The device according to claim 7, characterized in that The extraction unit is specifically used to: in response to a labeling operation triggered by a user, extract an image area covering the target to be tracked from the m-th frame of the visible light modal image to obtain a first template image, and extract an image area covering the target to be tracked from the m-th frame of the ultraviolet modal image to obtain a second template image.

10. The device according to claim 7, characterized in that The tracking unit comprises: An acquisition subunit is used to acquire a specific data set, wherein the specific data set includes a plurality of image sequences carrying real annotations, any of the image sequences in the specific data set is set with a challenge attribute, and any of the image sequences includes a plurality of frames of continuous sample visible light modality images or sample ultraviolet modality images; The training subunit is used to train the specified neural network model using the specific data set until the specified neural network model converges to obtain a target tracking model.

Images