A target tracking method and device based on visible light, infrared rays and ultraviolet rays

Abstract

The application provides a target tracking method and device based on visible light, infrared rays and ultraviolet rays, which comprises the following steps: collecting visible light modal images, infrared ray modal images and ultraviolet ray modal images in real time; extracting a first template image, a second template image and a third template image covering a target to be tracked from the mth visible light modal image, the mth infrared ray modal image and the mth ultraviolet ray modal image respectively; inputting the first template image, the second template image, the third template image, the nth visible light modal image, the nth infrared ray modal image and the nth ultraviolet ray modal image into a target tracking model to track the target, so as to obtain an nth prediction result image. Even if the target to be tracked is confused with the background color and there is infrared inefficiency, the target to be tracked can be tracked by using visible light, infrared rays and ultraviolet rays, so that the effect of target tracking is improved and the limitation of target tracking is reduced.

Description

Technical Field

[0001] This invention relates to the field of target tracking technology, and specifically to a target tracking method and apparatus based on visible light, infrared light and ultraviolet light. Background Technology

[0002] With the rapid development of visual target tracking, more and more target tracking algorithms have emerged. Currently, target tracking algorithms mainly use a combination of visible light and infrared light to track targets.

[0003] However, during target tracking, if the target is confused with the background color and infrared light is inefficient, the existing target tracking algorithm cannot distinguish the target based on visible light and infrared light, thus making it impossible to continue target tracking. The target tracking effect is poor and has great limitations. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a target tracking method and apparatus based on visible light, infrared light and ultraviolet light, to solve the problems of poor target tracking effect and large limitations of current target tracking algorithms.

[0005] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0006] The first aspect of this invention discloses a target tracking method based on visible light, infrared light, and ultraviolet light, the method comprising:

[0007] Real-time acquisition of visible light modal images, infrared modal images, and ultraviolet modal images;

[0008] From the visible light modal image, the infrared modal image, and the ultraviolet modal image of the m-th frame, respectively, extract the first template image, the second template image, and the third template image covering the target to be tracked;

[0009] The first template image, the second template image, the third template image, the visible light modal image of the nth frame, the infrared modal image of the nth frame, and the ultraviolet modal image of the nth frame are input into a preset target tracking model for target tracking to obtain the prediction result image of the nth frame, where n is greater than m. The prediction result image is marked with the predicted position of the target to be tracked. The target tracking model is obtained by training a specified neural network model based on a sample dataset. The specified neural network model includes at least a specified deep learning network and an adaptive candidate elimination mechanism.

[0010] Preferably, real-time acquisition of visible light modal images, infrared modal images, and ultraviolet modal images includes:

[0011] The system can use a visible light lens to acquire visible light modal images in real time, an infrared lens to acquire infrared modal images in real time, and an ultraviolet lens to acquire ultraviolet modal images in real time.

[0012] Preferably, from the visible light modal image of the m-th frame, the infrared modal image of the m-th frame, and the ultraviolet modal image of the m-th frame, a first template image, a second template image, and a third template image covering the target to be tracked are extracted, respectively, including:

[0013] In response to a user-triggered annotation operation, an image region covering the target to be tracked is extracted from the visible light modal image of the m-th frame to obtain a first template image, an image region covering the target to be tracked is extracted from the infrared modal image of the m-th frame to obtain a second template image, and an image region covering the target to be tracked is extracted from the ultraviolet modal image of the m-th frame to obtain a third template image.

[0014] Preferably, the process of training a specified neural network model based on a sample dataset to obtain a target tracking model includes:

[0015] Obtain a sample dataset, which contains multiple image sequences carrying real annotations. Any image sequence contains: multiple consecutive frames of sample visible light modal images, or multiple consecutive frames of sample infrared modal images, or multiple consecutive frames of sample ultraviolet modal images.

[0016] The specified neural network model is trained using the sample dataset until it converges to obtain a target tracking model.

[0017] Preferably, the prediction result image of the nth frame is: the visible light mode image of the nth frame marked with the predicted position of the target to be tracked, or the infrared mode image of the nth frame marked with the predicted position of the target to be tracked, or the ultraviolet mode image of the nth frame marked with the predicted position of the target to be tracked.

[0018] Preferably, after obtaining the prediction result image of the nth frame, the process further includes:

[0019] The output displays the predicted result image in the nth frame.

[0020] A second aspect of this invention discloses a target tracking device based on visible light, infrared light, and ultraviolet light, the device comprising:

[0021] The acquisition unit is used to acquire visible light modal images, infrared modal images, and ultraviolet modal images in real time;

[0022] The extraction unit is used to extract a first template image, a second template image, and a third template image covering the target to be tracked from the visible light modal image, the infrared modal image, and the ultraviolet modal image of the m-th frame, respectively.

[0023] The tracking unit is used to input the first template image, the second template image, the third template image, the visible light modal image of the nth frame, the infrared modal image of the nth frame, and the ultraviolet modal image of the nth frame into a preset target tracking model to perform target tracking, so as to obtain the prediction result image of the nth frame, wherein n is greater than m, and the prediction result image is marked with the predicted position of the target to be tracked. The target tracking model is obtained by training a specified neural network model based on a sample dataset. The specified neural network model includes at least a specified deep learning network and an adaptive candidate elimination mechanism.

[0024] Preferably, the acquisition unit is specifically used to: call the visible light lens to acquire visible light modal images in real time, call the infrared lens to acquire infrared modal images in real time, and call the ultraviolet lens to acquire ultraviolet modal images in real time.

[0025] Preferably, the extraction unit is specifically configured to: in response to a user-triggered annotation operation, extract an image region covering the target to be tracked from the visible light modal image of the m-th frame to obtain a first template image, extract an image region covering the target to be tracked from the infrared modal image of the m-th frame to obtain a second template image, and extract an image region covering the target to be tracked from the ultraviolet modal image of the m-th frame to obtain a third template image.

[0026] Preferably, the tracking unit includes:

[0027] The acquisition subunit is used to acquire a sample dataset, which contains multiple image sequences carrying real annotations. Any image sequence includes: multiple consecutive frames of sample visible light modal images, or multiple consecutive frames of sample infrared modal images, or multiple consecutive frames of sample ultraviolet modal images.

[0028] The training subunit is used to train a specified neural network model using the sample dataset until the specified neural network model converges, so as to obtain a target tracking model.

[0029] Based on the above embodiments of the present invention, a target tracking method and apparatus based on visible light, infrared light, and ultraviolet light are provided. The method comprises: real-time acquisition of visible light modal images, infrared modal images, and ultraviolet light modal images; extraction of a first template image, a second template image, and a third template image covering the target to be tracked from the m-th frame of the visible light modal image, the m-th frame of the infrared modal image, and the m-th frame of the ultraviolet light modal image, respectively; inputting the first template image, the second template image, the third template image, the n-th frame of the visible light modal image, the n-th frame of the infrared modal image, and the n-th frame of the ultraviolet light modal image into a target tracking model for target tracking to obtain the n-th frame prediction result image. In this scheme, the target tracking model processes the visible light modal image, the infrared modal image, and the ultraviolet light modal image to track the target. Even if the target to be tracked is confused with the background color and infrared light is inefficient, visible light, infrared light, and ultraviolet light can still be used to track the target, thereby improving the target tracking effect and reducing the limitations of target tracking. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0031] Figure 1 Comparative example diagrams of visible light imaging, infrared imaging, and ultraviolet imaging provided for embodiments of the present invention;

[0032] Figure 2 A flowchart of a target tracking method based on visible light, infrared light and ultraviolet light provided for an embodiment of the present invention;

[0033] Figure 3 A flowchart illustrating the training process for obtaining a target tracking model, as provided in this embodiment of the invention;

[0034] Figure 4 An example diagram of the architecture of a specified neural network model provided in an embodiment of the present invention;

[0035] Figure 5 An example diagram of the architecture of the adaptive candidate elimination mechanism provided in the embodiments of the present invention;

[0036] Figure 6 Examples of target tracking effects for different modal combinations provided in embodiments of the present invention;

[0037] Figure 7This is a structural block diagram of a target tracking device based on visible light, infrared light, and ultraviolet light, provided for an embodiment of the present invention. Detailed Implementation

[0038] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0039] In this application, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0040] With the rapid development of visual target tracking, more and more target tracking algorithms have emerged. Currently, target tracking algorithms mainly use a combination of visible light and infrared light to track targets.

[0041] However, during target tracking, if the target is confused with the background color and infrared light is inefficient, then existing target tracking algorithms cannot distinguish the target based on visible light and infrared light, thus making it impossible to continue target tracking. For example, in special scenarios where the target to be tracked is camouflaged with color and infrared light is inefficient, the information provided by visible light and infrared light is limited, and existing target tracking algorithms cannot distinguish the target based on visible light and infrared light.

[0042] For example, for camouflaged targets that are resistant to thermal infrared radiation, since the target's color is mixed with the environment and it has the function of resisting thermal infrared radiation, the information provided by visible light and infrared radiation is limited. Therefore, existing target tracking algorithms cannot distinguish targets based on visible light and infrared radiation. As a result, existing target tracking algorithms have poor performance and great limitations in target tracking.

[0043] The inventors discovered through research that ultraviolet sensors can sensitively image some camouflage materials (or in other words, ultraviolet imaging is quite sensitive to some camouflage materials), such as... Figure 1 The accompanying diagram shows a comparison of visible light imaging, infrared imaging, and ultraviolet imaging. Figure 1Each column of "image group" contains visible light imaging, infrared imaging, and ultraviolet imaging for the same scene. Visible light imaging is an image taken in the visible light mode, infrared imaging is an image taken in the thermal infrared mode, and ultraviolet imaging is an image taken in the ultraviolet mode.

[0044] It should be noted that all objects reflect ultraviolet light to varying degrees. Ultraviolet sensors use the difference in ultraviolet light reflected by the camouflage and the natural environment to achieve ultraviolet-sensitive imaging of the camouflage.

[0045] Combination Figure 1 It is evident that the target to be tracked ( Figure 1 When the object (highlighted in red) is camouflaged with color and has the function of resisting thermal infrared, the effect of ultraviolet imaging is significantly better than that of visible light imaging and infrared imaging.

[0046] To improve target tracking performance, ultraviolet light can be used as supplementary information to visible light and infrared light for target tracking. This results in more robust target tracking in most scenarios, especially in scenarios where the target being tracked is camouflaged and infrared light is inefficient, achieving excellent target tracking results.

[0047] Therefore, this solution proposes a target tracking method and device based on visible light, infrared light, and ultraviolet light. It uses a target tracking model to process visible light modal images, infrared modal images, and ultraviolet light modal images to track the target. Even if the target is confused with the background color and infrared light is inefficient, visible light, infrared light, and ultraviolet light can still be used to track the target, thereby improving the target tracking effect and reducing the limitations of target tracking.

[0048] It should be noted that, in order to better understand the content of this plan, some of the terms involved in this plan will be explained first.

[0049] ultraviolet (UV): ultraviolet light;

[0050] visible (RGB): Visible light;

[0051] thermal infrared (TIR): thermal infrared;

[0052] RGB-T-UV object tracking: tracking of visible light, thermal infrared, and ultraviolet targets;

[0053] Visible-thermal (RGB-T) object tracking: Tracking of visible light and thermal infrared targets;

[0054] Transformer: A deep learning network architecture;

[0055] ViT (Vision Transformer): The specified deep learning network in this scheme. ViT is specifically a deep learning network for image feature extraction.

[0056] VTUTrack algorithm (visible light, thermal infrared, and ultraviolet multi-modal object tracking): a target tracking algorithm based on visible light, infrared, and ultraviolet light.

[0057] See Figure 2 The diagram illustrates a flowchart of a target tracking method based on visible light, infrared light, and ultraviolet light, provided by an embodiment of the present invention. The target tracking method includes:

[0058] Step S201: Real-time acquisition of visible light modal images, infrared modal images, and ultraviolet modal images.

[0059] In the specific implementation step S201, the visible light lens is called to acquire visible light modal images (such as RGB images) in real time, the infrared lens is called to acquire infrared modal images (denoted as T images) in real time, and the ultraviolet lens is called to acquire ultraviolet modal images (denoted as UV images) in real time.

[0060] Specifically, during target tracking, a visible light lens is used to acquire multiple consecutive frames of visible light modal images in real time, an infrared lens is used to acquire multiple consecutive frames of infrared modal images in real time, and an ultraviolet lens is used to acquire multiple consecutive frames of ultraviolet modal images in real time.

[0061] Step S202: Extract the first template image, the second template image, and the third template image covering the target to be tracked from the m-th visible light modal image, the m-th infrared modal image, and the m-th ultraviolet modal image, respectively.

[0062] In the specific implementation step S202, during the target tracking process of acquiring visible light modal images, infrared modal images, and ultraviolet modal images, in response to the annotation operation triggered by the user, an image region covering the target to be tracked is extracted from the m-th frame of the visible light modal image to obtain a first template image, an image region covering the target to be tracked is extracted from the m-th frame of the infrared modal image to obtain a second template image, and an image region covering the target to be tracked is extracted from the m-th frame of the ultraviolet modal image to obtain a third template image, wherein the m-th frame is equivalent to the initial frame.

[0063] Specifically, during target tracking, the user marks the image regions covering the target to be tracked in the m-th visible light modal image, the m-th infrared modal image, and the m-th ultraviolet modal image, respectively.

[0064] In response to the annotation operation triggered by the user, the background area in the m-th visible light modal image except for the "image area covering the target to be tracked" is removed to obtain the first template image, the background area in the m-th infrared modal image except for the "image area covering the target to be tracked" is removed to obtain the second template image, and the background area in the m-th ultraviolet modal image except for the "image area covering the target to be tracked" is removed to obtain the third template image.

[0065] For example: in response to a user-triggered annotation operation, an image region covering the target to be tracked is extracted from the first frame (m=1) visible light modal image to obtain a first template image, an image region covering the target to be tracked is extracted from the first frame infrared modal image to obtain a second template image, and an image region covering the target to be tracked is extracted from the first frame ultraviolet modal image to obtain a third template image.

[0066] Step S203: Input the first template image, the second template image, the third template image, the nth frame visible light mode image, the nth frame infrared mode image, and the nth frame ultraviolet mode image into the preset target tracking model to perform target tracking and obtain the nth frame prediction result image.

[0067] It should be noted that n is greater than m, and the target tracking model is obtained by training a specified neural network model based on the sample dataset. This specified neural network model includes at least a specified deep learning network and an adaptive candidate elimination mechanism.

[0068] In the specific implementation step S203, after extracting the first template image, the second template image, and the third template image, the first template image, the second template image, the third template image, the nth frame visible light mode image, the nth frame infrared mode image, and the nth frame ultraviolet mode image are input into the target tracking model for target tracking, and the nth frame prediction result image output by the target tracking model is obtained. The prediction result image is marked with the predicted position of the target to be tracked.

[0069] Specifically, the nth frame prediction result image is: the nth frame visible light mode image with the predicted position of the target to be tracked marked, or the nth frame infrared mode image with the predicted position of the target to be tracked marked, or the nth frame ultraviolet mode image with the predicted position of the target to be tracked marked.

[0070] In some embodiments, after inputting the first template image, the second template image, the third template image, the nth frame visible light modal image, the nth frame infrared modal image, and the nth frame ultraviolet modal image into the target tracking model, the target tracking model performs feature matching based on the first template image in the search area shown in the nth frame visible light modal image (equivalent to the entire nth frame visible light modal image), thereby predicting and marking the position of the target to be tracked in the nth frame visible light modal image (called the predicted position).

[0071] Meanwhile, the target tracking model performs feature matching based on the second template image in the search area shown in the nth frame infrared modal image, thereby predicting and marking the position of the target to be tracked in the nth frame infrared modal image (called the predicted position);

[0072] The target tracking model performs feature matching based on the third template image in the search area shown in the nth frame of the ultraviolet modal image, thereby predicting and marking the position of the target to be tracked in the nth frame of the ultraviolet modal image (called the predicted position).

[0073] After predicting the location of the target to be tracked in the visible light modal image, infrared modal image, and ultraviolet modal image, the target tracking model outputs the following as the prediction result image for the nth frame based on the confidence scores of the predicted location of the target to be tracked in the nth frame visible light modal image, the nth frame infrared modal image, and the nth frame ultraviolet modal image: the nth frame visible light modal image with the predicted location of the target to be tracked marked, or the nth frame infrared modal image with the predicted location of the target to be tracked marked, or the nth frame ultraviolet modal image with the predicted location of the target to be tracked marked.

[0074] In other words, the predicted position of the target to be tracked in the visible light modal image of the nth frame, the predicted position of the target to be tracked in the infrared modal image of the nth frame, and the predicted position of the target to be tracked in the ultraviolet modal image of the nth frame are equivalent to three prediction results. The target tracking model outputs any one of the prediction results (usually the prediction result with the highest confidence) as the prediction result image of the nth frame based on the confidence of these three prediction results.

[0075] In practical applications, one application scenario for this solution is:

[0076] The image region covering the target to be tracked is extracted from the first frame (m=1) visible light modal image to obtain the first template image, the image region covering the target to be tracked is extracted from the first frame infrared modal image to obtain the second template image, and the image region covering the target to be tracked is extracted from the first frame ultraviolet modal image to obtain the third template image;

[0077] The first template image, the second template image, the third template image, and each subsequent frame (the nth frame, where n is greater than 1) of the visible light modal image, infrared modal image, and ultraviolet modal image are input into the target tracking model to perform target tracking, and multi-frame prediction result images are obtained.

[0078] By inputting a frame of visible light modal image, infrared modal image, and ultraviolet modal image into the target tracking model, a frame of prediction result image can be obtained.

[0079] That is, after extracting the first template image, the second template image, and the third template image from the first visible light modal image, the first infrared modal image, and the first ultraviolet modal image, respectively, a prediction result image can be predicted for each subsequent frame (the nth frame, where n is greater than 1) of visible light modal image, infrared modal image, and ultraviolet modal image (equivalent to predicting the predicted position of the target to be tracked for each subsequent frame).

[0080] In some embodiments, after obtaining the nth frame prediction result image, the nth frame prediction result image is output and displayed, and the nth frame prediction result image is marked with the predicted position of the target to be tracked;

[0081] It should be noted that the predicted position of the target to be tracked can be marked with a rectangle in the nth frame prediction image.

[0082] In this embodiment of the invention, a target tracking model is used to process visible light modal images, infrared modal images, and ultraviolet modal images to track the target. Even if the target is confused with the background color and infrared is inefficient, visible light, infrared, and ultraviolet light can still be used to track the target, thereby improving the target tracking effect and reducing the limitations of target tracking.

[0083] Regarding the above embodiments of the present invention Figure 2 The target tracking model mentioned in step S203, see [link / reference]. Figure 3 The flowchart for training the target tracking model is shown, including the following steps:

[0084] Step S301: Obtain the sample dataset.

[0085] In the specific implementation step S301, a sample dataset is obtained. The sample dataset contains multiple image sequences carrying real annotations. Any image sequence in the sample dataset includes: multiple consecutive sample visible light modal images, or multiple consecutive sample infrared modal images, or multiple consecutive sample ultraviolet modal images.

[0086] The true annotation carried by the image sequence is the bounding box of the true location of the sample target (the target that needs to be tracked or is being tracked).

[0087] It should be noted that during the acquisition of the sample dataset, visible light mode images of the samples were acquired using a visible light lens, infrared mode images using an infrared lens, and ultraviolet mode images using an ultraviolet lens. The acquired sample dataset contains multiple image sequences, totaling multiple image groups.

[0088] An image sequence consists of consecutive image frames; an image group refers to: a sample visible light mode image, a sample infrared mode image, and a sample ultraviolet mode image captured in the same scene.

[0089] In the subsequent training of the specified neural network model using the sample dataset, the sample dataset is divided into training set, validation set and test set; and the tracking effect is evaluated using three evaluation metrics: precision (PR), normalized precision (NPR) and success rate (SR).

[0090] Step S302: Train the specified neural network model using the sample dataset until the specified neural network model converges to obtain the target tracking model.

[0091] In the specific implementation step S302, for multiple consecutive visible light modal images in the sample dataset, a first sample template image (referred to as the target region) is extracted from the first frame of the visible light modal image; for multiple consecutive infrared modal images in the sample dataset, a second sample template image (referred to as the target region) is extracted from the first frame of the infrared modal image; and for multiple consecutive ultraviolet modal images in the sample dataset, a third sample template image (referred to as the target region) is extracted from the first frame of the ultraviolet modal image.

[0092] The first sample template image, the second sample template image, the third sample template image, and the visible light modal image, infrared modal image, and ultraviolet modal image of each subsequent frame (each frame after the first frame) are input into a specified neural network model for training, thereby training a target tracking model;

[0093] The visible light modal image, infrared modal image, and ultraviolet modal image of each frame after the first frame input into the specified neural network model are called the search region.

[0094] To better explain how a given neural network model works, the following will be used... Figure 4 The example diagram of the architecture of the specified neural network model is shown for illustration.

[0095] like Figure 4 As shown, a specified neural network model must include at least a specified deep learning network (ViT), an adaptive candidate elimination mechanism, connection layers, and a prediction head. ViT (Vision Transformer) is the transformer region that serves as the backbone network. The transformer region includes normalization, multi-head attention layers, candidate elimination, and multilayer perceptron heads.

[0096] Input the search region and target region into the specified neural network model, where, Figure 4 The target region corresponding to "visible light" is the extracted first sample template image, and the search region corresponding to "visible light" is the sample visible light modal image of subsequent frames (image frames after the first sample template image); Figure 4 The target region corresponding to "thermal infrared" is the extracted second sample template image, and the search region corresponding to "thermal infrared" is the sample infrared modal image of subsequent frames (image frames after the second sample template image); Figure 4 The target region corresponding to "ultraviolet" is the extracted third sample template image, and the search region corresponding to "ultraviolet" is the sample ultraviolet modal image of subsequent frames (image frames after the third sample template image).

[0097] Specify the embedding layer of the neural network model ( Figure 4 The segmentation and embedding part of the process segments and flattens the input RGB search region (sample visible light modal image), T search region (sample infrared modal image), UV search region (sample ultraviolet modal image), first sample template image, second sample template image, and third sample template image into sequence patches (equivalent to feature sequence blocks), and marks (i.e. number) the sequence patches so that the feature positions can be restored after subsequent feature extraction;

[0098] Then, these sequence patches are input into a series of transformer regions for feature extraction and "search-template matching" for each modality. Specifically, "search-template matching" for each modality means: matching the sample visible light modal image with the first sample template image to predict the predicted position of the sample target in the sample visible light modal image; matching the sample infrared image with the second sample template image to predict the predicted position of the sample target in the sample infrared modal image; and matching the sample ultraviolet modal image with the third sample template image to predict the predicted position of the sample target in the sample ultraviolet modal image.

[0099] An adaptive candidate elimination mechanism for the specified neural network model is inserted between transformer regions. This adaptive candidate elimination mechanism is used to select modal information that is more suitable for the current tracking scenario and discard more inefficient modal information.

[0100] The final prediction head (or tracking head) of the specified neural network model obtains the features of the connected RGB search region, T search region and UV search region from the backbone network, thereby obtaining the predicted position of the sample target;

[0101] in, Figure 4 Of the three images output by the prediction head, the top image is the ultraviolet mode image of the sample with the predicted location of the sample target marked, the middle image is the visible light mode image of the sample with the predicted location of the sample target marked, and the bottom image is the infrared mode image of the sample with the predicted location of the sample target marked.

[0102] Based on the confidence levels of the ultraviolet modal image, visible light modal image, and infrared modal image of the three predicted locations of the target (equivalent to the confidence levels of the three predicted locations), the neural network model is specified to output any one of the predicted locations (usually the predicted location with the highest confidence level). That is, the final prediction result output by the neural network model is specified to be: the ultraviolet modal image of the target with the predicted location, or the visible light modal image of the target with the predicted location, or the infrared modal image of the target with the predicted location.

[0103] The architecture of the specified neural network model is the overall architecture of the VTUTrack algorithm. In general, the target region and the search region are segmented, flattened, and linearly projected through a triple embedding layer, and then input into the transformer region for "feature extraction" and "search-template matching". The adaptive candidate elimination mechanism is responsible for retaining more reliable modal features and reducing interference from inefficient modalities. Finally, the features of the RGB search region, T search region, and UV search region are concatenated and input into the prediction head to predict the predicted position of the sample target.

[0104] The adaptive candidate elimination mechanism mentioned above can help ViT select more features of the search region of the efficient modality that is suitable for the current tracking scenario in the early stage, and remove more background features of the search region of the inefficient modality, thereby reducing the computational burden and reducing the negative impact of background noise caused by inefficient modality on feature learning.

[0105] An adaptive candidate elimination mechanism is inserted multiple times between the transformer regions of the ViT backbone network, and the feature selection modules of the three modalities are controlled by calculating the reliability parameters of the modalities.

[0106] The architecture of the adaptive candidate elimination mechanism is as follows: Figure 5 As shown, the adaptive candidate elimination mechanism consists of feature retention rate, reliable evaluation, candidate elimination, softmax, and other components. Figure 5 R in rgb R is the reliability score for the visible light mode. uv R is the reliability score for the ultraviolet mode (also known as the UV mode). t For the reliability score of the infrared mode, λ rgb For the adaptive weights of the visible light modes, λ uv For the adaptive weights of the purple line mode, λ t For the adaptive weights of the infrared modes.

[0107] It should be noted that for R rgb R uv and R t After processing, λ can be obtained. rgb , λ uv and λ t Specifically, λ rgb , λ uv and λ t The specific calculation process is as follows: λ rgb ,λ uv ,λ t = softmax(R rgb ,R uv ,R t ).

[0108] The above embodiments are detailed descriptions of this solution; through the above... Figures 1 to 5 As can be seen from the content, by utilizing the characteristics of ultraviolet imaging, ultraviolet light is used as a supplementary means to visible light and infrared light for target tracking. This solves the problems of target and background color confusion and infrared inefficiency when the target is camouflaged and resists infrared light. It realizes a multimodal tracker that can adapt to complex tracking scenarios.

[0109] Furthermore, this proposal puts forward the VTUTrack algorithm, which introduces an adaptive candidate elimination mechanism based on modal reliability to ensure efficient inference speed and improve tracking performance. The research results show that ultraviolet modes can provide information supplement to visible light and infrared light in complex field environments, have a wide range of application scenarios, and can play a key role in specific fields.

[0110] To better demonstrate the beneficial effects of this plan, the following explanation is provided:

[0111] Comparative experiments were conducted on the constructed sample dataset using seven different modal combinations to analyze the impact of these seven modal combinations on the accuracy (PR), normalized accuracy (NPR), and success rate (SR) of the target tracker.

[0112] The seven mode combinations are: RGB mode (visible light mode), T mode (infrared mode), UV mode (ultraviolet mode), T-UV mode (infrared-ultraviolet mode), RGB-UV mode (visible light-ultraviolet mode), RGB-T mode (visible light-infrared mode), and RGB-T-UV mode (visible light-infrared-ultraviolet mode).

[0113] In addition to the seven modal combinations mentioned above, this solution also provides the ground truth (GT) of the target to be tracked for comparison.

[0114] This scheme is a target tracking method based on RGB-T-UV modes.

[0115] Target tracking was performed using the seven modal combinations described above. The target tracking performance of each modal combination and the true location of the target to be tracked are as follows: Figure 6 As shown, Figure 6 The rectangles of each color (except the color corresponding to GT) represent the predicted positions of the targets to be tracked, obtained by combining different modes. Figure 6 The rectangle in red corresponding to the ground truth (GT) indicates the true location of the target to be tracked.

[0116] from Figure 6 As shown in the example diagrams illustrating the target tracking performance of different modal combinations, the RGB-T-UV modal combination demonstrates superior target tracking performance compared to other modal combinations.

[0117] Compared to modal combinations consisting of two modes (such as RGB-T mode, T-UV mode, etc.), the target tracking performance of the RGB-T-UV mode combination shows the following improvements in accuracy (PR), normalized accuracy (NPR), and success rate (SR): SR improves by 6.32%, PR improves by 5.46%, and NPR improves by 4.46%.

[0118] Compared to modal combinations consisting of single modalities (such as RGB modal, T modal, etc.), the target tracking performance of the RGB-T-UV modal combination shows the following improvements in accuracy (PR), normalized accuracy (NPR), and success rate (SR): SR is improved by 22.15%, PR by 28.63%, and NPR by 24.93%.

[0119] Based on the improvements described above, the target tracking solution based on RGB-T-UV modes provided in this paper can adapt to more complex tracking scenarios.

[0120] Furthermore, since the infrared mode is unaffected by light intensity, when the ultraviolet mode and the visible light mode are in a state of mode failure and mode inefficiency at night, respectively, the infrared mode in the RGB-T-UV mode can compensate for the information of the ultraviolet mode and the visible light mode.

[0121] When using this scheme for target tracking, an adaptive candidate elimination mechanism is introduced to enable the target tracking inference speed to reach 45 FPS, which meets the speed requirements for real-time inference.

[0122] To verify the effectiveness of the adaptive candidate elimination mechanism, a comparative experimental group was set up for "target tracking with adaptive candidate elimination mechanism" and "target tracking without adaptive candidate elimination mechanism". The performance of "target tracking with adaptive candidate elimination mechanism" and "target tracking without adaptive candidate elimination mechanism" was evaluated by inference speed (FPS), product computation (MACs), and tracking performance. The experimental results show that compared with "target tracking without adaptive candidate elimination mechanism", "target tracking with adaptive candidate elimination mechanism" has the following improvements: MACs reduced by 25.9%, tracking performance and inference speed improved by 50.2%, SR increased by 0.91%, PR increased by 1.76%, and NPR increased by 0.74%.

[0123] The aforementioned improvements in "target tracking with adaptive candidate elimination mechanism" demonstrate that adding an adaptive candidate elimination mechanism can significantly improve inference speed while maintaining good tracking performance.

[0124] Corresponding to the target tracking method based on visible light, infrared light, and ultraviolet light provided in the above embodiments of the present invention, see also... Figure 7 The present invention also provides a structural block diagram of a target tracking device based on visible light, infrared light and ultraviolet light, the target tracking device including: a collection unit 701, an extraction unit 702 and a tracking unit 703;

[0125] The acquisition unit 701 is used to acquire visible light modal images, infrared modal images, and ultraviolet modal images in real time.

[0126] In specific implementation, the acquisition unit 701 is used to: call the visible light lens to acquire visible light modal images in real time, call the infrared lens to acquire infrared modal images in real time, and call the ultraviolet lens to acquire ultraviolet modal images in real time.

[0127] Extraction unit 702 is used to extract a first template image, a second template image, and a third template image covering the target to be tracked from the m-th visible light modal image, the m-th infrared modal image, and the m-th ultraviolet modal image, respectively.

[0128] In a specific implementation, the extraction unit 702 is specifically used to: in response to a user-triggered annotation operation, extract an image region covering the target to be tracked from the m-th visible light modal image to obtain a first template image, extract an image region covering the target to be tracked from the m-th infrared modal image to obtain a second template image, and extract an image region covering the target to be tracked from the m-th ultraviolet modal image to obtain a third template image.

[0129] The tracking unit 703 is used to input the first template image, the second template image, the third template image, the nth frame visible light modal image, the nth frame infrared modal image, and the nth frame ultraviolet modal image into a preset target tracking model to perform target tracking, so as to obtain the nth frame prediction result image, where n is greater than m. The prediction result image is marked with the predicted position of the target to be tracked. The target tracking model is obtained by training a specified neural network model based on the sample dataset. The specified neural network model includes at least a specified deep learning network and an adaptive candidate elimination mechanism.

[0130] In some embodiments, the nth frame prediction result image is: the nth frame visible light mode image marked with the predicted position of the target to be tracked, or the nth frame infrared mode image marked with the predicted position of the target to be tracked, or the nth frame ultraviolet mode image marked with the predicted position of the target to be tracked.

[0131] In this embodiment of the invention, a target tracking model is used to process visible light modal images, infrared modal images, and ultraviolet modal images to track the target. Even if the target is confused with the background color and infrared is inefficient, visible light, infrared, and ultraviolet light can still be used to track the target, thereby improving the target tracking effect and reducing the limitations of target tracking.

[0132] Preferred, combined Figure 7 The tracking unit 703, as shown, includes an acquisition subunit and a training subunit. The execution principle of each subunit is as follows:

[0133] The acquisition sub-unit is used to acquire the sample dataset, which contains multiple image sequences carrying real annotations. Any image sequence contains: multiple consecutive sample visible light modal images, or multiple consecutive sample infrared modal images, or multiple consecutive sample ultraviolet modal images.

[0134] The training subunit is used to train a specified neural network model using a sample dataset until the specified neural network model converges, thereby obtaining a target tracking model.

[0135] Preferred, combined Figure 7 The target tracking device, as shown, also includes:

[0136] The display unit is used to output and display the prediction result image of the nth frame.

[0137] In summary, the embodiments of the present invention provide a target tracking method and apparatus based on visible light, infrared light, and ultraviolet light. The method utilizes a target tracking model to process visible light modal images, infrared modal images, and ultraviolet light modal images to track the target. Even if the target is confused with the background color and infrared light is inefficient, visible light, infrared light, and ultraviolet light can still be used to track the target, thereby improving the target tracking effect and reducing the limitations of target tracking.

[0138] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0139] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0140] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A target tracking method based on visible light, infrared light, and ultraviolet light, characterized in that, The method includes: Real-time acquisition of visible light modal images, infrared modal images, and ultraviolet modal images; From the visible light modal image, the infrared modal image, and the ultraviolet modal image of the m-th frame, respectively, extract the first template image, the second template image, and the third template image covering the target to be tracked; The first template image, the second template image, the third template image, the visible light modal image of the nth frame, the infrared modal image of the nth frame, and the ultraviolet modal image of the nth frame are input into a preset target tracking model for target tracking to obtain the prediction result image of the nth frame, where n is greater than m. The prediction result image is marked with the predicted position of the target to be tracked. The target tracking model is obtained by training a specified neural network model based on a sample dataset. The specified neural network model includes at least a specified deep learning network and an adaptive candidate elimination mechanism. The specified deep learning network includes normalization, multi-head attention layer, candidate elimination, and multilayer perceptron head. The adaptive candidate elimination mechanism is configured as follows: For each of the visible light mode, infrared mode, and ultraviolet mode, based on the features extracted from the mode, the reliability score of the mode in the current tracking scenario is calculated to obtain the reliability score of the visible light mode, the reliability score of the ultraviolet mode, and the reliability score of the infrared mode. For each of the visible light mode, infrared mode, and ultraviolet mode, an adaptive weight is generated for the mode based on the reliability score of the mode using the Softmax function, so as to obtain the adaptive weights of the visible light mode, the ultraviolet mode, and the infrared mode. Based on the adaptive weights of each modality, the search region features from different modalities are dynamically selected to enhance the features of modalities with high reliability and suppress or discard the background features in modalities with low reliability. The process by which the target tracking model obtains the predicted image of the nth frame includes: The target tracking model performs feature matching based on the first template image in the search area shown in the nth visible light modal image, thereby predicting and annotating the predicted position of the target to be tracked in the nth visible light modal image; The target tracking model performs feature matching based on the second template image in the search area shown in the nth frame infrared modal image, thereby predicting and annotating the predicted position of the target to be tracked in the nth frame infrared modal image; The target tracking model performs feature matching based on the third template image in the search area shown in the nth frame of the ultraviolet modal image, thereby predicting and annotating the predicted position of the target to be tracked in the nth frame of the ultraviolet modal image; The target tracking model outputs the nth frame of the predicted position of the target in the visible light modal image, the infrared modal image, and the ultraviolet modal image, labeled with the predicted position of the target, as the nth frame prediction result image.

2. The method according to claim 1, characterized in that, Real-time acquisition of visible light modal images, infrared modal images, and ultraviolet modal images, including: The system can use a visible light lens to acquire visible light modal images in real time, an infrared lens to acquire infrared modal images in real time, and an ultraviolet lens to acquire ultraviolet modal images in real time.

3. The method according to claim 1, characterized in that, From the visible light modal image, the infrared modal image, and the ultraviolet modal image of the m-th frame, respectively, extract a first template image, a second template image, and a third template image covering the target to be tracked, including: In response to a user-triggered annotation operation, an image region covering the target to be tracked is extracted from the visible light modal image of the m-th frame to obtain a first template image, an image region covering the target to be tracked is extracted from the infrared modal image of the m-th frame to obtain a second template image, and an image region covering the target to be tracked is extracted from the ultraviolet modal image of the m-th frame to obtain a third template image.

4. The method according to claim 1, characterized in that, The process of training a specified neural network model based on a sample dataset to obtain a target tracking model includes: Obtain a sample dataset, which contains multiple image sequences carrying real annotations. Any image sequence contains: multiple consecutive frames of sample visible light modal images, or multiple consecutive frames of sample infrared modal images, or multiple consecutive frames of sample ultraviolet modal images. The specified neural network model is trained using the sample dataset until it converges to obtain a target tracking model.

5. The method according to any one of claims 1-4, characterized in that, After obtaining the predicted image of the nth frame, the following steps are also included: The output displays the predicted result image in the nth frame.

6. A target tracking device based on visible light, infrared light, and ultraviolet light, characterized in that, The device includes: The acquisition unit is used to acquire visible light modal images, infrared modal images, and ultraviolet modal images in real time; The extraction unit is used to extract a first template image, a second template image, and a third template image covering the target to be tracked from the visible light modal image, the infrared modal image, and the ultraviolet modal image of the m-th frame, respectively. The tracking unit is used to input the first template image, the second template image, the third template image, the nth frame of the visible light modal image, the nth frame of the infrared modal image, and the nth frame of the ultraviolet modal image into a preset target tracking model to perform target tracking, so as to obtain the nth frame prediction result image, where n is greater than m. The prediction result image is marked with the predicted position of the target to be tracked. The target tracking model is obtained by training a specified neural network model based on a sample dataset. The specified neural network model includes at least a specified deep learning network and an adaptive candidate elimination mechanism. The specified deep learning network includes normalization, multi-head attention layer, candidate elimination, and multilayer perceptron head. The adaptive candidate elimination mechanism is configured as follows: For each of the visible light mode, infrared mode, and ultraviolet mode, based on the features extracted from the mode, the reliability score of the mode in the current tracking scenario is calculated to obtain the reliability score of the visible light mode, the reliability score of the ultraviolet mode, and the reliability score of the infrared mode. For each of the visible light mode, infrared mode, and ultraviolet mode, an adaptive weight is generated for the mode based on the reliability score of the mode using the Softmax function, so as to obtain the adaptive weights of the visible light mode, the ultraviolet mode, and the infrared mode. Based on the adaptive weights of each modality, the search region features from different modalities are dynamically selected to enhance the features of modalities with high reliability and suppress or discard the background features in modalities with low reliability. The process by which the target tracking model obtains the predicted image of the nth frame includes: The target tracking model performs feature matching based on the first template image in the search area shown in the nth visible light modal image, thereby predicting and annotating the predicted position of the target to be tracked in the nth visible light modal image; The target tracking model performs feature matching based on the second template image in the search area shown in the nth frame infrared modal image, thereby predicting and annotating the predicted position of the target to be tracked in the nth frame infrared modal image; The target tracking model performs feature matching based on the third template image in the search area shown in the nth frame of the ultraviolet modal image, thereby predicting and annotating the predicted position of the target to be tracked in the nth frame of the ultraviolet modal image; The target tracking model outputs the nth frame of the predicted position of the target in the visible light modal image, the infrared modal image, and the ultraviolet modal image, labeled with the predicted position of the target, as the nth frame prediction result image.

7. The apparatus according to claim 6, characterized in that, The acquisition unit is specifically used to: call the visible light lens to acquire visible light modal images in real time, call the infrared lens to acquire infrared modal images in real time, and call the ultraviolet lens to acquire ultraviolet modal images in real time.

8. The apparatus according to claim 6, characterized in that, The extraction unit is specifically used to: in response to a user-triggered annotation operation, extract an image region covering the target to be tracked from the visible light modal image of the m-th frame to obtain a first template image, extract an image region covering the target to be tracked from the infrared modal image of the m-th frame to obtain a second template image, and extract an image region covering the target to be tracked from the ultraviolet modal image of the m-th frame to obtain a third template image.

9. The apparatus according to claim 6, characterized in that, The tracking unit includes: The acquisition subunit is used to acquire a sample dataset, which contains multiple image sequences carrying real annotations. Any image sequence includes: multiple consecutive frames of sample visible light modal images, or multiple consecutive frames of sample infrared modal images, or multiple consecutive frames of sample ultraviolet modal images. The training subunit is used to train a specified neural network model using the sample dataset until the specified neural network model converges, so as to obtain a target tracking model.

Images