Target tracking method and device based on visible light, infrared ray and ultraviolet ray

Abstract

The invention provides a target tracking method and device based on visible light, infrared light and ultraviolet light. The method comprises the steps that a visible light modal image, an infrared light modal image and an ultraviolet light modal image are collected in real time; respectively extracting a first template image, a second template image and a third template image covering the target to be tracked from the mth frame of visible light modal image, the mth frame of infrared ray modal image and the mth frame of ultraviolet ray modal image; and inputting the first template image, the second template image, the third template image, the nth frame of visible light modal image, the nth frame of infrared ray modal image and the nth frame of ultraviolet ray modal image into a target tracking model for target tracking so as to obtain the nth frame of prediction result image. Even if the to-be-tracked target is mixed with the background color and the infrared efficiency is low, the visible light, the infrared ray and the ultraviolet ray can be used to track the to-be-tracked target, so that the target tracking effect is improved, and the limitation of target tracking is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of target tracking, and in particular to a target tracking method and device based on visible light, infrared light and ultraviolet light. Background Art

[0002] With the rapid development of visual target tracking, more and more target tracking algorithms have emerged. The current target tracking algorithms mainly use a combination of visible light and infrared to achieve target tracking.

[0003] However, in the process of tracking the target, if the target is confused with the color background and there is infrared inefficiency, the existing target tracking algorithm cannot distinguish the target based on visible light and infrared, which makes it impossible to continue target tracking, and the target tracking effect is poor and has great limitations. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a target tracking method and device based on visible light, infrared light and ultraviolet light to solve the problems of poor target tracking effect and large limitations existing in the current target tracking algorithm.

[0005] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0006] A first aspect of an embodiment of the present invention discloses a target tracking method based on visible light, infrared light and ultraviolet light, the method comprising:

[0007] Real-time acquisition of visible light modality images, infrared modality images, and ultraviolet modality images;

[0008] Extracting a first template image, a second template image, and a third template image covering the target to be tracked from the m-th frame of the visible light modality image, the m-th frame of the infrared modality image, and the m-th frame of the ultraviolet modality image, respectively;

[0009] The first template image, the second template image, the third template image, the n-th frame visible light modal image, the n-th frame infrared modal image, and the n-th frame ultraviolet modal image are input into a preset target tracking model for target tracking to obtain an n-th frame prediction result image, wherein n is greater than m, and the prediction result image is annotated with the predicted position of the target to be tracked. The target tracking model is obtained by training a specified neural network model based on a sample data set, and the specified neural network model at least includes a specified deep learning network and an adaptive candidate elimination mechanism.

[0010] Preferably, real-time acquisition of visible light modality images, infrared modality images and ultraviolet modality images includes:

[0011] The visible light lens is called to collect visible light modal images in real time, the infrared lens is called to collect infrared modal images in real time, and the ultraviolet lens is called to collect ultraviolet modal images in real time.

[0012] Preferably, extracting a first template image, a second template image, and a third template image covering the target to be tracked from the m-th frame of the visible light modal image, the m-th frame of the infrared modal image, and the m-th frame of the ultraviolet modal image, respectively, comprises:

[0013] In response to a labeling operation triggered by a user, an image area covering the target to be tracked is extracted from the visible light modal image of the mth frame to obtain a first template image, an image area covering the target to be tracked is extracted from the infrared modal image of the mth frame to obtain a second template image, and an image area covering the target to be tracked is extracted from the ultraviolet modal image of the mth frame to obtain a third template image.

[0014] Preferably, the process of training a specified neural network model based on a sample data set to obtain a target tracking model includes:

[0015] Acquire a sample data set, wherein the sample data set includes a plurality of image sequences carrying real annotations, and any of the image sequences includes: a plurality of frames of continuous sample visible light modality images, or a plurality of frames of continuous sample infrared modality images, or a plurality of frames of continuous sample ultraviolet modality images;

[0016] The specified neural network model is trained using the sample data set until the specified neural network model converges to obtain a target tracking model.

[0017] Preferably, the prediction result image of the nth frame is: the visible light modal image of the nth frame marked with the predicted position of the target to be tracked, or the infrared modal image of the nth frame marked with the predicted position of the target to be tracked, or the ultraviolet modal image of the nth frame marked with the predicted position of the target to be tracked.

[0018] Preferably, after obtaining the n-th frame prediction result image, the method further includes:

[0019] The output displays the predicted result image of the nth frame.

[0020] A second aspect of an embodiment of the present invention discloses a target tracking device based on visible light, infrared light and ultraviolet light, the device comprising:

[0021] An acquisition unit, used for acquiring visible light modality images, infrared modality images and ultraviolet modality images in real time;

[0022] an extraction unit, configured to respectively extract a first template image, a second template image, and a third template image covering the target to be tracked from the m-th frame of the visible light modal image, the m-th frame of the infrared modal image, and the m-th frame of the ultraviolet modal image;

[0023] A tracking unit is used to input the first template image, the second template image, the third template image, the n-th frame visible light modal image, the n-th frame infrared modal image, and the n-th frame ultraviolet modal image into a preset target tracking model for target tracking to obtain an n-th frame prediction result image, wherein n is greater than m, and the prediction result image is annotated with the predicted position of the target to be tracked. The target tracking model is obtained by training a specified neural network model based on a sample data set, and the specified neural network model at least includes a specified deep learning network and an adaptive candidate elimination mechanism.

[0024] Preferably, the acquisition unit is specifically used to: call the visible light lens to collect visible light modal images in real time, call the infrared lens to collect infrared modal images in real time, and call the ultraviolet lens to collect ultraviolet modal images in real time.

[0025] Preferably, the extraction unit is specifically used to: in response to a labeling operation triggered by a user, extract an image area covering the target to be tracked from the visible light modal image of the mth frame to obtain a first template image, extract an image area covering the target to be tracked from the infrared modal image of the mth frame to obtain a second template image, and extract an image area covering the target to be tracked from the ultraviolet modal image of the mth frame to obtain a third template image.

[0026] Preferably, the tracking unit comprises:

[0027] An acquisition subunit is used to acquire a sample data set, wherein the sample data set includes a plurality of image sequences carrying real annotations, and any of the image sequences includes: a plurality of continuous frames of sample visible light modality images, or a plurality of continuous frames of sample infrared modality images, or a plurality of continuous frames of sample ultraviolet modality images;

[0028] The training subunit is used to train the specified neural network model using the sample data set until the specified neural network model converges to obtain a target tracking model.

[0029] Based on the above embodiment of the present invention, a target tracking method and device based on visible light, infrared and ultraviolet light are provided. The method comprises: real-time acquisition of visible light modal images, infrared modal images and ultraviolet modal images; extracting the first template image, the second template image and the third template image covering the target to be tracked from the m-th frame of visible light modal images, the m-th frame of infrared modal images and the m-th frame of ultraviolet modal images respectively; inputting the first template image, the second template image, the third template image, the n-th frame of visible light modal images, the n-th frame of infrared modal images and the n-th frame of ultraviolet modal images into the target tracking model for target tracking to obtain the n-th frame of predicted result image. In this scheme, the target tracking model is used to process the visible light modal images, the infrared modal images and the ultraviolet modal images to track the target to be tracked. Even if the target to be tracked is confused with the background color and there is infrared inefficiency, the visible light, infrared and ultraviolet light can be used to track the target to be tracked, thereby improving the effect of target tracking and reducing the limitations of target tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0031] Figure 1 A comparative example diagram of visible light imaging, infrared imaging and ultraviolet imaging provided in an embodiment of the present invention;

[0032] Figure 2 A flow chart of a target tracking method based on visible light, infrared light and ultraviolet light provided by an embodiment of the present invention;

[0033] Figure 3 A flowchart of a target tracking model obtained by training provided by an embodiment of the present invention;

[0034] Figure 4 An example diagram of the architecture of a specified neural network model provided by an embodiment of the present invention;

[0035] Figure 5 An example diagram of the architecture of the adaptive candidate elimination mechanism provided by an embodiment of the present invention;

[0036] Figure 6 An example diagram of target tracking effects of different modal combinations provided by an embodiment of the present invention;

[0037] Figure 7A structural block diagram of a target tracking device based on visible light, infrared light and ultraviolet light provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] In this application, the terms "comprises", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element.

[0040] With the rapid development of visual target tracking, more and more target tracking algorithms have emerged. The current target tracking algorithms mainly use a combination of visible light and infrared to achieve target tracking.

[0041] However, in the process of tracking a target, if the target is confused with the color background and infrared is inefficient, the existing target tracking algorithm cannot distinguish the target based on visible light and infrared, which makes it impossible to continue tracking the target. For example, in special scenarios where the target to be tracked is color camouflaged and infrared is inefficient, the information provided by visible light and infrared is limited, and the existing target tracking algorithm cannot distinguish the target based on visible light and infrared.

[0042] For another example, for camouflaged targets that counteract thermal infrared, since the target is confused with the environment color and has the function of counteracting thermal infrared, the information provided by visible light and infrared is limited, and the existing target tracking algorithm cannot distinguish the target based on visible light and infrared; therefore, the existing target tracking algorithm has poor target tracking effect and great limitations.

[0043] The inventor has found through research that the UV sensor can be sensitive to some camouflage materials (or the UV imaging is more sensitive to some camouflage materials), such as Figure 1 The comparison example diagram of visible light imaging, infrared imaging and ultraviolet imaging is shown in FIG. Figure 1Each column of "image group" contains visible light imaging, infrared imaging and ultraviolet imaging of the same scene. Visible light imaging is an image taken in visible light mode, infrared imaging is an image taken in thermal infrared mode, and ultraviolet imaging is an image taken in ultraviolet mode.

[0044] It should be noted that any object reflects ultraviolet light to varying degrees, and the ultraviolet sensor uses the difference between the ultraviolet light reflected by the camouflage and the natural environment to achieve ultraviolet-sensitive imaging of the camouflage;

[0045] Combination Figure 1 It can be seen that the target to be tracked ( Figure 1 When the object (the object framed by the red frame) is color camouflaged and has the function of resisting thermal infrared, the effect of ultraviolet imaging is significantly better than that of visible light imaging and infrared imaging;

[0046] In order to improve the effect of target tracking (equivalent to improving target tracking performance), ultraviolet light can be used as supplementary information to visible light and infrared light for target tracking, thereby achieving more robust target tracking effects in most scenarios, especially in scenarios where the target to be tracked is color camouflaged and infrared is inefficient, to achieve very good target tracking effects.

[0047] Therefore, this scheme proposes a target tracking method and device based on visible light, infrared and ultraviolet rays, which utilizes the target tracking model to process visible light modal images, infrared modal images and ultraviolet modal images to track the target to be tracked. Even if the target to be tracked is confused with the background color and there is infrared inefficiency, visible light, infrared and ultraviolet rays can be used to track the target to be tracked, thereby improving the target tracking effect and reducing the limitations of target tracking.

[0048] It should be noted that in order to better understand the subsequent contents of this plan, some of the terms involved in this plan are explained here.

[0049] ultraviolet(UV): ultraviolet;

[0050] visible (RGB): visible light;

[0051] thermal infrared(TIR):thermal infrared;

[0052] RGB-T-UV object tracking: visible light, thermal infrared and ultraviolet target tracking;

[0053] visible-thermal (RGB-T) object tracking: visible-thermal and thermal infrared object tracking;

[0054] Transformer: deep learning network structure;

[0055] ViT (Vision Transformer): The specified deep learning network in this solution. ViT is a deep learning network for image feature extraction.

[0056] VTUTrack algorithm (visible light, thermal infrared, and ultraviolet multi-modal object tracking): target tracking algorithm based on visible light, infrared and ultraviolet.

[0057] See also Figure 2 , shows a flow chart of a target tracking method based on visible light, infrared light and ultraviolet light provided by an embodiment of the present invention, the target tracking method comprising:

[0058] Step S201: real-time acquisition of visible light modality images, infrared modality images, and ultraviolet modality images.

[0059] In the specific implementation of step S201, the visible light lens is called to collect visible light modal images (such as RGB images) in real time, the infrared lens is called to collect infrared modal images (recorded as T images) in real time, and the ultraviolet lens is called to collect ultraviolet modal images (recorded as UV images) in real time.

[0060] Specifically, when tracking a target, a visible light lens is used to collect multiple continuous frames of visible light modal images in real time, an infrared lens is used to collect multiple continuous frames of infrared modal images in real time, and an ultraviolet lens is used to collect multiple continuous frames of ultraviolet modal images in real time.

[0061] Step S202: extracting a first template image, a second template image and a third template image covering the target to be tracked from the m-th visible light modality image frame, the m-th infrared modality image frame and the m-th ultraviolet modality image frame respectively.

[0062] In the process of specifically implementing step S202, in the process of collecting visible light modal images, infrared modal images and ultraviolet modal images for target tracking, in response to a labeling operation triggered by a user, an image area covering the target to be tracked is extracted from the m-th frame of the visible light modal image to obtain a first template image, and an image area covering the target to be tracked is extracted from the m-th frame of the infrared modal image to obtain a second template image, and an image area covering the target to be tracked is extracted from the m-th frame of the ultraviolet modal image to obtain a third template image, wherein the m-th frame is equivalent to the initial frame.

[0063] Specifically, during the target tracking process, the user marks the image area covering the target to be tracked in the mth frame of the visible light modality image, the mth frame of the infrared modality image, and the mth frame of the ultraviolet modality image;

[0064] In response to a labeling operation triggered by a user, the background area except the “image area covering the target to be tracked” in the m-th frame of the visible light modal image is eliminated to obtain a first template image, the background area except the “image area covering the target to be tracked” in the m-th frame of the infrared modal image is eliminated to obtain a second template image, and the background area except the “image area covering the target to be tracked” in the m-th frame of the ultraviolet modal image is eliminated to obtain a third template image.

[0065] For example: in response to a labeling operation triggered by a user, an image area covering the target to be tracked is extracted from the first frame (m=1) of the visible light modal image to obtain a first template image, and an image area covering the target to be tracked is extracted from the first frame of the infrared modal image to obtain a second template image, and an image area covering the target to be tracked is extracted from the first frame of the ultraviolet modal image to obtain a third template image.

[0066] Step S203: input the first template image, the second template image, the third template image, the nth visible light modality image, the nth infrared modality image, and the nth ultraviolet modality image into a preset target tracking model for target tracking to obtain the nth prediction result image.

[0067] It should be noted that n is greater than m, and the target tracking model is obtained by training a specified neural network model based on a sample data set. The specified neural network model at least includes a specified deep learning network and an adaptive candidate elimination mechanism.

[0068] In the specific implementation of step S203, after extracting the first template image, the second template image, and the third template image, the first template image, the second template image, the third template image, the n-th visible light modality image, the n-th infrared modality image, and the n-th ultraviolet modality image are input into the target tracking model for target tracking, and the n-th prediction result image output by the target tracking model is obtained, and the prediction result image is annotated with the predicted position of the target to be tracked;

[0069] Specifically, the nth frame prediction result image is: the nth frame visible light modal image marked with the predicted position of the target to be tracked, or the nth frame infrared modal image marked with the predicted position of the target to be tracked, or the nth frame ultraviolet modal image marked with the predicted position of the target to be tracked.

[0070] In some embodiments, after the first template image, the second template image, the third template image, the n-th visible light modality image, the n-th infrared modality image, and the n-th ultraviolet modality image are input into the target tracking model, the target tracking model performs feature matching in the search area (equivalent to the entire n-th visible light modality image) shown in the n-th visible light modality image based on the first template image, thereby predicting the position of the target to be tracked in the n-th visible light modality image (referred to as the predicted position) and marking it;

[0071] At the same time, the target tracking model performs feature matching in the search area shown in the n-th frame of the infrared modal image based on the second template image, thereby predicting the position of the target to be tracked in the n-th frame of the infrared modal image (called the predicted position) and marking it;

[0072] The target tracking model performs feature matching in the search area shown in the nth frame of the ultraviolet modal image based on the third template image, thereby predicting the position of the target to be tracked in the nth frame of the ultraviolet modal image (called the predicted position) and marking it.

[0073] After predicting the predicted position of the target to be tracked in the visible light modal image, the infrared modal image and the ultraviolet modal image, the target tracking model outputs the following content as the nth frame prediction result image according to the confidence of the predicted position of the target to be tracked in the nth frame visible light modal image, the confidence of the predicted position of the target to be tracked in the nth frame infrared modal image, and the confidence of the predicted position of the target to be tracked in the nth frame ultraviolet modal image: the nth frame visible light modal image marked with the predicted position of the target to be tracked, or the nth frame infrared modal image marked with the predicted position of the target to be tracked, or the nth frame ultraviolet modal image marked with the predicted position of the target to be tracked.

[0074] In other words, the predicted position of the target to be tracked in the n-th visible light modal image, the predicted position of the target to be tracked in the n-th infrared modal image, and the predicted position of the target to be tracked in the n-th ultraviolet modal image are equivalent to three prediction results; the target tracking model outputs any one prediction result (usually the prediction result with the highest confidence) as the prediction result image of the n-th frame based on the confidence of these three prediction results.

[0075] In practical applications, one of the application scenarios of this solution is:

[0076] Extracting an image region covering the target to be tracked from the first frame (m=1) of the visible light modality image to obtain a first template image, extracting an image region covering the target to be tracked from the first frame of the infrared modality image to obtain a second template image, and extracting an image region covering the target to be tracked from the first frame of the ultraviolet modality image to obtain a third template image;

[0077] Input the first template image, the second template image, the third template image, and each frame (nth frame, n is greater than 1) of subsequently acquired visible light modality image, infrared modality image, and ultraviolet modality image into the target tracking model to perform target tracking, and obtain a multi-frame prediction result image;

[0078] By inputting a frame of visible light modality image, infrared modality image and ultraviolet modality image into the target tracking model, a frame of prediction result image can be obtained;

[0079] That is, after extracting the first template image, the second template image and the third template image from the first frame of visible light modal image, the first frame of infrared modal image and the first frame of ultraviolet modal image respectively, each subsequently acquired frame (nth frame, n is greater than 1) of visible light modal image, infrared modal image and ultraviolet modal image can be predicted to obtain a frame of prediction result image (equivalent to each subsequent frame of image being able to predict the predicted position of the target to be tracked).

[0080] In some embodiments, after obtaining the n-th frame prediction result image, the n-th frame prediction result image is output and displayed, and the n-th frame prediction result image is marked with the predicted position of the target to be tracked;

[0081] It should be noted that the n-th frame prediction result image can be marked with the predicted position of the target to be tracked in the form of a rectangular frame.

[0082] In an embodiment of the present invention, a target tracking model is used to process visible light modal images, infrared modal images and ultraviolet modal images to track a target. Even if the target to be tracked is confused with the background color and there is infrared inefficiency, visible light, infrared and ultraviolet rays can be used to track the target, thereby improving the effect of target tracking and reducing the limitations of target tracking.

[0083] For the above-mentioned embodiments of the present invention Figure 2 The target tracking model mentioned in step S203 is shown in Figure 3 , shows a flowchart of training a target tracking model, including the following steps:

[0084] Step S301: Obtain a sample data set.

[0085] In the process of specifically implementing step S301, a sample data set is obtained, which includes multiple image sequences carrying real annotations, and any image sequence in the sample data set includes: multiple frames of continuous sample visible light modality images, or, multiple frames of continuous sample infrared modality images, or, multiple frames of continuous sample ultraviolet modality images.

[0086] Among them, the real annotations carried by the image sequence are: the label boxes of the real positions of the sample targets (the targets to be tracked or tracked).

[0087] It should be noted that when collecting the sample data set, the sample visible light modal images are collected through the visible light lens, the sample infrared modal images are collected through the infrared lens, and the sample ultraviolet modal images are collected through the ultraviolet lens. The collected sample data set contains multiple image sequences, and a total of multiple image groups.

[0088] The image sequence consists of continuous image frames; the image group refers to: a sample visible light modal image, a sample infrared modal image, and a sample ultraviolet modal image captured in the same scene.

[0089] In the subsequent process of training the specified neural network model using the sample data set, the sample data set is divided into training set, validation set and test set respectively; and the tracking effect is evaluated by three evaluation indicators: precision rate (PR), normalized precision rate (NPR) and success rate (SR).

[0090] Step S302: Train the specified neural network model using the sample data set until the specified neural network model converges to obtain a target tracking model.

[0091] In the specific implementation of step S302, for multiple frames of continuous sample visible light modal images in the sample data set, a first sample template image (called a target area) is extracted from the first frame of the sample visible light modal image; for multiple frames of continuous sample infrared modal images in the sample data set, a second sample template image (called a target area) is extracted from the first frame of the sample infrared modal image; for multiple frames of continuous sample ultraviolet modal images in the sample data set, a third sample template image (called a target area) is extracted from the first frame of the sample ultraviolet modal image;

[0092] Inputting the first sample template image, the second sample template image, the third sample template image, each subsequent frame (each frame after the first frame) of the sample visible light modality image, the sample infrared modality image, and the sample ultraviolet modality image into the specified neural network model for training, thereby training to obtain a target tracking model;

[0093] Among them, each frame of sample visible light modal image, sample infrared modal image, and sample ultraviolet modal image after the first frame input into the specified neural network model is called a search area.

[0094] To better explain the working principle of the specified neural network model, the following Figure 4 The example diagram of the architecture of the specified neural network model is shown for illustration.

[0095] like Figure 4 As shown in the figure, the specified neural network model at least includes a specified deep learning network (ViT), an adaptive candidate elimination mechanism, a connection layer and a prediction head. ViT (Vision Transformer) is the transformer area as the backbone network. The transformer area includes normalization, a multi-head attention layer, candidate elimination, and a multi-layer perceptron head.

[0096] The search area and target area are input into the specified neural network model, where: Figure 4 The target area corresponding to "visible light" is the first sample template image extracted, and the search area corresponding to "visible light" is the sample visible light modality image of the subsequent frame (the image frame after the first sample template image); Figure 4 The target area corresponding to the “thermal infrared” in is the extracted second sample template image, and the search area corresponding to the “thermal infrared” is the sample infrared modality image of the subsequent frame (the image frame after the second sample template image); Figure 4 The target area corresponding to “ultraviolet” in is the extracted third sample template image, and the search area corresponding to “ultraviolet” is the sample ultraviolet modal image of the subsequent frame (the image frame after the third sample template image).

[0097] Specify the neural network model embedding layer ( Figure 4 The segmentation and embedding part in the image segmentation and embedding part) divides and flattens the input RGB search area (sample visible light modality image), T search area (sample infrared modality image), UV search area (sample ultraviolet modality image), the first sample template image, the second sample template image, and the third sample template image into sequence patches (equivalent to feature sequence blocks), and marks (that is, numbers) the sequence patches so as to restore the feature positions after subsequent feature extraction;

[0098] These sequence patches are then input into a series of transformer regions for feature extraction and "search-template matching" under each modality, wherein the "search-template matching" under each modality specifically refers to: matching the sample visible light modality image with the first sample template image to predict the predicted position of the sample target in the sample visible light modality image, matching the sample infrared image with the second sample template image to predict the predicted position of the sample target in the sample infrared modality image, and matching the sample ultraviolet modality image with the third sample template image to predict the predicted position of the sample target in the sample ultraviolet modality image.

[0099] The adaptive candidate elimination mechanism of the specified neural network model is inserted between the transformer regions. The adaptive candidate elimination mechanism is used to select the modal information that is more suitable for the current tracking scene and discard more inefficient modal information;

[0100] Specify the final prediction head (or tracking head) of the neural network model to obtain the features of the connected RGB search area, T search area, and UV search area from the backbone network to obtain the predicted position of the sample target;

[0101] in, Figure 4 Among the three images output by the prediction head, the top image is a sample ultraviolet modality image with the predicted position of the sample target marked, the middle image is a sample visible light modality image with the predicted position of the sample target marked, and the bottom image is a sample infrared modality image with the predicted position of the sample target marked;

[0102] According to the confidences of the aforementioned three sample ultraviolet modal images, sample visible light modal images, and sample infrared modal images marked with the predicted positions of the sample targets (equivalent to the confidences of the three predicted positions), the neural network model is specified to output any one predicted position (usually the predicted position with the highest confidence), that is, the prediction result finally output by the specified neural network model is: the sample ultraviolet modal image marked with the predicted position of the sample target, or, the sample visible light modal image marked with the predicted position of the sample target, or, the sample infrared modal image marked with the predicted position of the sample target.

[0103] The architecture of the above-specified neural network model is the overall architecture of the VTUTrack algorithm. In general, the target area and the search area are segmented, flattened, and linearly projected through a triple embedding layer, and then input into the transformer area for "feature extraction" and "search-template matching"; the adaptive candidate elimination mechanism is responsible for retaining the features of modalities with higher reliability and reducing the interference of inefficient modalities; finally, the features of the RGB search area, the T search area, and the UV search area are connected and input into the prediction head to predict the predicted position of the sample target.

[0104] Regarding the adaptive candidate elimination mechanism mentioned above, this adaptive candidate elimination mechanism can help ViT select more features of the search area of ​​the efficient mode that is suitable for the current tracking scene in the early stage, and remove more background features of the search area of ​​the inefficient mode, thereby reducing the computational burden and reducing the negative impact of background area noise caused by the inefficient mode on feature learning;

[0105] An adaptive candidate elimination mechanism is inserted multiple times between the transformer regions of the ViT backbone network, and the feature selection modules of the three modalities are controlled by calculating the reliability parameters of the modalities.

[0106] The architecture of the adaptive candidate elimination mechanism is as follows Figure 5 As shown in Figure 1, the adaptive candidate elimination mechanism consists of feature retention rate, reliable evaluation, candidate elimination, softmax, etc. Figure 5 R rgb is the reliability score of the visible light modality, R uv is the reliability score of the UV modality (also called the UV modality), R t is the reliability score of the infrared modality, λ rgb is the adaptive weight of the visible light modality, λ uv is the adaptive weight of the purple line mode, λ t is the adaptive weight of the infrared modality.

[0107] It should be noted that for R rgb , R uv and R t After processing, we can get λ rgb , uv and λ t , specifically, λ rgb , uv and λ t The specific calculation process of λ is: rgb ,λ uv ,λ t = softmax(R rgb ,R uv ,R t ).

[0108] The above embodiments are detailed descriptions of this solution. Figures 1 to 5 As can be seen from the content, by utilizing the characteristics of ultraviolet imaging, ultraviolet light is used as a supplementary means of visible light and infrared to track the target, which solves the problem of target and background color confusion and inefficient infrared tracking caused by color camouflage and anti-infrared of the target, and realizes a multimodal tracker that can adapt to complex tracking scenarios.

[0109] In addition, this scheme proposes the VTUTrack algorithm, which ensures efficient reasoning speed and improves tracking performance by introducing an adaptive candidate elimination mechanism based on modal reliability; from the research results, the ultraviolet modality can provide information supplement to visible light and infrared in complex field environments, has a variety of wide application scenarios, and can play a key role in specific fields.

[0110] In order to better reflect the beneficial effects of this solution, the following contents are used for explanation:

[0111] Seven different modal combinations are used to conduct comparative experiments on the constructed sample dataset, and the effects of these seven modal combinations on the precision rate (PR), normalized precision rate (NPR) and success rate (SR) of the target tracker are analyzed respectively.

[0112] Among them, the seven modal combinations are: RGB modality (visible light modality), T modality (infrared modality), UV modality (ultraviolet modality), T-UV modality (infrared-ultraviolet modality), RGB-UV modality (visible light-ultraviolet modality), RGB-T modality (visible light-infrared modality), and RGB-T-UV modality (visible light-infrared-ultraviolet modality).

[0113] In addition to the above 7 modal combinations, this solution also provides the ground truth (GT) of the target to be tracked for comparison.

[0114] This solution is a target tracking method based on RGB-T-UV modality.

[0115] The above 7 modal combinations are used to track the target respectively. The target tracking effect of each modal combination and the real position of the target to be tracked are shown in the figure below. Figure 6 As shown, Figure 6 The rectangular boxes of various colors (except the color corresponding to GT) are: the predicted positions of the targets to be tracked obtained by combining the predictions of various modes; Figure 6 The rectangular box with the corresponding color (red) of GT is the actual position of the target to be tracked.

[0116] from Figure 6 It can be seen from the example figures of target tracking effects of different modal combinations that the target tracking effect of the RGB-T-UV modality combination is better than the tracking effects of other modal combinations.

[0117] Compared with the modal combination consisting of two modalities (RGB-T modality, T-UV modality and other modal combinations), the precision rate (PR), normalized precision rate (NPR) and success rate (SR) of the target tracking effect of the modal combination of RGB-T-UV modality have been improved as follows: SR has increased by 6.32%, PR has increased by 5.46%, and NPR has increased by 4.46%.

[0118] Compared with the modal combination composed of a single modality (RGB modality, T modality and other modal combinations), the precision rate (PR), normalized precision rate (NPR) and success rate (SR) of the target tracking effect of the modal combination of RGB-T-UV modality have been improved as follows: SR increased by 22.15%, PR increased by 28.63%, and NPR increased by 24.93%.

[0119] From the above improvements, the target tracking solution based on RGB-T-UV modality provided by this solution can adapt to more complex tracking scenarios;

[0120] In addition, since the infrared modality is not affected by light intensity, when the ultraviolet modality and visible light modality are in modal failure and modal inefficiency respectively at night, the infrared modality in the RGB-T-UV modality can compensate for the information of the ultraviolet modality and visible light modality.

[0121] When using this solution for target tracking, an adaptive candidate elimination mechanism is introduced to enable the target tracking reasoning speed to reach 45FPS, which meets the speed of real-time reasoning.

[0122] In order to verify the effectiveness of the adaptive candidate elimination mechanism, a comparative experimental group of "target tracking with adaptive candidate elimination mechanism" and "target tracking without adaptive candidate elimination mechanism" was set up, and the effects of "target tracking with adaptive candidate elimination mechanism" and "target tracking without adaptive candidate elimination mechanism" were evaluated by inference speed (FPS), product calculation amount (MACs) and tracking performance. From the experimental results, compared with "target tracking without adaptive candidate elimination mechanism", the improvement of "target tracking with adaptive candidate elimination mechanism" is as follows: MACs are reduced by 25.9%, tracking performance and inference speed are improved by 50.2%, SR is increased by 0.91%, PR is increased by 1.76%, and NPR is increased by 0.74%.

[0123] The improvements in “Object Tracking with Adaptive Candidate Elimination” mentioned above show that adding the adaptive candidate elimination mechanism can significantly improve the inference speed while ensuring good tracking performance.

[0124] Corresponding to the target tracking method based on visible light, infrared light and ultraviolet light provided in the above embodiment of the present invention, see Figure 7 , the embodiment of the present invention also provides a structural block diagram of a target tracking device based on visible light, infrared light and ultraviolet light, the target tracking device includes: a collection unit 701, an extraction unit 702 and a tracking unit 703;

[0125] The acquisition unit 701 is used to acquire visible light modality images, infrared modality images and ultraviolet modality images in real time.

[0126] In a specific implementation, the acquisition unit 701 is specifically used to: call the visible light lens to collect visible light modal images in real time, call the infrared lens to collect infrared modal images in real time, and call the ultraviolet lens to collect ultraviolet modal images in real time.

[0127] The extraction unit 702 is used to extract a first template image, a second template image and a third template image covering the target to be tracked from the m-th visible light modality image frame, the m-th infrared modality image frame and the m-th ultraviolet modality image frame, respectively.

[0128] In a specific implementation, the extraction unit 702 is specifically used to: in response to a labeling operation triggered by a user, extract an image area covering the target to be tracked from the m-th frame of the visible light modal image to obtain a first template image, extract an image area covering the target to be tracked from the m-th frame of the infrared modal image to obtain a second template image, and extract an image area covering the target to be tracked from the m-th frame of the ultraviolet modal image to obtain a third template image.

[0129] The tracking unit 703 is used to input the first template image, the second template image, the third template image, the n-th visible light modal image, the n-th infrared modal image, and the n-th ultraviolet modal image into a preset target tracking model for target tracking to obtain an n-th prediction result image, wherein n is greater than m, and the prediction result image is annotated with a predicted position of a target to be tracked. The target tracking model is obtained by training a specified neural network model based on a sample data set, and the specified neural network model at least includes a specified deep learning network and an adaptive candidate elimination mechanism.

[0130] In some embodiments, the nth frame prediction result image is: the nth frame visible light modal image marked with the predicted position of the target to be tracked, or the nth frame infrared modal image marked with the predicted position of the target to be tracked, or the nth frame ultraviolet modal image marked with the predicted position of the target to be tracked.

[0131] In an embodiment of the present invention, a target tracking model is used to process visible light modal images, infrared modal images and ultraviolet modal images to track a target. Even if the target to be tracked is confused with the background color and there is infrared inefficiency, visible light, infrared and ultraviolet rays can be used to track the target, thereby improving the effect of target tracking and reducing the limitations of target tracking.

[0132] Preferably, combined Figure 7 As shown in the figure, the tracking unit 703 includes an acquisition subunit and a training subunit, and the execution principle of each subunit is as follows:

[0133] The acquisition subunit is used to acquire a sample data set, which includes multiple image sequences carrying real annotations. Any image sequence includes: multiple frames of continuous sample visible light modal images, or multiple frames of continuous sample infrared modal images, or multiple frames of continuous sample ultraviolet modal images.

[0134] The training subunit is used to train the specified neural network model using the sample data set until the specified neural network model converges to obtain a target tracking model.

[0135] Preferably, combined Figure 7 The target tracking device further comprises:

[0136] The display unit is used to output and display the prediction result image of the nth frame.

[0137] In summary, the embodiments of the present invention provide a target tracking method and device based on visible light, infrared and ultraviolet rays, which utilize a target tracking model to process visible light modal images, infrared modal images and ultraviolet modal images to track a target to be tracked. Even if the target to be tracked is confused with the background color and there is infrared inefficiency, visible light, infrared and ultraviolet rays can be used to track the target to be tracked, thereby improving the effect of target tracking and reducing the limitations of target tracking.

[0138] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0139] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0140] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A target tracking method based on visible light, infrared light and ultraviolet light, characterized in that: The method comprises: Real-time acquisition of visible light modality images, infrared modality images, and ultraviolet modality images; Extracting a first template image, a second template image, and a third template image covering the target to be tracked from the m-th frame of the visible light modality image, the m-th frame of the infrared modality image, and the m-th frame of the ultraviolet modality image, respectively; The first template image, the second template image, the third template image, the n-th frame visible light modal image, the n-th frame infrared modal image, and the n-th frame ultraviolet modal image are input into a preset target tracking model for target tracking to obtain an n-th frame prediction result image, wherein n is greater than m, and the prediction result image is annotated with the predicted position of the target to be tracked. The target tracking model is obtained by training a specified neural network model based on a sample data set, and the specified neural network model at least includes a specified deep learning network and an adaptive candidate elimination mechanism.

2. The method according to claim 1, characterized in that Real-time acquisition of visible light, infrared and ultraviolet modality images, including: The visible light lens is called to collect visible light modal images in real time, the infrared lens is called to collect infrared modal images in real time, and the ultraviolet lens is called to collect ultraviolet modal images in real time.

3. The method according to claim 1, characterized in that: Extracting a first template image, a second template image, and a third template image covering the target to be tracked from the m-th frame of the visible light modal image, the m-th frame of the infrared modal image, and the m-th frame of the ultraviolet modal image, respectively, including: In response to a labeling operation triggered by a user, an image area covering the target to be tracked is extracted from the visible light modal image of the mth frame to obtain a first template image, an image area covering the target to be tracked is extracted from the infrared modal image of the mth frame to obtain a second template image, and an image area covering the target to be tracked is extracted from the ultraviolet modal image of the mth frame to obtain a third template image.

4. The method according to claim 1, characterized in that The process of training a specified neural network model based on a sample data set to obtain a target tracking model includes: Acquire a sample data set, wherein the sample data set includes a plurality of image sequences carrying real annotations, and any of the image sequences includes: a plurality of frames of continuous sample visible light modality images, or a plurality of frames of continuous sample infrared modality images, or a plurality of frames of continuous sample ultraviolet modality images; The specified neural network model is trained using the sample data set until the specified neural network model converges to obtain a target tracking model.

5. The method according to any one of claims 1 to 4, characterized in that: The prediction result image of the nth frame is: the visible light modality image of the nth frame marked with the predicted position of the target to be tracked, or the infrared modality image of the nth frame marked with the predicted position of the target to be tracked, or the ultraviolet modality image of the nth frame marked with the predicted position of the target to be tracked.

6. The method according to any one of claims 1 to 4, characterized in that: After obtaining the n-th frame prediction result image, it also includes: The output displays the predicted result image of the nth frame.

7. A target tracking device based on visible light, infrared light and ultraviolet light, characterized in that: The device comprises: An acquisition unit, used for acquiring visible light modality images, infrared modality images and ultraviolet modality images in real time; an extraction unit, configured to respectively extract a first template image, a second template image, and a third template image covering the target to be tracked from the m-th frame of the visible light modal image, the m-th frame of the infrared modal image, and the m-th frame of the ultraviolet modal image; A tracking unit is used to input the first template image, the second template image, the third template image, the n-th frame visible light modal image, the n-th frame infrared modal image, and the n-th frame ultraviolet modal image into a preset target tracking model for target tracking to obtain an n-th frame prediction result image, wherein n is greater than m, and the prediction result image is annotated with the predicted position of the target to be tracked. The target tracking model is obtained by training a specified neural network model based on a sample data set, and the specified neural network model at least includes a specified deep learning network and an adaptive candidate elimination mechanism.

8. The device according to claim 7, characterized in that The acquisition unit is specifically used to: call the visible light lens to collect visible light modal images in real time, call the infrared lens to collect infrared modal images in real time, and call the ultraviolet lens to collect ultraviolet modal images in real time.

9. The device according to claim 7, characterized in that The extraction unit is specifically used to: in response to a labeling operation triggered by a user, extract an image area covering the target to be tracked from the visible light modal image of the mth frame to obtain a first template image, extract an image area covering the target to be tracked from the infrared modal image of the mth frame to obtain a second template image, and extract an image area covering the target to be tracked from the ultraviolet modal image of the mth frame to obtain a third template image.

10. The device according to claim 7, characterized in that The tracking unit comprises: An acquisition subunit is used to acquire a sample data set, wherein the sample data set includes a plurality of image sequences carrying real annotations, and any of the image sequences includes: a plurality of continuous frames of sample visible light modality images, or a plurality of continuous frames of sample infrared modality images, or a plurality of continuous frames of sample ultraviolet modality images; The training subunit is used to train the specified neural network model using the sample data set until the specified neural network model converges to obtain a target tracking model.

Images