Hyperspectral image-based tracking network training method, tracking method and medium
By constructing a combination of HSI and RGB domain transfer modules and loss functions, and fusing HSI image information, the problem of insufficient accuracy of existing deep learning target tracking methods in complex scenes is solved. This enables the effective application of hyperspectral images in deep learning networks and improves the robustness and accuracy of target tracking.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHENZHEN MAXVISION TECH
- Filing Date
- 2023-07-18
- Publication Date
- 2026-06-09
Smart Images
Figure CN116933844B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of digital image processing, and more specifically, to a method for training a tracking network based on hyperspectral images, a tracking method, and a medium. Background Technology
[0002] Target tracking is a fundamental yet challenging task in computer vision research. It is widely used in fields such as video surveillance, robotics, and military reconnaissance. Robust target tracking is achieved by continuously predicting and updating the target state information in subsequent frames using the target information provided in the first frame of a video. Traditional target tracking methods sometimes cannot handle or adapt to complex tracking changes, and their robustness and accuracy have been surpassed by deep learning target tracking algorithms.
[0003] Deep learning tracking network models can adapt to more scenarios compared to existing traditional tracking methods. However, current deep learning-based tracking methods are primarily trained on RGB image datasets, which have limitations in describing the physical properties of objects. This can easily lead to insufficient accuracy for RGB video-based trackers in scenarios where the target and background colors or textures are similar. In contrast to RGB video tracking, existing techniques based on hyperspectral image sensing (HSI) utilize the rich radiometric, spatial, and spectral information of HSI images, features that improve tracking accuracy. However, HSI is difficult to directly apply to existing deep learning tracking networks, preventing them from fully extracting HSI image features to improve target tracking accuracy. Summary of the Invention
[0004] In view of the prior art, the purpose of this application is to provide a hyperspectral image-based tracking network training method, tracking method and medium that can effectively fuse HSI image information to improve tracking accuracy.
[0005] In a first aspect, this application provides a method for training a hyperspectral tracking network, comprising:
[0006] Acquire HSI and RGB video frame images of the same scene;
[0007] Constructing a target tracking network model: Input the image patch region of the same tracked target in the HSI images of every two consecutive frames into a domain transfer module used to align the HSI domain and RGB domain, and input the output features of the domain transfer module into the deep learning tracking network;
[0008] The loss function of the target tracking network model is constructed as follows: For the same tracked target in each frame of HSI image and its corresponding frame of RGB image, an inter-domain loss function S1 is constructed between the image patch region processed by the domain transfer module and the image patch region in the RGB image. The loss function S2 for the deep learning tracking network and the inter-domain loss function S1 are then combined to obtain the tracking training network loss function S; and
[0009] The loss function S is trained to its minimum to obtain the target tracking network model.
[0010] In the training method of hyperspectral tracking network, HSI image information is fused, and the ability of feature alignment between the two domains is improved by using the domain transfer module and the inter-domain loss function between the HSI domain and the RGB domain. This enhances the ability of the deep learning tracking network to extract the target spectral features in HSI, thereby ultimately improving the accuracy of the trained target tracking model in target tracking.
[0011] Based on the first aspect, in one possible implementation, obtaining the image block region of the same tracked target in the HSI images of every two consecutive frames specifically involves:
[0012] Initialize the target to be tracked in the HSI video: Given the initial center position of the target to be tracked in the first frame of the HSI image, select the initial image block region of the target to be tracked with the initial center position as the center;
[0013] The target to be tracked is acquired in a loop in the HSI video: In each subsequent frame of HSI image, the center position of the HSI image block region of the previous frame is used as the center to zoom in to obtain the image block region of the target to be tracked in the next frame of HSI image.
[0014] Based on the first aspect, in one possible implementation, the domain migration module includes:
[0015] The system consists of a first convolutional layer that extracts image patch regions for each tracked target, a normalization layer that normalizes the features extracted by the convolutional layer, an activation layer that receives and processes the normalization results, and a second convolutional layer that extracts features again from the activation results.
[0016] Based on the first aspect, in one possible implementation, for the same tracking target in each HSI image and its corresponding RGB image, the inter-domain loss function S1 between the image patch region processed by the domain transfer module and the image patch region in the RGB image is constructed as follows:
[0017] Initialize the target to be tracked in the RGB video: Given the initial center position of the target to be tracked in the first frame of the HSI image, select the initial image block region of the target to be tracked with the initial center position as the center;
[0018] The target to be tracked is obtained in a loop in the RGB video: In each subsequent frame of HSI image, the center position of the HSI image block region of the previous frame is used as the center to zoom in to obtain the image block region of the target to be tracked in the HSI image of the next frame.
[0019] Constructing an inter-domain loss function S1 for image patch regions: The third convolutional layer processes the image patch regions of each frame of the RGB image, unifying the channels and sizes of the feature maps of the image patch regions in the RGB image and the corresponding HSI image after passing through the first convolutional layer. Then, an inter-domain loss function S1 for image patch regions is constructed between each HSI image and its corresponding RGB image.
[0020] Where W and H are the width and height of the image patch region, respectively, C is the total number of channels, and n represents the nth channel. This represents the feature value of a feature map obtained after processing an image patch region through an activation layer in a frame of an HSI image when the feature map is located at position (i, j) in the nth channel. The feature value is the feature map of the image block region in the RGB image corresponding to the HSI image of this frame when the position in the nth channel is (i, j) after the third convolution processing.
[0021] Based on the first aspect, in one possible implementation, the first, second, and third convolutional layers are all two-dimensional convolutional layers, the normalization layer uses BatchNormal normalization, and the activation layer uses the sigmoid activation function.
[0022] Based on the first aspect, one possible implementation involves combining the loss function S2 for the deep learning tracking network and the inter-domain loss function S1 to obtain the tracking training network loss function S:
[0023] S = α*S1 + β*S2;
[0024] Where α and β are weighting factors, and α + β = 1.
[0025] Based on the first aspect, in one possible implementation, the deep learning tracking network is a siamCAR baseline tracker, which uses ResNet-50 as the backbone network for feature extraction and CAR-Head as the classification and regression network.
[0026] Where S2=λ1L cen+λ2L reg +λ3L d
[0027] Where λ1, λ2, and λ3 are weighting factors, λ1 + λ2 + λ3 = 1, L cen L cen and L reg These represent the cross-entropy loss, centrality loss, and regression loss in the siamCAR baseline tracker, respectively.
[0028] Based on the first aspect, one possible implementation is that during training...
[0029] First training: Using RGB video images as the training set, the SiamCAR baseline tracker is trained using loss function S2 as the loss function during the first training.
[0030] Second training: On the trained siamCAR baseline tracker, HSI video images and RGB video images are used as the training set, and the loss function S is used as the loss function for the second training.
[0031] Secondly, this application provides a hyperspectral-based target tracking method, which includes:
[0032] Acquire HSI video images of the target to be tracked; and
[0033] The target tracking model obtained by the hyperspectral-based target tracking training method described above is used to track targets in HSI video images.
[0034] Thirdly, this application provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the hyperspectral-based target tracking training method. Attached Figure Description
[0035] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0036] Figure 1 This is a flowchart of a hyperspectral-based tracking network training method according to an embodiment of this application;
[0037] Figure 2 This is a schematic diagram illustrating the structure of the target tracking network model in an embodiment of this application;
[0038] Figure 3 This is a schematic diagram of the domain migration module for building a target tracking network model according to an embodiment of this application. Detailed Implementation
[0039] To make the technical problems, technical solutions, and beneficial effects to be solved by this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this application.
[0040] It should be noted that when a component is referred to as being "fixed to" or "set on" another component, it can be directly on or indirectly on that other component. When a component is referred to as being "connected to" another component, it can be directly connected to or indirectly connected to that other component.
[0041] It should be understood that the terms "length", "width", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.
[0042] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.
[0043] The training method, tracking method, and medium of the tracking network based on hyperspectral images of this application will now be described in detail with reference to the accompanying drawings.
[0044] Reference Figure 1 The tracking network training method based on hyperspectral images provided in this application includes steps S100 to S400.
[0045] Step S100: Obtain training dataset: Obtain HSI (Hyperspectral image) video frame images and RGB video frame images of the same scene.
[0046] To improve the accuracy of target tracking in this embodiment, HSI images are fused into a deep learning target tracking network. Target tracking is performed between consecutive frames. For the HSI and RGB video frame images of the same scene obtained in step S1, one HSI image and one RGB image are acquired simultaneously. This ensures a one-to-one temporal correspondence between the HSI and RGB video frame images, guaranteeing that the positions of the same target in the HSI and RGB images at the same time correspond. Each frame of the hyperspectral image has many channels; the exact number depends on the wavelength resolution of the sensor, with each channel capturing a specific wavelength of light. It is worth noting that both the HSI and RGB video frame images are captured continuously in time, with a frame rate of up to 24fps.
[0047] Step S200: Construct the target tracking network model: Input the image patch region of the same tracked target in the HSI images of every two consecutive frames into the domain transfer module used to align the HSI domain and RGB domain, and input the output features of the domain transfer module into the deep learning tracking network.
[0048] Step S300: Construct the loss function of the target tracking network model: For the same tracking target in each frame of HSI image and the corresponding frame of RGB image, construct the inter-domain loss function S1 between the image patch region processed by the domain transfer module and the image patch region in the RGB image. Combine the loss function S2 of the deep learning tracking network and the inter-domain loss function S1 to obtain the tracking training network loss function S.
[0049] For example, if the image patch region of a tracked target in a frame of HSI image is denoted as F1, and the image patch region of the same tracked target in a frame of RGB image acquired at the corresponding time of the same frame of HSI image is denoted as F2, then the constructed loss function S1 is the difference between the image patch region denoted as F1 and the image patch region denoted as F2.
[0050] For step S200, as Figure 2 and Figure 3 As shown, the domain transfer module used in this application embodiment specifically includes a first convolutional layer for extracting image block regions of each tracked target, a normalization layer for normalizing the features extracted by the convolutional layer, an activation layer for receiving and processing the normalization result, and a second convolutional layer for extracting features again from the activation result.
[0051] In one embodiment, both the first and second convolutional layers are two-dimensional convolutional layers, the normalization layer uses BatchNormal normalization, and the activation layer uses the sigmoid activation function. Normalizing the feature data to a range of 0 to 1 improves the computational efficiency of the domain transfer module and enhances the standardization of data features.
[0052] For step S200, in constructing the target tracking network model, obtaining the image patch region of the same tracked target in the HSI images of every two consecutive frames specifically involves:
[0053] Step S210: Initialize the target to be tracked in the HSI video: In the first frame of the HSI image, give the initial center position of the target to be tracked, and select the initial image block region of the target to be tracked with the initial center position as the center.
[0054] Step S220: Loop through the target to be tracked in the HSI video: In each subsequent frame of HSI image, zoom in on the center of the HSI image block region of the previous frame to obtain the image block region of the target to be tracked in the next frame of HSI image.
[0055] It is worth noting that in continuously captured video frames, the positional change of the same moving target between consecutive frames is minimal, and the higher the frame rate, the smaller the positional change. Based on this, in steps S210 and S220, given the position of the target to be tracked in the initial frame and the selected image block region of the target, the image block region of the previous frame is expanded using the center position of the image block region in the previous frame as the center, and this expanded image block region is used as the image block region of the target to be tracked in the next frame. The iterative acquisition of image block regions in steps S210 and S220 does not require complex tracking algorithms to track the corresponding image block region of the target. It is understandable that using existing target tracking algorithms to obtain the moving target region in each frame would take longer than steps S210 and S220; therefore, steps S210 and S220 are beneficial for improving the running speed.
[0056] For step S300, for the same tracking target in each HSI image and its corresponding RGB image, the inter-domain loss function S1 between the image patch region processed by the domain transfer module and the image patch region in the RGB image is constructed as follows:
[0057] Step S310: Initialize the target to be tracked in the RGB video: In the first frame HSI image, give the initial center position of the target to be tracked, and select the initial image block region of the target to be tracked with the initial center position as the center.
[0058] Step S320: Loop through the RGB video to acquire the target to be tracked: In each subsequent frame of HSI image, zoom in on the center of the HSI image block region of the previous frame to obtain the image block region of the target to be tracked in the next frame of HSI image.
[0059] Step S330: Constructing an inter-domain loss function for image patch regions. S1: Process the image patch regions of each frame of the RGB image using the third convolutional layer, unifying the channels and sizes of the feature maps of the image patch regions of the RGB image in that frame and the image patch regions in the corresponding HSI image of that frame after passing through the activation layer. Construct an inter-domain loss function S1 for image patch regions between each frame of the HSI image and its corresponding RGB image. The third convolutional layer can be a two-dimensional convolutional layer.
[0060] In the above steps, W and H are the width and height of the image patch region, respectively, C is the total number of channels, and n represents the nth channel. This represents the feature value of a feature map obtained after processing a patch region in an HSI image frame through the first convolutional layer when the feature map is located at position (i, j) in the nth channel. The feature value is the feature map of the image block region in the RGB image corresponding to the HSI image of this frame when the position in the nth channel is (i, j) after the third convolution processing.
[0061] It is worth noting that the benefits of steps S310 and S320 are the same as those of steps S210 and S220, and will not be repeated here.
[0062] In one embodiment, the process of magnifying the HSI image block region in the previous frame as the center position in step S220 to obtain the image block region of the target to be tracked in the next frame HSI image, and the process of magnifying the HSI image block region in the previous frame as the center position in step S320 to obtain the image block region of the target to be tracked in the next frame HSI image, can be performed using the following steps:
[0063] (1) Set the center position of the target to be tracked in the initial frame as (x0, y0) and the corresponding image block region (w0, h0) of the target to be tracked in the initial frame, where w0 and h0 are the width and height of the image block region in the initial frame, respectively;
[0064] (2) Obtain the image block region (w1,h1) of the target to be tracked in the next frame: crop out the region with a width of w1 = w0 + (w0 + h0) / 2 and a height of h1 = h0 + (w0 + h0) / 2 with (x0,y0) as the center point, and use it as the image block region (w1,h1) of the next frame;
[0065] (3) Thus, continuously acquire the image block region of the tracked target in the next frame, w k =w k-1 +(w k-1 +h k-1 ) / 2,h k =h k-1 +(w k-1 +h k-1 ) / 2, where k and k-1 represent the two frames of images respectively; and the size of the image block area of all frames is made consistent, and all are resized to 255×255.
[0066] It is worth noting that, such as Figure 2 and Figure 3 As shown, the first convolutional layer, normalization layer, and activation layer of the domain transfer module in step S200 constitute the feature distribution adaptation part, wherein... Figure 2 Feature distribution alignment and Figure 3 The first convolutional layer has the same meaning as the second convolutional layer, which is another feature extraction part. The outputs of the feature distribution adaptation part and the third convolutional layer are used as the basis for calculating the loss function S1. Since hyperspectral HSI images are multi-channel images, to facilitate the subsequent calculation of the loss function S1, it is necessary to unify the size and number of channels of the output feature map of the feature distribution adaptation part and the output feature map of the third convolutional layer. For example, if the image block region of the tracking target extracted in each HSI image in step S200 has 16 channels and a size of 255×255, then the third convolutional layer can unify the image block region image of each frame of RGB image to 16 channels and also unify the size to 255×255.
[0067] Furthermore, since existing deep learning tracking networks are initially trained based on RGB images, a domain transfer module and an inter-domain loss function S1 are designed to bring the HSI source domain closer to the target RGB domain, i.e., to reduce the difference between the HSI source domain and the target RGB domain. During training, the inter-domain loss function S1 is continuously optimized, adjusting the parameters of each layer in the domain transfer module and the parameters of the third convolutional layer until S1 is minimized. At this point, the original deep learning tracking network can better utilize the information from the HSI hyperspectral image, thereby improving the target tracking accuracy of the target tracking model in the final training step S400. Thus, the hyperspectral image-based tracking network training method of this application can not only utilize deep learning to adapt to more scenarios but also leverage HSI images to improve the tracking accuracy of deep learning target tracking networks.
[0068] For step S300, the loss function S2 and the inter-domain loss function S1 of the deep learning tracking network are combined to obtain the tracking training network loss function S: S=α*S1+β*S2;
[0069] Where α and β are weighting factors, and α + β = 1.
[0070] And, as Figure 2 and Figure 3 As shown, the deep learning tracking network mentioned in step S300 is the siamCAR baseline tracker, which uses ResNet-50 as the backbone network for feature extraction and CAR-Head as the classification and regression network.
[0071] Where S2=λ1L cen +λ2L reg +λ3L d λ1, λ2, and λ3 are weighting factors, λ1 + λ2 + λ3 = 1, L cen L cen and L reg These represent the cross-entropy loss, centrality loss, and regression loss in the siamCAR baseline tracker, respectively.
[0072] Step S400: Train the tracking network model: train the loss function S to a minimum to obtain the target tracking network model.
[0073] During training, the following training steps are performed:
[0074] First training: Using RGB video images as the training set, the SiamCAR baseline tracker is trained using the loss function S2 as the loss function during the first training.
[0075] Second training: On the trained siamCAR baseline tracker, HSI video images and RGB video images are used as the training set, and the loss function S is used as the loss function for the second training.
[0076] It is worth noting that in the first training process, the siamCAR baseline tracker is trained, and then the final tracking network model is trained when HSI video images are fused. In the second training process, the domain transfer module is continuously optimized and S1 is continuously optimized to align the HSI domain and RGB domain. This continuously improves the ability of the siamCAR baseline tracker's HSI backbone network to extract the target spectral features, ultimately improving the tracking accuracy of the deep learning target tracker on HSI video images.
[0077] This application also provides a hyperspectral target tracking method, the method comprising:
[0078] Acquire HSI video images of the target to be tracked; and,
[0079] The target tracking model obtained by the above-mentioned hyperspectral-based target tracking training method is used to track targets in HSI video images.
[0080] In this hyperspectral-based target tracking method, only HSI video images are needed as input, eliminating the need for RGB video images. The target tracking model can then remove the position of the moving target in each frame of the HSI image and select the size of the output box for the moving target. The area selected by the output box can be understood as an image block region related to the moving target.
[0081] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the hyperspectral-based target tracking training method.
[0082] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the protection scope of this application.
Claims
1. A target tracking training method based on hyperspectral imaging, characterized in that, include: Acquire HSI and RGB video frame images of the same scene; Constructing a target tracking network model: Input the image patch region of the same tracked target in the HSI images of every two consecutive frames into a domain transfer module used to align the HSI domain and RGB domain, and input the output features of the domain transfer module into the deep learning tracking network; The specific steps for obtaining the image patch region of the same tracked target in two consecutive HSI images are as follows: Initialize the target to be tracked in the HSI video: In the first frame of the HSI image, give the initial center position of the target to be tracked, and select the initial image patch region of the target to be tracked with the initial center position as the center; Loop through the target to be tracked in the HSI video: In each subsequent frame of the HSI image, zoom in with the center position of the HSI image patch region of the previous frame as the center to obtain the image patch region of the target to be tracked in the next frame of the HSI image; The domain transfer module includes: a first convolutional layer for extracting image block regions of each tracked target, a normalization layer for normalizing the features extracted by the convolutional layer, an activation layer for receiving and processing the normalization result, and a second convolutional layer for extracting features again from the activation result; wherein the first, second, and third convolutional layers are all two-dimensional convolutional layers, the normalization layer uses BatchNormal normalization processing, and the activation layer uses the sigmoid activation function; The loss function of the target tracking network model is constructed as follows: For the same tracked target in each frame of HSI image and its corresponding frame of RGB image, an inter-domain loss function S1 is constructed between the image patch region processed by the domain transfer module and the image patch region in the RGB image. The loss function S2 for the deep learning tracking network and the inter-domain loss function S1 are then combined to obtain the tracking training network loss function S; and Train the loss function S to its minimum to obtain the target tracking network model; Construct an inter-domain loss function S1 between each frame of the HSI image and the corresponding RGB image of the same image patch region of the tracked target, including: Initialize the target to be tracked in the RGB video: Given the initial center position of the target to be tracked in the first frame of the HSI image, select the initial image block region of the target to be tracked with the initial center position as the center; The target to be tracked is obtained in a loop in the RGB video: In each subsequent frame of HSI image, the center position of the HSI image block region of the previous frame is used as the center to zoom in to obtain the image block region of the target to be tracked in the HSI image of the next frame. Constructing an inter-domain loss function S1 for image patch regions: The third convolutional layer processes the image patch regions of each frame of the RGB image, unifying the channels and sizes of the feature maps of the image patch regions in the RGB image and the corresponding HSI image after passing through the first convolutional layer. Then, an inter-domain loss function S1 for image patch regions is constructed between each HSI image and its corresponding RGB image. , Where W and H are the width and height of the image patch region, respectively, C is the total number of channels, n represents the nth channel, and n represents the feature value of the feature map of the image patch region after activation layer processing in a frame of HSI image when the position of the feature map in the nth channel is (i, j). The feature value is the feature map of the image block region in the RGB image corresponding to the HSI image of this frame when the position in the nth channel is (i, j) after the third convolution process.
2. The target tracking training method based on hyperspectral imaging as described in claim 1, characterized in that, The loss function S2 for deep learning tracking networks and the inter-domain loss function S1 are combined to obtain the tracking training network loss function S: S = α*S1 + β*S2; Where α and β are weighting factors, and α+β=1.
3. The target tracking training method based on hyperspectral imaging as described in claim 2, characterized in that, The deep learning tracking network is the siamCAR baseline tracker, which uses ResNet-50 as the backbone network for feature extraction and CAR-Head as the classification and regression network. in, , in, , and As a weighting factor, , , and These represent the cross-entropy loss, centrality loss, and regression loss in the siamCAR baseline tracker, respectively.
4. The target tracking training method based on hyperspectral imaging as described in claim 3, characterized in that, During training, First training: Using RGB video images as the training set, the SiamCAR baseline tracker is trained using loss function S2 as the loss function during the first training. Second training: On the trained siamCAR baseline tracker, HSI video images and RGB video images are used as the training set, and the loss function S is used as the loss function for the second training.
5. A target tracking method based on hyperspectral imaging, characterized in that: Acquire HSI video images of the target to be tracked; as well as The target tracking model is obtained by using the hyperspectral-based target tracking training method as described in any one of claims 1 to 4 to perform target tracking on HSI video images.
6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the hyperspectral-based target tracking training method according to any one of claims 1 to 4.