Target tracking method, imaging device, and computer program product

Abstract

The application provides a target tracking method, an imaging device and a computer program product, and relates to the technical field of tracking and shooting. The target tracking method comprises the following steps: selecting a target initial region of a target to be tracked in a target frame; processing a frame to be processed subsequent to the target frame based on the target initial region by using a target tracking model to obtain a position region of the target to be tracked in the frame to be processed; the target tracking model comprises a template region feature extraction network and a search region feature extraction network; and based on the position region, the imaging device is controlled to track and shoot the target to be tracked. The target initial region corresponding to the target to be tracked is processed by using the target tracking model capable of realizing the feature region function of different regions, so as to determine the position region of the target to be tracked, and then the imaging device is controlled to track and shoot the target to be tracked based on the position region. Stable tracking precision can be maintained under the conditions of rapid target motion, deformation, local occlusion and the like.

Description

Technical Field

[0001] This application relates to the field of tracking and imaging technology, and more specifically, to a target tracking method, imaging device, and computer program product. Background Technology

[0002] In photography, target tracking algorithms are typically used for tracking and capturing images of objects. Existing target tracking algorithms mainly employ correlation filtering series, optical flow, and feature point tracking. These algorithms are weak against rapid motion, unreliable for large-scale rotation and deformation, lack sufficient feature points, and are prone to loss of tracking when the target is occluded. Algorithms suitable for embedded deployments, such as Naon and Siamese, do not have high enough accuracy, easily leading to misidentification and loss of the target. Therefore, current target tracking algorithms offer poor tracking performance and cannot meet practical tracking and shooting needs. Summary of the Invention

[0003] In view of this, the purpose of this application is to provide a target tracking method, imaging device and computer program product to improve the problem of poor target tracking and shooting effect in the prior art.

[0004] To address the aforementioned problems, in a first aspect, embodiments of this application provide a target tracking method, the method comprising: Select the initial region of the target to be tracked in the target frame; The target tracking model processes subsequent frames of the target frame based on the initial target region to obtain the location region of the target in the frame to be tracked; wherein, the target tracking model includes a template region feature extraction network and a search region feature extraction network; Based on the location area, the imaging device is controlled to track and capture images of the target to be tracked.

[0005] In the above implementation process, the initial target region of the target to be tracked can be determined first in the target frame. During tracking processing, a target tracking model containing a template region feature extraction network and a search region feature extraction network is used. Based on the initial target region, feature extraction and other processing are performed on subsequent frames to be processed in the target frame to determine the position region of the target to be tracked in each frame. Based on the determined position region, the imaging device is controlled to track and capture images of the target. This approach, which processes frames based on the target tracking model, effectively improves the accuracy and effectiveness of the position region under adverse conditions such as high-speed movement, deformation, and partial occlusion of the target, reducing the risk of target misidentification or loss, and improving the effect of target tracking and capturing, thus meeting the tracking and capturing needs of different targets in various scenarios.

[0006] Optionally, the step of processing subsequent frames to be processed based on the initial region of the target using a target tracking model to obtain the location region of the target to be tracked in the frames to be processed includes: The template region feature extraction network extracts features based on the initial target region in the target frame to obtain template features. The search region feature extraction network is used to extract multiple region features from the frame to be processed, thereby obtaining multiple region features. Based on the template features and multiple region features, the location region of the target to be tracked in the frame to be processed is determined.

[0007] In the above implementation process, the target tracking model first uses a template region feature extraction network to extract features from the initial target region determined in the target frame, obtaining the corresponding template features. Then, a search region feature extraction network performs multi-region feature extraction on the frame to be processed, obtaining multiple region features. Based on the template features and multiple region features, the location region in the frame to be processed that may contain the target to be tracked is determined. The ability to use different feature extraction networks to extract features from different regions in different frames effectively improves the effectiveness of the finally determined location region.

[0008] Optionally, the target tracking model further includes a contrastive network; The step of determining the location region of the target to be tracked in the frame to be processed based on the template features and multiple region features includes: The similarity between the template features and each region feature is compared using the comparison network to determine the comparison score of each region feature. Based on the comparison score, the location region of the target to be tracked is determined in the frame to be processed.

[0009] In the above implementation process, to determine the final location region, the target tracking model can also include a contrast network. This network compares the similarity between the template features and each region feature to determine a contrast score for each region feature. A higher contrast score indicates a higher similarity between the region feature and the template feature, meaning a higher probability of the target appearing within that region. By using the contrast score, the location region with the highest probability of the target appearing can be determined from among multiple region features corresponding to the frame being processed. Determining the location region through similarity comparison between different region features further improves the effectiveness of the location region determination.

[0010] Optionally, the comparison score includes a confidence score and an auxiliary score; The confidence score is used to characterize the correlation between the regional features and the template features; The auxiliary score is used to characterize the loss status of the target to be tracked.

[0011] In the above implementation process, the comparison score can include a confidence score that represents the correlation between regional features and template features, and an auxiliary score that represents whether the target to be tracked is lost. The comparison score can be obtained by comprehensively evaluating from the perspectives of correlation and loss. It is applicable to tracking processing in various situations such as rapid movement, deformation and local occlusion of the target to be tracked, so as to improve the effectiveness of determining the location region based on the comparison score.

[0012] Optionally, the method further includes: Determine the working cycle of the template region feature extraction network; The template region feature extraction network updates the template features based on the latest frame in each working cycle.

[0013] In the above implementation process, considering the movement characteristics of the target being tracked, if the same template feature is used throughout the process, the accuracy of determining the location region will be low when the target moves or changes pose. Therefore, the working cycle of the template region feature extraction network can be pre-set. Within each working cycle, the template region feature extraction network can perform feature extraction processing on the latest frame to periodically update the template features, thereby ensuring the real-time performance and effectiveness of the template features. Regularly updating the template features improves the accuracy of the location region determined based on the template features and region features.

[0014] Optionally, the method further includes: The target tracking model is optimized based on a deep learning model architecture; The deep learning model includes a self-attention mechanism neural network.

[0015] In the above implementation process, considering that the tracking method of simply relying on a static template to match the target has poor tracking performance in cases of rapid target movement, deformation, and partial occlusion, a deep learning model architecture including a self-attention mechanism neural network can be used to optimize the target tracking model. This allows the target tracking model to continuously generate more robust target representations through progressive representation learning, which can enhance the consistency of target appearance and semantics, rather than simply replacing the template, thereby improving the tracking performance of the target tracking model.

[0016] Optionally, the template region feature extraction network and the search region feature extraction network include: a shallow feature layer and a deep feature layer.

[0017] In the above implementation process, both the template region feature extraction network and the search region feature extraction network can include corresponding shallow feature layers and deep feature layers. The shallow feature layer can implement a control mechanism based on the features of shallow appearance information to suppress appearance interference and enhance target-related features, thus solving the shallow feature problem. The deep feature layer can refine information layer by layer to solve the deep feature problem. By combining the shallow feature layer and the deep feature layer, the processing effect of feature extraction is effectively improved.

[0018] Optionally, the training process of the target tracking model includes: Determine the constraint coefficient that continuously restricts the region of the target to be tracked; Determine the constraint coefficients for target changes between multiple frames; Determine the learning rate for the changes in the target to be tracked; The initial tracking model is trained based on the constraint coefficients, the learning rate, and the constraint coefficients to obtain the target tracking model.

[0019] In the above implementation process, during the training of the target tracking model, considering the actual situation of changes in region size, a constraint coefficient can be determined to continuously limit the region of the target to be tracked. Considering the positional changes of the target to be tracked within different frames, a constraint coefficient can be determined to constrain the target changes across multiple frames. Considering the morphological changes of the target to be tracked within different frames, a learning rate can be determined to account for these changes. The initial tracking model is then jointly trained based on the constraint coefficients, learning rate, and constraint coefficients to obtain the corresponding target tracking model. This allows for training the model based on the actual situation of the target to be tracked, effectively reducing the adverse effects of mistracking or loss of tracking.

[0020] Optionally, controlling the imaging device to track and capture images of the target based on the location area includes: Based on the location region and the imaging parameters of the imaging device, the field of view deviation data of the imaging device is determined; wherein, the field of view deviation data includes pitch angle deviation and azimuth angle deviation; Based on the field deviation data, the operating parameters of the adjustment device in the imaging device are determined; The adjustment device is controlled to operate based on the operating parameters using a control algorithm; wherein, when the adjustment device operates, it adjusts the field of view of the imaging device.

[0021] In the above implementation process, after determining the location area of ​​the target to be tracked, since the target may not be at the center of the current field of view of the imaging device, in order to track and capture the target, the current field of view deviation data of the imaging device can be determined first based on the location area and the imaging parameters of the imaging device. Then, based on the field of view deviation data, the working parameters for adjusting the field of view of the imaging device are determined. A control algorithm then controls the adjustment device based on these working parameters to maintain the target to be tracked at the center of the image within the field of view of the imaging device, thus achieving target tracking. The ability to control the field of view adjustment of the imaging device according to the actual location of the area effectively improves the target tracking and capturing effect.

[0022] Optionally, selecting the initial target region of the target to be tracked in the target frame includes: Determine the target frame from multiple initial frames; Based on the target tracking requirements, the target to be tracked and the target initial region of the target to be tracked are determined among multiple targets in the target frame.

[0023] In the above implementation process, users can determine the target frame to begin tracking and shooting from multiple initial frames based on actual tracking and shooting needs. Then, based on the actual target tracking requirements, they can determine the target to be tracked and its initial region from one or more targets present in the target frame. This allows users to select targets and their corresponding regions based on their actual needs, meeting different tracking requirements in different application scenarios.

[0024] Secondly, embodiments of this application also provide an imaging device, the imaging device including a memory and a processor, the memory storing program instructions, and the processor executing the steps in any of the methods described in the first aspect above when running the program instructions.

[0025] Thirdly, embodiments of this application also provide a computer program product, the computer program product including a computer program / instruction, which, when executed by a processor, implements the steps in the method described in any of the first aspects above.

[0026] In summary, the embodiments of this application provide a target tracking method, imaging device, and computer program product, which can process the frame to be processed based on the target tracking model. This effectively improves the accuracy and effectiveness of the position area of ​​the target under adverse conditions such as high-speed movement, deformation, and partial occlusion, thereby reducing the adverse situations of target misidentification or target loss, improving the effect of target tracking and shooting, and meeting the tracking and shooting needs of different targets in various scenarios. Attached Figure Description

[0027] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 A flowchart illustrating the first target tracking method provided in this application embodiment; Figure 2 A detailed flowchart of step S200 provided for an embodiment of this application; Figure 3 A detailed flowchart of step S230 provided for an embodiment of this application; Figure 4 A flowchart illustrating the second target tracking method provided in this application embodiment; Figure 5 A flowchart illustrating the third target tracking method provided in this application embodiment; Figure 6 A detailed flowchart of step S300 provided for an embodiment of this application; Figure 7 This is a detailed flowchart of step S100 provided in an embodiment of this application. Detailed Implementation

[0029] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of the embodiments of this application.

[0030] Existing target tracking algorithms mainly employ correlation filtering series, optical flow, and feature point tracking. These algorithms are weak in resisting rapid motion, not robust to large-scale rotation and deformation, lack sufficient feature points, and are prone to loss of tracking when the target is occluded. Algorithms suitable for embedded deployment, such as Naon and Siamese, do not have high enough accuracy, easily leading to misidentification and loss of the target. Therefore, current target tracking algorithms have poor tracking performance and cannot meet the needs of practical tracking and shooting.

[0031] To address the aforementioned issues, embodiments of this application provide a target tracking method, imaging device, and computer program product. These methods can process frames based on target tracking models, effectively improving the accuracy and effectiveness of the location area under adverse conditions such as high-speed movement, deformation, and partial occlusion of the target to be tracked. This reduces the adverse situations of target misidentification or target loss, improves the effect of target tracking and shooting, and meets the tracking and shooting needs of different targets in various scenarios.

[0032] This application provides a target tracking method applied to an imaging device, such as a smart telescope or drone, which possesses both logical computing and imaging capabilities. The imaging device may include an image sensor for capturing images, an adjustment device for adjusting the field of view of the image sensor, and devices such as a memory and processor for controlling the image sensor's capture. The target tracking method provided in this application can be applied to any target tracking scenario under various shooting modes. The user selects the target to be tracked from the real-time viewing screen. Since the target to be tracked has uncertainties in terms of shape, size, brightness distribution, and background environment, existing target tracking algorithms based on fixed features or predefined models are difficult to apply. This application, however, can provide a more accurate and stable tracking effect.

[0033] For example, image sensors can include different types of modules, such as telephoto modules and wide-angle modules, to meet the shooting needs of different scenarios. A telephoto module can include one or more image sensors with long focal lengths. These image sensors can be various types of CMOS (Complementary Metal-Oxide-Semiconductor) sensors, CCD (Charge-Coupled Device) devices, etc., that can convert light signals into electrical signals to achieve the corresponding imaging function. A wide-angle module can be an image sensor with lower resolution but higher sensitivity and guiding capabilities, such as a highly sensitive monochrome CMOS sensor, which has higher light-gathering efficiency and a wider shooting angle.

[0034] For example, the adjustment device can be a drive motor, drive unit, or theodolite / equatorial mount that is connected to the image sensor and can control the movement and angle adjustment of the image sensor.

[0035] It should be noted that the processor can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a digital signal processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor.

[0036] It should be noted that the processor can be integrated inside the imaging device and electrically connected to the image sensor and adjustment device via a cable, or it can be located outside the imaging device, such as in a mobile phone, computer, or other device, and communicate with the image sensor and adjustment device via a network, Bluetooth, or other means.

[0037] The imaging device in this embodiment can be used to perform various steps in the target tracking methods provided in the embodiments of this application. The implementation process of the target tracking method is described in detail below through several embodiments.

[0038] Please see Figure 1 , Figure 1 This is a flowchart illustrating a first target tracking method provided in an embodiment of this application. The method may include steps S100-S300.

[0039] Step S100: Select the initial target region of the target to be tracked in the target frame.

[0040] Since tracking and shooting usually involves long-term shooting, the target frame can be selected from the multiple captured images, and the initial target area where the target to be tracked is located can be determined in the target frame.

[0041] Optionally, the target to be tracked can be different targets in different shooting scenarios. For example, it can be celestial bodies such as the sun and moon in astronomical shooting scenarios, or various targets such as birds, people, vehicles, and flying objects in other shooting scenarios. The initial target region can be the region where the target to be tracked is located on the target frame image, which can be a corresponding rectangular bounding box region.

[0042] Optionally, the target to be tracked can be determined in the target frame by user selection, and the outline of the target to be tracked can be identified and determined by image recognition algorithm to determine the corresponding initial target area.

[0043] Step S200: Using the target tracking model, based on the initial target region, process the subsequent frames to be processed in the target frame to obtain the position region of the target to be tracked in the frames to be processed.

[0044] The target tracking model includes a template region feature extraction network and a search region feature extraction network. During tracking, the target tracking model, which includes the template region feature extraction network and the search region feature extraction network, is used to perform feature extraction and other processing on subsequent frames to be processed based on the initial target region, thereby determining the location region of the target to be tracked in each frame to be processed.

[0045] Optionally, the target tracking model can extract features from different regions in different frames of images to determine the region where the target to be tracked is most likely to appear in subsequent frames to be processed, i.e., the location region.

[0046] Optionally, the subsequent frames to be processed are multiple consecutive image frames following the target frame. The target tracking model can extract high-dimensional feature representations for different regions of different frames using a deep neural network and perform matching in the feature space, thereby achieving the tracking of the overall structural features of the target. Compared with existing tracking algorithms that rely solely on brightness differences or fixed handcrafted features, this application can maintain continuous tracking of the target even when its appearance changes, brightness fluctuates, or background interference increases, obtaining a location region with higher accuracy.

[0047] Step S300: Based on the location area, control the imaging device to track and capture images of the target to be tracked.

[0048] In order to track and photograph the target, the imaging device can be moved based on a defined location area.

[0049] exist Figure 1 In the illustrated embodiment, the frame to be processed can be processed based on the target tracking model, which effectively improves the accuracy and effectiveness of the position area under adverse conditions such as high speed movement, deformation, and partial occlusion of the target to be tracked, thereby reducing the adverse situation of target misidentification or target loss, improving the effect of target tracking and shooting, and meeting the tracking and shooting needs of different targets in various scenarios.

[0050] Optionally, please refer to Figure 2 , Figure 2The following is a detailed flowchart of step S200 provided in an embodiment of this application. Step S200 may include steps S210-S230.

[0051] Step S210: Using a template region feature extraction network, feature extraction is performed based on the initial target region in the target frame to obtain template features.

[0052] In the target tracking model, the initial target region in the target frame is first extracted using a template region feature extraction network to obtain the corresponding template features.

[0053] Optionally, the template features can be feature maps of the initial target region, which can represent low-level visual features such as color features, texture features, and contour features of the initial target region, as well as deep learning features such as fully connected layer features, convolutional layer features, and self-attention features.

[0054] Step S220: The frame to be processed is subjected to multi-region feature extraction by searching the region feature extraction network to obtain multiple region features.

[0055] In this process, a search region feature extraction network can be used to perform multi-region feature extraction on the frame to be processed, thereby obtaining multiple corresponding region features.

[0056] Optionally, the frame to be processed can be divided into multiple search regions. The region features can include the feature maps of the corresponding search regions, which can represent the low-level visual features such as color features, texture features, and contour features of the search regions, as well as deep learning features such as fully connected layer features, convolutional layer features, and self-attention features.

[0057] Step S230: Based on template features and multiple region features, determine the location region of the target to be tracked in the frame to be processed.

[0058] Among them, based on template features and multiple region features, the location regions in the frame to be processed that may contain the target to be tracked can be determined.

[0059] Optionally, the location region is not the coordinates of a single pixel, but rather a rectangular region. For example, the location region can be set as a rectangular bounding box, the position and size of which are determined by the coordinates of the center point of the bounding box in the image coordinate system and the size parameters.

[0060] exist Figure 2 In the illustrated embodiment, different feature extraction networks can be used to extract features from different regions in different frames, effectively improving the effectiveness of the finally determined location regions.

[0061] Optionally, to determine the final location region, the target tracking model may also include a contrastive network; see [link to relevant documentation]. Figure 3 , Figure 3 The following is a detailed flowchart of step S230 provided in an embodiment of this application. Step S230 may include steps S231-S232.

[0062] Step S231: By comparing the similarity between the template features and each region feature through the comparison network, the comparison score of each region feature is determined.

[0063] In this process, a contrast network compares the similarity between the template features and each region feature to determine the contrast score for each region feature.

[0064] It should be noted that the higher the comparison score, the higher the similarity between the features of the region and the template features, that is, the higher the probability that the target to be tracked appears in the region.

[0065] Optionally, the contrast score may include a confidence score and an auxiliary score. The confidence score characterizes the correlation between the region features and the template features, while the auxiliary score characterizes the loss status of the target being tracked. The contrast score can include a confidence score that characterizes the degree of correlation between the region features and the template features, and an auxiliary score that characterizes whether the target being tracked is lost. This allows for a comprehensive evaluation from both correlation and loss perspectives, making it suitable for tracking in various situations such as rapid movement, deformation, and partial occlusion of the target, thereby improving the effectiveness of determining the location region based on the contrast score.

[0066] For example, the mean of the sum of the confidence score and the auxiliary score can be calculated as the corresponding contrast score.

[0067] It should be noted that, considering the significant changes that can occur when a tracked target is lost, an auxiliary score is set to balance the final comparison score when the target is lost. The auxiliary score is higher when the target is not lost and extremely low when the target is lost.

[0068] Step S232: Based on the comparison score, determine the location region of the target to be tracked in the frame to be processed.

[0069] By comparing scores, the region with the highest probability of the target being tracked can be determined from multiple regional features corresponding to the frame to be processed.

[0070] Optionally, a comparison score can be calculated between the region features and template features corresponding to all search regions on the frame to be processed. The search region with the highest comparison score (above a preset score threshold) is selected as the corresponding location region. The score threshold can be set according to actual conditions and needs to improve the effectiveness of the location region and reduce the adverse effects of target misidentification or loss.

[0071] It should be noted that in the process of comparing similarity in the comparison network, the matching method between template features and region features is not limited to correlation calculation. Convolution matching and other methods can also be used to calculate, treating the template feature as a "convolution kernel" and sliding it on multiple search features to calculate whether each position "looks like the template".

[0072] exist Figure 3 In the illustrated embodiment, the location region can be determined by comparing the similarity between different regional features, which further improves the effectiveness of the location region.

[0073] Optionally, please refer to Figure 4 , Figure 4 This is a flowchart illustrating a second target tracking method provided in an embodiment of this application. The method may further include steps S240-S250.

[0074] Step S240: Determine the working cycle of the template region feature extraction network.

[0075] Considering the movement characteristics of the target to be tracked, if the same template feature is used for processing throughout the process, the accuracy of determining the location region will be low when the target moves or changes pose. Therefore, the working cycle of the template region feature extraction network can be preset.

[0076] Optionally, the corresponding working cycle can be set according to the actual shooting situation and needs, such as taking 10 frames as one cycle.

[0077] Step S250: The template features are updated based on the latest frame in each working cycle through the template region feature extraction network.

[0078] In each working cycle, the template region feature extraction network can perform feature extraction processing on the latest frame to periodically update the template features, thereby ensuring the real-time performance and effectiveness of the template features.

[0079] Optionally, the latest frame can be the current frame to be processed when the working cycle is reached. The template region feature extraction network can continue to process the latest frame. The processing method is similar to that of the target frame, and will not be described in detail here.

[0080] Optionally, in order to reduce computation and improve the timeliness of target tracking, each subsequent frame to be processed within a working cycle determines the corresponding location region based on the same template features.

[0081] Optionally, the template region feature extraction network can adaptively update template features between consecutive frames, enabling the target tracking model to dynamically adjust according to changes in the appearance of the target being tracked, thereby avoiding tracking failures caused by changes in the target's shape or short-term occlusion. It can maintain stable representations when the target deforms or changes in appearance, reducing template feature drift and directly improving continuous localization accuracy. In scenarios with short-term occlusion or rapid displacement, relying on full-screen similarity search and a stacked candidate re-identification mechanism can significantly improve the re-capture rate of the target being tracked.

[0082] exist Figure 4 In the illustrated embodiment, the template features can be updated periodically, thereby improving the accuracy of the location region determined based on the template features and region features.

[0083] It should be noted that, considering the poor tracking performance of simply relying on a static template to match the target under conditions of rapid target movement, deformation, or partial occlusion, the method also includes: optimizing the target tracking model based on a deep learning model architecture; wherein the deep learning model includes a self-attention neural network. The target tracking model can be optimized using a deep learning model architecture including a self-attention neural network, enabling the target tracking model to continuously generate more robust target representations through progressive representation learning, strengthening the consistency of target appearance and semantics, rather than simply replacing templates, thereby improving the tracking performance of the target tracking model.

[0084] For example, a deep learning model architecture can include a deep learning algorithm capable of target tracking, such as the PRL-track deep learning algorithm. PRL-Track only updates the position state, not the template. The core idea of ​​PRL-Track is to make "similarity measurement independent of appearance details" without requiring changes to template features. This is achieved primarily through: an Appearance-aware Regulator (AR) to suppress shallow appearance noise, changes in lighting, color, and texture; a Semantic-aware Regulator (SR) to enhance semantic consistency, addressing what something is and pose changes; and a Hierarchical Modeling Generator (HMG) to perform cross-layer semantic alignment, independent of any single scale or local region. This method exhibits higher stability in complex scenarios such as occlusion and interference from similar targets, and is more suitable for real-time deployment on computationally limited UAV platforms. Alternatively, Nano series networks, Siamese network families, or a hybrid strategy of "deep learning + correlation filtering" can be used (deep features for feature extraction, and correlation filtering for efficient search).

[0085] Optionally, a deep convolutional network based on a deep learning model architecture can be used to perform end-to-end representation of the target to be tracked, which can maintain stable target localization accuracy under conditions such as rapid movement, deformation, and partial occlusion of the target.

[0086] For example, a deep learning model architecture can be a lightweight Transformer module, such as a combination of a VisionTransformer (ViT) structure and a CNN (Convolutional Neural Network). This combination can further fuse multiple types of information to obtain a more stable and discriminative target representation. It can align and fuse global relationships between multi-layer features, thereby significantly improving tracking stability and recapture capability under conditions of occlusion, scale changes, and deformation. Since CNNs lack the ability to globally model long-range dependencies and struggle to capture global contextual information between multi-scale features, ViT's global modeling ability shows an advantage in handling appearance changes (such as aspect ratio changes). Compared to CNNs, ViT tends to ignore local spatial information. To fully utilize both global contextual information and local spatial information, CNNs and ViT can be organically combined to address target appearance changes, occlusion, deformation, and other situations, thereby improving tracking stability and robustness.

[0087] Optionally, the Transformer structure is not limited to using a hierarchical cross-attention approach. It can also use self-attention networks, lightweight attention networks, graph neural networks, or feature fusion modules based on fully connected mappings to achieve relationship modeling and information fusion between multi-layer features.

[0088] It should be noted that the template region feature extraction network and the search region feature extraction network include: shallow feature layers and deep feature layers. The shallow feature layer can implement a control mechanism based on shallow appearance information to suppress appearance interference and enhance target-related features, thus solving the problem of shallow features. The deep feature layer can refine information layer by layer to solve the problem of deep features. By combining the shallow and deep feature layers, the processing effect of feature extraction is effectively improved.

[0089] Optionally, the shallow and deep feature layers are not limited to being implemented using convolutional neural networks. They can also be modulated and enhanced using depthwise separable convolution, dilated convolution, attention mechanisms, or multi-branch feature fusion structures to extract the appearance and semantic information of the target to be tracked.

[0090] For example, the shallow feature layer can be denoted as R1, which can suppress shallow appearance noise, illumination changes, color changes, and texture changes; the deep feature layer can be denoted as R2, which can enhance semantic consistency, including what the target is and pose changes. Semantic alignment can also be performed on the shallow and deep feature layers to prevent the target tracking model from relying solely on a single scale or local region.

[0091] For example, a deep learning model architecture can incorporate multiple algorithms capable of target tracking, including UAV algorithms or various improved tracking algorithms, such as filtering algorithms like CSK, KCF, DCF, SRDCF, STRCF, and ARCF, or deep learning tracking algorithms like the Siamese series and Transformer series. In this application, shallow appearance noise, illumination changes, color changes, and texture changes can be suppressed through R1, while semantic consistency is enhanced through R2. Pose changes are aligned across layers using a lightweight Transformer module, independent of any single scale or local region. This method exhibits higher stability in complex scenarios such as occlusion and interference from similar targets, and is more suitable for real-time deployment on computationally limited UAV platforms. Nano series sensors can also be used, or a hybrid strategy of "deep learning + correlation filtering" can be employed (deep features for feature extraction, and correlation filtering for efficient search).

[0092] Optionally, the deep learning tracking algorithm used in this application can update template features based on the deformation of the target to be tracked, and more accurately identify the target to be tracked when the target moves quickly or is occluded, and control the rotation of the imaging device to achieve stable tracking and shooting.

[0093] Optionally, please refer to Figure 5 , Figure 5 This is a flowchart illustrating a third target tracking method provided in an embodiment of this application. The method may further include steps S411-S414.

[0094] Step S411: Determine the constraint coefficient for continuously restricting the area of ​​the target to be tracked.

[0095] In the training process of the target tracking model, considering the actual situation of changes in the region size, a constraint coefficient can be determined to continuously restrict the region of the target to be tracked.

[0096] Optionally, the constraint factor can be set to 0.04-0.08. The constraint factor can be used to penalize candidate boxes whose size or aspect ratio changes drastically. Increasing the constraint factor makes the approach more conservative, suppressing size changes, which meets the actual needs of astronomical photography scenarios where the target does not change much, resulting in more conservative tracking. In other application scenarios, its value can be adjusted according to the actual situation.

[0097] Step S412: Determine the constraint coefficients for target changes between multiple frames.

[0098] In particular, considering the positional changes of the target to be tracked within different frame images, the constraint coefficients for the target changes between multiple frame images can be determined.

[0099] Optionally, the constraint coefficient is the spatial smoothing (center constraint) coefficient, which controls the constraint strength of the previous frame position of the target to be tracked on the current predicted position. It can be set to 0.4-0.6. Increasing the constraint coefficient will make it more stable and reduce the adverse situation of the target jumping around. It meets the actual needs of the actual situation in astronomical shooting scenarios where the target movement distance is not large, and the tracking is more conservative. In other application scenarios, its value can be adjusted according to the actual situation.

[0100] Step S413: Determine the learning rate for tracking changes in the target.

[0101] In particular, considering the morphological changes of the target to be tracked in different frame images, the learning rate of the changes of the target to be tracked can be determined.

[0102] Optionally, the learning rate can be set to 0.2-0.3. A lower learning rate will result in greater stability and a slower response to changes in appearance, but it is less prone to drift. This meets the practical needs of astronomical photography scenarios where the target shape does not change much, and the tracking is more conservative. In other application scenarios, its value can be adjusted according to the actual situation.

[0103] It should be noted that there is no restriction on the execution order of steps S411-S413; the three steps can occur simultaneously or sequentially based on the data acquisition time.

[0104] Step S414: Train the initial tracking model based on the constraint coefficient, the learning rate, and the limiting coefficient to obtain the target tracking model.

[0105] The initial tracking model can be jointly trained based on the constraint coefficient, the learning rate, and the limiting coefficient to obtain the corresponding target tracking model.

[0106] For example, the initial tracking model can be a deep learning model with a tracking algorithm.

[0107] Optionally, the constraint coefficient, learning rate, and other factors can be combined for overall parameter tuning to balance false tracking and missing tracking while prioritizing avoiding false tracking in the target update model.

[0108] Optionally, a corresponding loss frame value can be set in the target tracking model for limitation. For example, if the loss frame value is set to 10, the target to be tracked is considered to have disappeared when the target is lost for 10 consecutive frames, and the corresponding failure prompt message can be returned. If the target to be tracked reappears within 10 frames, tracking continues.

[0109] exist Figure 5In the embodiments shown, the model can be trained according to the actual situation of the target to be tracked, which can effectively reduce the adverse situation of the target tracking model generating false tracking or loss of tracking.

[0110] Optionally, please refer to Figure 6 , Figure 6 This is a detailed flowchart of step S300 provided in an embodiment of the present application. Step S300 may include steps S310-S330.

[0111] Step S310: Determine the field of view deviation data of the imaging device based on the location area and the imaging parameters of the imaging device.

[0112] In order to track and capture images of the target, after determining the location area of ​​the target, since the target may not be at the center of the current field of view of the imaging device, the current field of view deviation data of the imaging device can be determined based on the location area and the imaging parameters of the imaging device.

[0113] Optionally, the field of view deviation data includes pitch angle deviation and azimuth angle deviation. Pitch angle deviation refers to the deviation in pitch angle of the imaging device's field of view, and azimuth angle deviation refers to the deviation in azimuth angle of the imaging device's field of view.

[0114] It should be noted that the processor may have an attitude calculation module. This module can calculate the field of view deviation data based on the position area output by the target tracking model and the current attitude of the imaging device. Specific calculation methods may include: converting the target offset into an offset value in the field of view of the imaging device based on the image sensor's imaging size and lens focal length; then constructing the deviations of the pitch and azimuth angles to obtain the corresponding field of view deviation data.

[0115] Step S320: Based on the field deviation data, determine the operating parameters of the adjustment device in the imaging equipment.

[0116] Among them, the working parameters for adjusting the field of view of the imaging device can be determined based on the field deviation data.

[0117] Optionally, the operating parameters can be parameters such as the adjustment range and direction of the adjustment device.

[0118] Step S330: The control algorithm controls and adjusts the device to operate based on the working parameters.

[0119] Specifically, the adjustment device adjusts the field of view of the imaging device during operation. A control algorithm can be used to control the adjustment device based on operating parameters, thereby ensuring that the target to be tracked remains centered within the field of view of the imaging device, achieving target tracking.

[0120] Optionally, the processor can also include a gimbal control module. This module can perform closed-loop control based on operating parameters, using PID or adaptive control algorithms to drive adjustment devices, ensuring the target being tracked is always centered within the imaging device's field of view. Pixel-level deviations are directly converted into angular deviations and fed back into the control: pixel-to-angle conversion + PID / adaptive closed-loop control allows the imaging device to be aligned with the center of the target in real time, fundamentally reducing tracking failures caused by imaging device attitude errors.

[0121] For example, model predictive control (MPC), adaptive control, or state estimation plus feedforward control based on Kalman filtering can also be used to control the operation of the adjustment device.

[0122] exist Figure 6 In the illustrated embodiment, the imaging device can be controlled to adjust the field of view according to the actual position of the location area, which effectively improves the effect of tracking and shooting the target.

[0123] Optionally, please refer to Figure 7 , Figure 7 This is a detailed flowchart of step S100 provided in an embodiment of the present application. Step S100 may include steps S110-S120.

[0124] Step S110: Determine the target frame among multiple initial frames.

[0125] Users can determine the target frame to begin tracking and shooting from among multiple initial frames based on their actual tracking and shooting needs.

[0126] For example, the tracking shooting requirement can be a corresponding time requirement. For instance, if a user wants to perform tracking shooting at a certain moment, the tracking shooting requirement can be the corresponding time point, and the corresponding target frame is an image captured at the corresponding time point, or the image captured at the time closest to the time point.

[0127] Step S120: Based on the target tracking requirements, determine the target to be tracked and the initial target region of the target to be tracked among multiple targets in the target frame.

[0128] Based on actual target tracking requirements, the target to be tracked and the initial target region where the target to be tracked are located can be determined from one or more targets present in the target frame.

[0129] Optionally, target tracking can be performed by the user clicking on the selected target.

[0130] exist Figure 7In the illustrated embodiment, the target and its corresponding location can be selected based on the user's actual needs, meeting the different tracking needs of users in different application scenarios.

[0131] For example, when the scene depicts a pedestrian walking behind a car, being obscured, and then reappearing and starting to run, a viewing mode can be used. The pedestrian is selected as the target to be tracked, and the initial target region is determined. The bounding box of this initial target region is then scaled to a 127x127 template image and fed into the template region feature extraction network. Subsequently, the search region in the frames to be processed is scaled to 287x287 and fed into the search region feature extraction network. The design uses the AlexNet CNN network, but adds shallow and deep feature layers (R1 and R2). A lightweight Transformer structure is then used to align and fuse the global relationships of the multi-layer features. The R1 module can suppress changes in the person's appearance. The system addresses the impact of time on tracking, suppressing background texture interference and improving stable responses to target edges and shapes, thus reducing the likelihood of mistracking other pedestrians with similar appearances. The R2 module maintains semantic consistency even after the target is occluded, making it less prone to loss during scale / viewpoint changes. For instance, if a pedestrian is occluded by a vehicle and then reappears with a running form instead of walking, it is less likely to be lost. CNNs have limited receptive fields, and local information disappears after occlusion. In this case, the Transformer enhances global information, further improving resistance to occlusion and loss. Even when a pedestrian is briefly and completely occluded, tracking can resume after their reappearance. The obtained template features and region features are fed into a similarity comparison network for similarity comparison, obtaining corresponding comparison scores, and the location region is determined based on the comparison scores.

[0132] This application also provides a computer program product, which includes a computer program / instructions. When the computer program / instructions are executed by a processor, they implement the steps in the target tracking method described in any of the above embodiments.

[0133] In summary, this application addresses the problem of existing technologies' inability to effectively handle shooting requirements in scenarios involving "weakly textured targets + rapid movement + deformation + occlusion" by introducing deep learning target representation, template self-updating mechanisms, and pose closed-loop control. This significantly improves the target tracking stability and response speed of astronomical cameras in handheld or gimbaled scenarios. The aforementioned deep learning-based target tracking mechanism can be introduced into viewing mode, enabling stable tracking control of any user-selected target. This ensures that the imaging device keeps the target centered in the field of view even in complex observation environments, thereby significantly enhancing the continuity of the viewing process and the user experience.

[0134] In the several embodiments provided in this application, it should be understood that the disclosed device can also be implemented in other ways. The device embodiments described above are merely illustrative; for example, the block diagrams in the accompanying drawings illustrate the possible architecture, functions, and operations of the device according to various embodiments of this application. In this regard, each block in the block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram, and combinations of block diagrams, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0135] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0136] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0137] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0138] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

[0139] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A target tracking method, characterized in that, The method includes: Select the initial region of the target to be tracked in the target frame; The target tracking model processes subsequent frames of the target frame based on the initial target region to obtain the location region of the target in the frame to be tracked; wherein, the target tracking model includes a template region feature extraction network and a search region feature extraction network; Based on the location area, the imaging device is controlled to track and capture images of the target to be tracked.

2. The method according to claim 1, characterized in that, The step of processing subsequent frames of the target frame based on the initial region of the target using a target tracking model to obtain the location region of the target to be tracked in the frames of the target frame includes: The template region feature extraction network extracts features based on the initial target region in the target frame to obtain template features. The search region feature extraction network is used to extract multiple region features from the frame to be processed, thereby obtaining multiple region features. Based on the template features and multiple region features, the location region of the target to be tracked in the frame to be processed is determined.

3. The method according to claim 2, characterized in that, in, The target tracking model also includes a contrast network; The step of determining the location region of the target to be tracked in the frame to be processed based on the template features and multiple region features includes: The similarity between the template features and each region feature is compared using the comparison network to determine the comparison score of each region feature. Based on the comparison score, the location region of the target to be tracked is determined in the frame to be processed.

4. The method according to claim 3, characterized in that, in, The comparison scores include confidence scores and auxiliary scores; The confidence score is used to characterize the correlation between the regional features and the template features; The auxiliary score is used to characterize the loss status of the target to be tracked.

5. The method according to claim 2, characterized in that, The method further includes: Determine the working cycle of the template region feature extraction network; The template region feature extraction network updates the template features based on the latest frame in each working cycle.

6. The method according to any one of claims 1-5, characterized in that, The method further includes: The target tracking model is optimized based on a deep learning model architecture; The deep learning model includes a self-attention mechanism neural network.

7. The method according to any one of claims 1-5, characterized in that, The template region feature extraction network and the search region feature extraction network include: a shallow feature layer and a deep feature layer.

8. The method according to any one of claims 1-5, characterized in that, The training process of the target tracking model includes: Determine the constraint coefficient that continuously restricts the region of the target to be tracked; Determine the constraint coefficients for target changes between multiple frames; Determine the learning rate for the changes in the target to be tracked; The initial tracking model is trained based on the constraint coefficients, the learning rate, and the constraint coefficients to obtain the target tracking model.

9. The method according to any one of claims 1-5, characterized in that, The step of controlling the imaging device to track and capture images of the target based on the location region includes: Based on the location region and the imaging parameters of the imaging device, the field of view deviation data of the imaging device is determined; wherein, the field of view deviation data includes pitch angle deviation and azimuth angle deviation; Based on the field deviation data, the operating parameters of the adjustment device in the imaging device are determined; The adjustment device is controlled to operate based on the operating parameters using a control algorithm; wherein, when the adjustment device operates, it adjusts the field of view of the imaging device.

10. The method according to any one of claims 1-5, characterized in that, The step of selecting the initial target region of the target to be tracked in the target frame includes: Determine the target frame from multiple initial frames; Based on the target tracking requirements, the target to be tracked and the target initial region of the target to be tracked are determined among multiple targets in the target frame.

11. An imaging device, characterized in that, The imaging device includes a memory and a processor. The memory stores program instructions, and when the processor runs the program instructions, it performs the steps of the method according to any one of claims 1-10.

12. A computer program product, characterized in that, The computer program product includes a computer program / instruction that, when executed by a processor, implements the steps of the method according to any one of claims 1-10.

Images