Dynamic target tracking method and system based on night vision scope image processing

Abstract

The application provides a dynamic target tracking method and system based on night vision scope image processing, and relates to the technical field of image processing.The method comprises the following steps: synchronously collecting night vision images and thermal imaging images at the same time by using a night vision scope, performing image time alignment and pixel-level registration, performing image preprocessing and multi-modal feature fusion on the night vision image sequence and the thermal imaging image sequence, calling a pre-constructed target tracking model, calling a first prompt vector or a second prompt vector of the target tracking model, jointly tracking the target of the night vision image sequence and the thermal imaging image sequence, and obtaining a target tracking trajectory.The technical problem of unstable dynamic target perception and low tracking precision caused by the lack of multi-modal information fusion and time sequence feature analysis in the prior art under low-illumination environment is solved.The technical effect of stable perception and continuous tracking of dynamic targets under low-illumination environment is achieved, and the stability and accuracy of dynamic target tracking are effectively improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically to a dynamic target tracking method and system based on night vision sight image processing. Background Technology

[0002] Existing target tracking technologies based on night vision image processing mainly rely on a single imaging modality, such as visible light night vision imaging or infrared thermal imaging. Visible light night vision imaging suffers significant degradation in image contrast and detail under conditions of insufficient light, strong noise, or smoke obstruction, easily leading to blurred target outlines and unstable features. While single infrared thermal imaging offers some resistance to light interference, it is prone to false detections when the temperature difference between the target and background is small, environmental thermal noise is high, or the target's attitude changes significantly. Furthermore, most existing tracking algorithms primarily process each frame independently, lacking effective modeling of the evolutionary trends of temporal features. Under conditions of rapid target movement, short-term occlusion, or enhanced background interference, this can easily lead to tracking interruptions or amplified cumulative errors, thus affecting the accuracy and overall reliability of target tracking.

[0003] Existing technologies lack multimodal information fusion and temporal feature analysis in low-light environments, leading to technical problems such as unstable dynamic target perception and low tracking accuracy. Summary of the Invention

[0004] The purpose of this application is to provide a dynamic target tracking method and system based on night vision sight image processing, which solves the technical problems of existing technologies lacking multimodal information fusion and temporal feature analysis in low-light environments, resulting in unstable dynamic target perception and low tracking accuracy.

[0005] In view of the above problems, this application provides a dynamic target tracking method and system based on night vision aiming scope image processing.

[0006] The first aspect of this application provides a dynamic target tracking method based on night vision scope image processing. This method includes: simultaneously acquiring night vision images and thermal imaging images using the night vision detector and thermal imaging detector of the night vision scope within a preset window; performing image temporal alignment and pixel-level registration according to a pre-calibrated spatial mapping relationship to obtain a night vision image sequence and a thermal imaging image sequence; traversing the night vision image sequence and thermal imaging image sequence to perform image preprocessing and multimodal feature fusion to obtain a multimodal fusion feature map; calling a pre-constructed target tracking model, and based on the multimodal fusion feature map, calling a first cue vector or a second cue vector of the target tracking model to perform joint target tracking on the night vision image sequence and the thermal imaging image sequence to obtain a target tracking trajectory.

[0007] Optionally, the night vision detector and the thermal imaging detector are calibrated using internal parameters; after the internal parameter calibration is completed, the external parameter relationship between the night vision detector and the thermal imaging detector is obtained based on the infrared-visible light joint calibration board; and the external parameter relationship is used as the pre-calibration spatial mapping relationship.

[0008] Optionally, the night vision image sequence is subjected to low-light image enhancement preprocessing to obtain an enhanced night vision image sequence; the thermal imaging image sequence is subjected to non-uniformity calibration and thermal noise suppression processing to obtain an enhanced thermal imaging image sequence; and the enhanced night vision image sequence and the enhanced thermal imaging image sequence are subjected to one-to-one mapping and multimodal feature temporal processing to obtain the multimodal fusion feature map.

[0009] Optionally, the night view image sequence is subjected to luminance separation processing based on the Retinex model to obtain an illuminance component sequence; the illuminance component sequence is subjected to adaptive contrast stretching, and random noise is suppressed by combining a frequency domain denoising algorithm, thereby obtaining an enhanced night view image sequence.

[0010] Optionally, the enhanced night vision image sequence and the enhanced thermal imaging image sequence are subjected to one-to-one mapping and joint multimodal feature processing to obtain an initial multimodal fusion feature map sequence; the first initial multimodal fusion feature map and the second initial multimodal fusion feature map in the initial multimodal fusion feature map sequence are subjected to sequential trend superposition processing to obtain a first superimposed multimodal fusion feature map; the first superimposed multimodal fusion feature map is used to perform sequential trend superposition processing on the third initial multimodal fusion feature map in the initial multimodal fusion feature map sequence to obtain a second superimposed multimodal fusion feature map, and so on, to obtain multimodal fusion feature maps.

[0011] Optionally, the first initial multimodal fusion feature map and the second initial multimodal fusion feature map are approximated from two dimensions: night vision features and thermal imaging features, respectively, to obtain the night vision feature similarity and thermal imaging feature similarity. The night vision feature similarity and thermal imaging feature similarity are normalized using the max-min normalization method, and the normalization result is filled into an initially empty two-dimensional diagonal matrix to construct a trend superposition matrix. The trend superposition matrix is ​​then used to superimpose the second initial multimodal fusion feature map to obtain the first superimposed multimodal fusion feature map.

[0012] Optionally, multiple historical multimodal fusion feature maps, along with corresponding multiple sample night vision image sequences, multiple sample thermal imaging image sequences, and multiple sample target tracking trajectories, are acquired as a training sample set; the multiple historical multimodal fusion feature maps are aggregated within the same category to obtain multiple aggregated historical multimodal fusion feature map sets; based on the size of the number of feature maps within each of the multiple aggregated historical multimodal fusion feature map sets, base class and new class data partitioning is performed to obtain a base class aggregated historical multimodal fusion feature map set and a new class aggregated historical multimodal fusion feature map set; according to the base class aggregated historical multimodal fusion feature map set and The novel aggregated historical multimodal fusion feature map set is used to map and divide the multiple sample night view image sequences, multiple sample thermal imaging image sequences, and multiple sample target tracking trajectories to obtain a base class data set and a novel class data set. The training sample set is used to train the model to obtain an initial target tracking model. The initial target tracking model is then trained using the base class data set and the novel class data set to obtain a first cue vector and a second cue vector. The first cue vector and the second cue vector are then stored in parallel in the semantic encoding module of the initial target tracking model to obtain the target tracking model.

[0013] Optionally, an initial cue vector is loaded from the semantic encoding module of the initial target tracking model as a common cue vector, and copied to generate a first initial cue vector and a second initial cue vector. Based on the first and second initial cue vectors, the initial target tracking model is trained using a base class dataset and a new class dataset to obtain base class model state parameters and optimized base class cue vectors, as well as new class model state parameters and optimized new class cue vectors. The optimized base class cue vectors are correlated using the base class model state parameters to obtain a first cue vector. The optimized new class cue vectors are correlated using the new class model state parameters to obtain a second cue vector.

[0014] Optionally, the base class dataset is input into the initial target tracking model. While keeping the main model parameters and the second initial cue vector parameters frozen and not updated, only the first initial cue vector is allowed to participate in backpropagation. After multiple rounds of optimization, the optimized base class cue vector and base class model state parameters are obtained. Similarly, the new class dataset is input into the initial target tracking model. While keeping the main model parameters and the first initial cue vector parameters frozen and not updated, only the second initial cue vector is allowed to participate in backpropagation. After multiple rounds of optimization, the optimized new class cue vector and new class model state parameters are obtained.

[0015] A second aspect of this application provides a dynamic target tracking system based on night vision scope image processing. The system comprises: an image sequence acquisition module, used to simultaneously acquire night vision images and thermal imaging images within a preset window using the night vision detector and thermal imaging detector of the night vision scope, and perform image temporal alignment and pixel-level registration according to a pre-calibrated spatial mapping relationship to obtain a night vision image sequence and a thermal imaging image sequence; an image sequence processing module, used to traverse the night vision image sequence and thermal imaging image sequence to perform image preprocessing and multimodal feature fusion to obtain a multimodal fusion feature map; and a joint target tracking module, used to call a pre-constructed target tracking model, and based on the multimodal fusion feature map, call the first or second cue vector of the target tracking model to perform joint target tracking on the night vision image sequence and thermal imaging image sequence to obtain a target tracking trajectory.

[0016] One or more technical solutions provided in this application have at least the following technical effects or advantages: The method provided in this application simultaneously acquires night vision images and thermal imaging images using a night vision scope's night vision detector and thermal imaging detector within a preset window. It then performs image temporal alignment and pixel-level registration based on a pre-calibrated spatial mapping relationship to obtain a night vision image sequence and a thermal imaging image sequence. The method iterates through the night vision image sequence and thermal imaging image sequence, performing image preprocessing and multimodal feature fusion to obtain a multimodal fusion feature map. Finally, it calls a pre-constructed target tracking model, invoking either the first or second cue vector of the target tracking model based on the multimodal fusion feature map, to jointly track the target from the night vision image sequence and the thermal imaging image sequence, obtaining the target tracking trajectory. Through precise calibration and alignment of night vision and thermal imaging multimodal images, targeted enhancement preprocessing, and a temporal trend overlay fusion mechanism based on feature similarity, the method achieves stable perception and continuous tracking of dynamic targets in low-light environments, effectively improving the stability and accuracy of dynamic target tracking.

[0017] The above description is merely an overview of the technical solution of this application. To better understand the technical means of this application and to facilitate its implementation according to the description, and to make the above and other objects, features, and advantages of this application more apparent, specific embodiments of this application are described below. It should be understood that the content described in this section is not intended to identify key or important features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent through the following description. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely exemplary. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the dynamic target tracking method based on night vision scope image processing provided in this application.

[0020] Figure 2 A schematic diagram of the dynamic target tracking system based on night vision scope image processing provided in this application.

[0021] Figure labeling: Image sequence acquisition module 11, image sequence processing module 12, joint target tracking module 13. Detailed Implementation

[0022] This application provides a dynamic target tracking method and system based on night vision sight image processing. It addresses the technical problems of existing technologies lacking multimodal information fusion and temporal feature analysis in low-light environments, leading to unstable dynamic target perception and low tracking accuracy. Through precise calibration and alignment of night vision and thermal imaging multimodal images, targeted enhancement preprocessing, and a temporal trend overlay fusion mechanism based on feature similarity, stable perception and continuous tracking of dynamic targets in low-light environments are achieved, effectively improving the stability and accuracy of dynamic target tracking.

[0023] The technical solutions of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. It should be understood that the present invention is not limited to the exemplary embodiments described herein. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention. It should also be noted that, for ease of description, only the parts related to the present invention are shown in the accompanying drawings, not all of them.

[0024] Example 1, as Figure 1 As shown, this application provides a dynamic target tracking method based on night vision sight image processing, the dynamic target tracking method based on night vision sight image processing includes: Within a preset window, the night vision detector and thermal imaging detector of the night vision scope are used to simultaneously acquire night vision images and thermal imaging images at the same time. Based on the pre-calibrated spatial mapping relationship, image time alignment and pixel-level registration are performed to obtain night vision image sequences and thermal imaging image sequences.

[0025] Furthermore, image time alignment and pixel-level registration are performed based on the pre-calibrated spatial mapping relationship to obtain night vision image sequences and thermal imaging image sequences, including: performing intrinsic parameter calibration on the night vision detector and the thermal imaging detector; after completing the intrinsic parameter calibration, obtaining the extrinsic parameter relationship between the night vision detector and the thermal imaging detector based on the infrared-visible light joint calibration board; and using the extrinsic parameter relationship as the pre-calibrated spatial mapping relationship.

[0026] Specifically, within a preset window time range, a unified hardware trigger signal is used to control the night vision detector and thermal imaging detector integrated in the night vision scope to complete image acquisition simultaneously, thereby obtaining night vision images and thermal imaging images with consistent timestamps. Synchronous acquisition avoids cross-modal time shifts caused by target movement or equipment jitter. The preset window can be flexibly set according to actual application scenarios and needs, such as a window period of 10 images per second, to ensure timely capture of dynamic changes in the target. The night vision detector mainly utilizes low-light detection technology to convert weak light signals into electrical signals, thus presenting a visible night vision image. It can capture the reflection information of the target in the visible light band. The thermal imaging detector, based on the principle of thermal radiation, generates a thermal imaging image by detecting the infrared radiation emitted by the object itself, reflecting the temperature distribution of the target.

[0027] Intrinsic parameter calibration was performed on both the night vision detector and the thermal imaging detector. This calibration employed an imaging model calibration method based on a planar calibration plate. A standard checkerboard or dot array calibration plate was placed sequentially at different distances and orientations, allowing the detector to image the calibration plate from multiple perspectives and acquire multiple sets of calibration images. For the night vision detector, corner points or feature points of the calibration plate were extracted from the visible light image. Combined with a pinhole camera imaging model, a least-squares optimization algorithm was used to solve for the focal length parameters, principal point coordinates, and radial and tangential distortion coefficients of the imaging device. For the thermal imaging detector, high-low temperature contrast areas were imaged on the calibration plate surface. Corresponding thermal feature points were extracted from the infrared image, and the same imaging model and optimization method as the night vision detector were used to calculate its intrinsic parameters. This yielded accurate intrinsic parameters for both the night vision detector and the thermal imaging detector. This intrinsic parameter calibration was used to determine the focal length, principal point position, and radial and tangential distortion parameters of each imaging device to eliminate the influence of lens distortion on pixel position mapping accuracy.

[0028] After completing the intrinsic parameter calibration, an infrared-visible light joint calibration board is introduced to perform extrinsic parameter calibration on the night vision detector and the thermal imaging detector. The infrared-visible light joint calibration board is a calibration tool whose surface simultaneously displays visible light and infrared feature patterns. The night vision detector and the thermal imaging detector image the joint calibration board respectively. By extracting the corresponding feature points of the calibration board in the night vision image and the thermal imaging image, and combining them with their respective intrinsic parameter models, the two-dimensional image feature points are back-projected into the three-dimensional calibration board coordinate system. Then, a pose solution method based on spatial point set registration is used to calculate the relative rotation matrix and translation vector between the night vision detector and the thermal imaging detector, thereby obtaining the extrinsic parameter relationship describing the spatial relationship between the two imaging coordinate systems. This extrinsic parameter relationship is used as the pre-calibration spatial mapping relationship. For example, let P be the three-dimensional coordinates of the feature points on the calibration board in the world coordinate system. w Its representation in the coordinate systems of night vision detectors and thermal imaging detectors is as follows: P nv =R nv P w +t nv P ir =R ir P w +t ir , where R nv t nv Let R be the rotation matrix and translation vector of the night vision detector relative to the calibration plate. ir t ir Let be the rotation matrix and translation vector of the thermal imaging detector relative to the calibration plate. From this, the relative extrinsic parameters between the night vision detector and the thermal imaging detector can be calculated: in, These represent the rotation matrix and translation vector, respectively, for transforming from the thermal imaging detector coordinate system to the night vision detector coordinate system. This result serves as the pre-calibration spatial mapping relationship.

[0029] Based on the pre-calibrated spatial mapping relationship, image time alignment and pixel-level registration are performed on the synchronously acquired night vision images and thermal imaging images. Image time alignment is achieved through timestamp synchronization, and pixel-level registration is performed through image transformation algorithms, such as affine transformation or perspective transformation, to rotate, translate and scale the thermal imaging images to achieve the best match with the night vision images at the pixel level. This ensures that the projection position of the same physical target in the two images can be one-to-one at the pixel level, ultimately forming a temporally continuous and spatially consistent sequence of night vision images and a sequence of thermal imaging images.

[0030] By jointly calibrating internal and external parameters to construct a stable pre-calibrated spatial mapping relationship, night vision images and thermal imaging images are simultaneously aligned in both time and space dimensions. This avoids feature misalignment problems caused by differences in viewpoint and imaging distortion, enabling the extraction of more accurate and comprehensive target feature information, thereby improving the accuracy and stability of target tracking.

[0031] The night vision image sequence and the thermal imaging image sequence are traversed to perform image preprocessing and multimodal feature fusion to obtain a multimodal fusion feature map.

[0032] Furthermore, image preprocessing and multimodal feature fusion are performed on the night vision image sequence and thermal imaging image sequence to obtain a multimodal fusion feature map. This includes: performing low-light image enhancement preprocessing on the night vision image sequence to obtain an enhanced night vision image sequence; performing non-uniformity calibration and thermal noise suppression processing on the thermal imaging image sequence to obtain an enhanced thermal imaging image sequence; and performing one-to-one mapping and joint multimodal feature temporal processing on the enhanced night vision image sequence and the enhanced thermal imaging image sequence to obtain the multimodal fusion feature map.

[0033] Specifically, low-light image enhancement preprocessing is performed on each night image in the night image sequence. The low-light image enhancement is used to improve the problems of insufficient brightness, low contrast and severe noise interference in night imaging. By separating the illuminance component and reflectance component of each night image in the night image sequence and adaptively enhancing the illuminance component, the target contour and texture features are effectively improved in the low-light environment, thereby obtaining an enhanced night image sequence.

[0034] Non-uniformity calibration and thermal noise suppression are performed on each thermal image in the thermal imaging image sequence. Non-uniformity calibration compensates for fixed pattern noise caused by inconsistent pixel responses in the infrared detector. Common non-uniformity calibration methods include calibration-based and scene-based methods. Calibration-based methods use standard targets such as blackbody radiation sources to calibrate the detector and obtain correction parameters. Scene-based methods use scene information in the thermal imaging image sequence to estimate correction parameters. For example, for calibration-based non-uniformity calibration, the temperature range of the blackbody radiation source can be set between 20℃ and 60℃, and at least three temperature points are selected for calibration to cover common nighttime ambient temperatures and target temperature variations. This can be further extended to a temperature range of 0℃ to 80℃ to adapt to complex environments or high-temperature target scenes. For scene-based non-uniformity calibration, 5-30 consecutive frames of thermal imaging images can be selected as a statistical window to estimate pixel response deviation parameters, achieving a balance between calibration accuracy and computational complexity.

[0035] Thermal noise suppression is used to reduce random interference introduced by ambient temperature fluctuations or noise from the thermal imaging detector itself. Filtering algorithms such as median filtering and Gaussian filtering are employed. Median filtering replaces the value of each pixel in the thermal image with the median value of its neighboring pixels, effectively removing impulse noise. Gaussian filtering smooths the thermal image using a Gaussian function, suppressing high-frequency noise. In the thermal noise suppression process, the neighborhood window size used for median filtering can be set to 3×3, 5×5, or 7×7 pixels. In scenes with high noise levels, a 5×5 or 7×7 window can be used to enhance denoising, while a 3×3 window can be used to reduce target outline blurring in scenes requiring high edge detail. The standard deviation parameter σ used in Gaussian filtering can be set between 0.5 and 2.0, and the corresponding filter kernel size can be set to a range of 3×3 to 9×9 pixels to suppress high-frequency noise of varying intensities.

[0036] After non-uniform calibration and thermal noise suppression, an enhanced thermal imaging image sequence with higher contrast and more stable background is obtained. Then, the enhanced night vision image sequence and the enhanced thermal imaging image sequence are mapped one-to-one in chronological order. One-to-one mapping means that the images at the same time point in the enhanced night vision image and the enhanced thermal imaging image are processed accordingly to ensure the correspondence between the two images in time and space. Based on the corresponding images at the same time, joint multimodal feature temporal processing is performed to fuse the visible light texture, edge and other structural features contained in the night vision image with the infrared features in the thermal imaging image that reflect the target temperature distribution and thermal contrast characteristics, forming a multimodal fusion feature map that contains both spatial information and temporal evolution information.

[0037] By preprocessing and enhancing the imaging defects of night vision and thermal imaging images separately, the quality of both types of images is improved, effectively avoiding the negative impact of insufficient single-modal feature quality on the fusion result. Simultaneously, through one-to-one mapping of multimodal feature temporal processing, the detailed structural information of the night vision modality is fused with the target information of the thermal imaging modality. This fully utilizes the information from both different modalities, compensating for the limitations of a single modality and resulting in more comprehensive and accurate extracted target features.

[0038] Furthermore, the night view image sequence is subjected to low-light image enhancement preprocessing to obtain an enhanced night view image sequence, including: performing luminance separation processing on the night view image sequence based on the Retinex model to obtain an illuminance component sequence; performing adaptive contrast stretching on the illuminance component sequence and combining it with a frequency domain denoising algorithm to suppress random noise, thereby obtaining an enhanced night view image sequence.

[0039] Specifically, a luminance separation process based on the Retinex model is introduced for each frame of the night image sequence. Retinex theory posits that an image is composed of illumination and reflection components. The illumination component reflects the overall lighting conditions of the image, while the reflection component reflects the reflective properties of objects. A single-scale Retinex algorithm is used to separate the illumination component from the night image by convolving the night image with a Gaussian function. The standard deviation of the Gaussian function determines the convolution scale, which can be set according to actual needs. For example, setting the standard deviation of the Gaussian function to 30, each frame of the night image sequence is convolved with the set Gaussian function to obtain the corresponding illumination component sequence.

[0040] Then, an adaptive contrast stretching method based on local statistical information is used to adaptively stretch the illuminance component sequence. For each frame of the illuminance component sequence, it is divided into several local regions, for example, into 16×16 pixel blocks. For each local region, the mean μ and standard deviation σ of its pixel values ​​are calculated, and the pixel values ​​within the local region are stretched according to the following formula: I new =a×(I-μ)+μ, where I is the original pixel value, I new This refers to the stretched pixel value, where 'a' is a stretching coefficient dynamically adjusted based on the standard deviation. The formula is: a = 1 + k × σ, where k is an empirical parameter, such as setting k = 0.5. The stretching coefficient is then limited based on actual needs and experience. By performing targeted contrast stretching based on the characteristics of local image regions, the problem of over-enhancement or under-enhancement that may occur with global stretching is avoided. This improves the contrast of night vision images, brightening dark areas while preventing overexposure in bright areas, thereby enhancing the visual effect of night vision images.

[0041] After contrast stretching, a frequency domain denoising algorithm is introduced to reduce the high-frequency noise component by performing a frequency domain transformation on the contrast component. The frequency domain transformation can be achieved by using a Fast Fourier Transform to convert the contrast-stretched image to the frequency domain, resulting in a frequency domain image. In the frequency domain, noise typically manifests as high-frequency components, while the main information of the image is concentrated in the low-frequency and mid-frequency components. A Butterworth low-pass filter is chosen, and its transfer function is: in, Here, D(u,v) is the frequency domain coordinate, D(u,v) is the distance from point (u,v) to the center of the frequency domain, D0 is the cutoff frequency, and n is the filter order. For example, the cutoff frequency is determined to be D0 = 0.2 × N. N is the image size, and the filter order n = 2. Multiplying the frequency domain image by the low-pass filter yields the filtered frequency domain image, which is then converted back to the spatial domain using an inverse fast Fourier transform, thus obtaining an enhanced night view image sequence with suppressed random noise.

[0042] By using brightness separation based on the Retinex model, the problem of brightness attenuation caused by uneven illumination in night vision images is effectively alleviated. At the same time, adaptive contrast stretching enhances target information in dark areas, making target contours and texture features clearer under low illumination conditions. Combined with frequency domain denoising, random noise and particle noise commonly found in night vision imaging are effectively suppressed, thereby preventing noise from being amplified or misjudged as target features during multimodal feature fusion, and improving the stability and continuity of overall dynamic target tracking.

[0043] Furthermore, the enhanced night vision image sequence and the enhanced thermal imaging image sequence are subjected to one-to-one mapping and joint multimodal feature temporal processing to obtain the multimodal fusion feature map. This includes: performing one-to-one mapping and joint multimodal feature processing on the enhanced night vision image sequence and the enhanced thermal imaging image sequence to obtain an initial multimodal fusion feature map sequence; performing temporal trend superposition processing on the first initial multimodal fusion feature map and the second initial multimodal fusion feature map in the initial multimodal fusion feature map sequence to obtain a first superimposed multimodal fusion feature map; using the first superimposed multimodal fusion feature map to perform temporal trend superposition processing on the third initial multimodal fusion feature map in the initial multimodal fusion feature map sequence to obtain a second superimposed multimodal fusion feature map, and so on, to obtain the multimodal fusion feature map.

[0044] Specifically, based on the pre-calibrated spatial mapping relationship, a precise one-to-one correspondence is established between the image frames in the enhanced night vision image sequence and the enhanced thermal imaging image sequence. The pre-calibrated spatial mapping relationship is determined by calibrating the spatial position and adjusting the parameters of the night vision device and the thermal imaging device, which ensures that the enhanced night vision image and the enhanced thermal imaging image achieve a pixel-level correspondence in the spatial coordinate system under the same time index, thereby constructing a multimodal synchronized image pair sequence.

[0045] For each pair of multimodal synchronized images, feature extraction networks with identical structures but independent parameters are used to extract features from both types of images. For enhanced night vision images, night vision visual features are extracted, which include at least texture features and edge features. Texture features are extracted using the Local Binary Pattern (LBP) algorithm. Each pixel in the night vision image is taken as the center pixel, and several neighboring pixels within a fixed radius are selected around it. Commonly, eight pixels within a 3×3 neighborhood are used as comparison objects. The gray values ​​of the neighboring pixels are compared one by one with the gray value of the center pixel. If the gray value of a neighboring pixel is greater than or equal to the gray value of the center pixel, the corresponding position is assigned a value of 1; otherwise, it is assigned a value of 0, thus forming a binary sequence of 0s and 1s. This binary sequence is converted into a decimal number according to a preset order and used as the texture encoding value of the center pixel. By performing the above operation on all pixels in the entire night vision image and representing the obtained texture encoding values ​​statistically or using a histogram, a texture feature descriptor reflecting the local texture distribution characteristics of the night vision image can be formed, used to characterize the microstructural information of the target surface. Edge features are used to reflect the contours and boundaries of objects. The Canny edge detection algorithm can be used. First, Gaussian filtering is performed on the enhanced night view image for noise reduction. By convolving with a two-dimensional Gaussian filter kernel, the night view image is smoothed to reduce the impact of random noise on subsequent gradient calculations. Gray-level gradients are calculated in the horizontal and vertical directions of the smoothed night view image to obtain the gradient magnitude and gradient direction of each pixel. The gradient magnitude represents the strength of gray-level changes at the pixel, and the gradient direction represents the direction of the edge. Based on this, non-maximum suppression is performed, that is, the gradient magnitude of the current pixel is compared with the gradient magnitude of its neighboring pixels along the gradient direction, and only local gradient maxima are retained, thereby refining the edge width. Finally, a dual-threshold detection strategy is adopted. Based on experience or historical data, high and low thresholds are set to classify the gradient magnitude. Pixels with a value greater than the high threshold are directly identified as strong edges, pixels with a value less than the low threshold are discarded, and pixels in between are determined based on whether they are connected to strong edges, thus obtaining continuous and clear edge features of the night view image.

[0046] For enhanced thermal imaging images, infrared thermal features are extracted, including at least thermal distribution features and target contour features. Thermal distribution features reflect the temperature distribution on the surface of the target object and can be obtained by dividing the thermal imaging image into several small regions and calculating the average temperature of each region. Target contour features are obtained through a segmentation method based on a temperature threshold. A threshold is set according to the temperature difference between the target and the background in the current scene, and pixel regions above the threshold are taken as target regions, thereby extracting the spatial contour shape of the target as the target structural features in the infrared mode.

[0047] The extracted night vision visual features and infrared thermal features are aligned using scale normalization and channel mapping operations. Scale normalization, through methods such as bilinear interpolation, adjusts feature maps of different resolutions to the same size. Channel mapping, through fully connected layers or 1×1 convolutional structures, maps feature vectors of different dimensions to the same feature dimension, ensuring consistency in both spatial resolution and feature dimension, forming a feature representation capable of joint operations. An attention weighting mechanism is introduced into the aligned feature space. Adaptive weights are assigned to different modal features based on the response intensity of the night vision visual features and infrared thermal features within the target region. Specifically, this can be achieved by statistically analyzing the response intensity of the two types of features within the target region, for example, calculating the average or maximum value of feature activation values ​​within the target region, and using this response intensity as the basis for weight generation. When the night vision visual features exhibit a higher response intensity within the target region, a larger weight coefficient is assigned, such as 0.6; when the infrared thermal feature response is more significant, the weight of the infrared thermal feature is correspondingly increased. By multiplying the two types of features by their respective weight coefficients, redundant features in the background region are suppressed, while significant features in the target region are enhanced. For example, a dual-threshold strategy using Canny edge detection is employed to enhance night vision images. The high threshold is set to 100, and the low threshold to 50, with a grayscale value range of 0-255. After calculating the gradient magnitude of each pixel, pixels with a gradient magnitude greater than the high threshold of 100 are directly identified as strong edges, while pixels with a gradient magnitude less than the low threshold of 50 are discarded. For pixels with a gradient magnitude between 50 and 100, it is determined whether they are connected to strong edge pixels. If connected, they are retained as weak edges; otherwise, they are discarded. This processing enhances the recognizability of target contours in low-light environments and the stability of subsequent feature extraction in night vision images.

[0048] Finally, while maintaining the spatial resolution, the weighted night vision visual features and infrared thermal features are fused using a channel-by-channel stitching method. This results in a fused feature map that simultaneously contains the texture and edge information of the night vision modality and the thermal distribution and contour information of the infrared modality, thereby generating an initial multimodal fusion feature map corresponding to each frame, forming an initial multimodal fusion feature map sequence. This fusion method achieves information complementarity without destroying the original feature structure of each modality. For example, after feature extraction and scale normalization and channel mapping alignment of the enhanced night vision image and enhanced thermal imaging image, the spatial resolution of the night vision visual feature map is 160×120, and the number of channels is 64. The spatial resolution of the infrared thermal feature map is also 160×120, and the number of channels is 64. The two types of feature maps are consistent in both spatial size and channel dimension. Under the attention weighting mechanism, night vision visual features and infrared thermal features are multiplied by the target region adaptive weights, with night vision features having a weight of 0.45 and infrared thermal features having a weight of 0.55. Then, the weighted feature maps of the two types are stitched together along the channel dimension. The resulting single-frame multimodal fusion feature map still has a spatial resolution of 160×120 and 128 channels. It contains both the texture and edge information of the night vision modality and the thermal distribution and target contour information of the infrared modality, achieving effective complementarity between the two modalities.

[0049] After obtaining the initial multimodal fusion feature map sequence, a temporal trend superposition mechanism is introduced. This mechanism performs recursive temporal fusion processing on adjacent initial multimodal fusion feature maps within the sequence. Specifically, it superimposes the first and second initial multimodal fusion feature maps using temporal trends. This superposition process analyzes the changing trends between the initial multimodal fusion feature maps of consecutive frames, adding information from the previous frame's initial multimodal fusion feature map to the current frame's initial multimodal fusion feature map with a certain weight, thereby enhancing the temporal continuity and stability of the features. Specifically, the similarity between the first and second initial multimodal fusion feature maps is calculated using methods such as cosine similarity, and the superposition weight is determined based on the similarity. For example, if the similarity between the two initial multimodal fusion feature maps is high, it indicates that the target motion changes less, and the previous frame's initial multimodal fusion feature map can be given a larger weight; conversely, if the similarity is low, it indicates that the target motion changes more, and the previous frame's initial multimodal fusion feature map can be given a smaller weight. The first initial multimodal fusion feature map is superimposed onto the second initial multimodal fusion feature map according to the calculated weights to obtain the first superimposed multimodal fusion feature map. Using this first superimposed multimodal fusion feature map as a temporal reference, the same trend superposition process is performed on the third initial multimodal fusion feature map to obtain the second superimposed multimodal fusion feature map. This process is repeated, ensuring that the current frame features retain their spatial information while incorporating historical temporal trend information, ultimately forming a multimodal fusion feature map that combines spatial expressiveness with temporal continuity.

[0050] One-to-one mapping combined with multimodal feature processing can effectively fuse different features of night vision images and thermal imaging images, giving full play to the advantages of both modal images and improving the accuracy of target detection and recognition. Temporal trend overlay processing takes into account the temporal information of the image sequence. By overlaying the features of consecutive frames, the stability and continuity of the features are enhanced, which helps to track targets more accurately and reliably in dynamic scenes.

[0051] Furthermore, a temporal trend overlay processing is performed on the first and second initial multimodal fusion feature maps in the initial multimodal fusion feature map sequence to obtain a first overlaid multimodal fusion feature map. This includes: performing approximate analysis on the first and second initial multimodal fusion feature maps from two dimensions: night vision features and thermal imaging features, respectively, to obtain night vision feature similarity and thermal imaging feature similarity; normalizing the night vision feature similarity and thermal imaging feature similarity using the max-min normalization method, and filling the normalization result into an initially empty two-dimensional diagonal matrix to construct a trend overlay matrix; and using the trend overlay matrix to perform trend overlay on the second initial multimodal fusion feature map to obtain the first overlaid multimodal fusion feature map.

[0052] Specifically, when performing temporal trend overlay processing on the initial multimodal fusion feature map sequence, the first and second initial multimodal fusion feature maps in the sequence are selected first. Approximate analyses are then performed on the two initial multimodal fusion feature maps in the night vision visual feature dimension and the infrared thermal feature dimension, respectively, to measure the feature consistency between the two initial multimodal fusion feature maps in the target and background regions. Specifically, night vision feature similarity reflects the degree of matching between the two night vision modal features in texture and edge structure, while infrared thermal feature similarity reflects the degree of matching between the two infrared modal features in target thermal distribution and contour. The approximate analysis uses a cosine similarity algorithm to calculate the similarity. Cosine similarity is obtained by calculating the difference between the two vectors in the night vision feature and infrared thermal feature directions, respectively.

[0053] The night vision feature similarity and thermal imaging feature similarity are normalized using the min-max normalization method, mapping the feature similarity values ​​to the 0-1 range to eliminate the influence of numerical scale differences on the superposition result. The normalized result is then filled into an initially empty two-dimensional diagonal matrix, with non-zero elements on the main diagonal and zero elements elsewhere. This normalized night vision and thermal imaging feature similarities are used as elements on the main diagonal of the two-dimensional diagonal matrix to construct a 2×2 trend superposition matrix. This trend superposition matrix reflects the similarity between the initial multimodal fusion feature maps of two consecutive frames in the night vision and thermal imaging feature dimensions. The trend superposition matrix is ​​then used to superimpose the second initial multimodal fusion feature map. A matrix multiplication operation is then performed between the trend superposition matrix and the second initial multimodal fusion feature map to obtain the first superimposed multimodal fusion feature map. Through this trend superposition operation, the information from the first initial multimodal fusion feature map is integrated into the second initial multimodal fusion feature map according to the weights determined by the trend superposition matrix, enhancing the temporal continuity and stability of the fused features.

[0054] By fusing feature trend information from the previous frame, the current initial multimodal fusion feature map achieves smooth processing of short-term dynamic changes while maintaining spatial resolution and target details. This improves the ability of multimodal fusion features to describe fast-moving targets or partially occluded targets, and enhances the stability and accuracy of dynamic target tracking.

[0055] A pre-built target tracking model is invoked, and the first or second cue vector of the target tracking model is invoked based on the multimodal fusion feature map to perform joint target tracking on the night view image sequence and the thermal imaging image sequence to obtain the target tracking trajectory.

[0056] Furthermore, a pre-built target tracking model is invoked, and based on the multimodal fusion feature map, the first or second cue vector of the target tracking model is invoked to perform joint target tracking on the night view image sequence and the thermal imaging image sequence to obtain the target tracking trajectory. This includes: acquiring multiple historical multimodal fusion feature maps, as well as multiple sample night view image sequences, multiple sample thermal imaging image sequences, and multiple sample target tracking trajectories, as training sample sets; aggregating the multiple historical multimodal fusion feature maps of the same type to obtain multiple aggregated historical multimodal fusion feature map sets; and performing base class and new class data partitioning based on the number of feature maps within the multiple aggregated historical multimodal fusion feature map sets to obtain base class aggregated historical multimodal fusion feature maps. The system generates a base class data set and a new class data set. Based on these two sets, the system maps and partitions the multiple sample night vision image sequences, multiple sample thermal imaging image sequences, and multiple sample target tracking trajectories. The system then trains a model using the training sample set to obtain an initial target tracking model. Finally, it performs two-dimensional cue training on the initial target tracking model using the base class data set and the new class data set to obtain a first cue vector and a second cue vector. The first and second cue vectors are then stored in parallel in the semantic encoding module of the initial target tracking model to obtain the target tracking model.

[0057] Specifically, a training sample set is obtained, which consists of multiple historical multimodal fusion feature maps, as well as multiple sample night vision image sequences, multiple sample thermal imaging image sequences, and multiple sample target tracking trajectories. The historical multimodal fusion feature maps are feature representations that fuse multimodal information such as night vision and thermal imaging, integrating the characteristics of the target under different modalities, and can more comprehensively describe the target. The sample night vision image sequences and sample thermal imaging image sequences record the state of the target at different times from different imaging methods, and the sample target tracking trajectories record the motion path of the target in multiple image sequences.

[0058] Clustering algorithms, such as K-means clustering, are used to aggregate historical multimodal fusion feature maps into similar clusters. K-means clustering iteratively assigns data points to the nearest cluster centers, continuously updating the positions of these centers until convergence. This groups feature maps with similar target feature patterns into the same class, achieving classification and aggregation of historical multimodal fusion feature maps into multiple aggregated historical multimodal fusion feature map sets to reflect the typical feature distribution under different target categories or scenarios. Based on the number of feature maps within each aggregated historical multimodal fusion feature map set, base class and new class data partitioning is performed. A partitioning threshold is set based on experience or historical data. When the number of feature maps in a particular aggregated historical multimodal fusion feature map set is greater than or equal to the threshold, it indicates that the category represented by that aggregated historical multimodal fusion feature map set has a sufficient number of samples, and it is classified as the base class; otherwise, it is classified as the new class. By comparing the partitions, base class aggregated historical multimodal fusion feature map sets and new class aggregated historical multimodal fusion feature map sets are obtained. For example, setting a data partitioning threshold of 50, if a certain aggregated historical multimodal fusion feature map set contains 60 feature maps, then that aggregated historical multimodal fusion feature map set is classified as the base class; if another aggregated historical multimodal fusion feature map set contains 30 feature maps, then that aggregated historical multimodal fusion feature map set is classified as the new class. Based on the base class aggregated historical multimodal fusion feature map set and the new class aggregated historical multimodal fusion feature map set, multiple sample night view image sequences, multiple sample thermal imaging image sequences, and multiple sample target tracking trajectories are mapped and partitioned. The night view image sequences, thermal imaging image sequences, and corresponding target trajectories in the training sample set are mapped to the base class data set and the new class data set, achieving training to differentiate between common targets and newly emerging targets.

[0059] When using training samples to initially train a convolutional neural network to obtain an initial target tracking model, the historical multimodal fusion feature map is first preprocessed. This historical multimodal fusion feature map is obtained by fusing night vision modal features and thermal imaging modal features, containing a comprehensive representation of the target in both visible and infrared bands. The preprocessing includes size unification and normalization of the feature map, ensuring that the spatial resolution of each frame of the multimodal fusion feature map is uniformly 160×120 and the number of channels is 128. Each training sample includes one or more consecutive frames of historical multimodal fusion feature maps as input, and the corresponding target tracking trajectory as a supervision label. The target tracking trajectory is represented as a sequence of pixel coordinates of the target in the feature map coordinate system.

[0060] Based on this, a convolutional neural network model structure is constructed. The model is a convolutional neural network architecture, including an input layer, five convolutional layers, a pooling layer, a fully connected layer, and an output layer. The input layer receives historical multimodal fusion feature maps with a size of 160×120×128. Convolutional layer 1 uses a 3×3 kernel, 32 channels, a stride of 1, and the same padding method to extract low-level spatial features of the target. Batch normalization and ReLU activation are applied after convolution. Convolutional layer 2 uses a 3×3 kernel, 64 channels, and a stride of 1. It also uses batch normalization and ReLU activation, and downsampling is performed using 2×2 max pooling to reduce the spatial resolution of the feature map and enhance the translation invariance of the features. Convolutional layers 3 and 4 both use 3×3 kernels and 128 channels to extract higher-level semantic features. Convolutional layer 5 uses a 3×3 kernel and 256 channels. After this layer, 2×2 max pooling is applied to downsample the feature map to a spatial size of 40×30. All of the above convolutional layers are followed by ReLU activation to enhance the non-linear expressive power of the model. The feature map output by convolutional layer 5 is flattened into a one-dimensional feature vector and input to the fully connected layer. The fully connected layer uses ReLU activation to comprehensively model the high-dimensional features, which is used to learn the complex mapping relationship between multimodal features and target motion state. The output layer generates the predicted target tracking trajectory based on the output of the fully connected layer. The predicted result is represented in the form of a coordinate sequence of the target center point. For example, in the two-dimensional feature map, the (x,y) coordinates describe the position change of the target in each frame, thus forming a continuous target tracking trajectory.

[0061] During training, training samples are sequentially input into the convolutional neural network for forward propagation. The historical multimodal fusion feature map passes through convolutional layers, pooling layers, fully connected layers, and the output layer to obtain the predicted target tracking trajectory. Subsequently, the predicted trajectory is compared with the actual target tracking trajectory, and the mean squared error loss function is used to calculate the error between the predicted and actual trajectories. This loss value is propagated layer by layer in the network using the backpropagation algorithm to calculate the gradients of trainable parameters such as the weights of each convolutional kernel and fully connected layer. The Adam optimizer is used to update the model parameters based on the gradient information, with an initial learning rate of 0.001 and a batch size of 32. The training process iterates for 100 epochs, randomly shuffling the order of training samples in each epoch to improve the model's generalization ability. This training process is repeated until the loss function value no longer decreases significantly or the preset number of iterations is reached, thus obtaining the initial target tracking model. This initial target tracking model can accurately predict the continuous trajectory of the target in an image sequence based on the input multimodal fusion feature map.

[0062] A dual-cue training method is used to train the initial target tracking model using both base class and new class datasets. This dual-cue training enables the initial target tracking model to better adapt to different types of data. By learning the feature differences between base class and new class data, a first cue vector and a second cue vector are generated. The first cue vector provides feature cues for the base class data, and the second cue vector provides feature cues for the new class data, thereby enhancing the initial target tracking model's tracking ability across different data categories. The first and second cue vectors are then stored in parallel in the semantic encoding module of the initial target tracking model to obtain the final target tracking model. The semantic encoding module is the part of the initial target tracking model that performs semantic understanding and feature encoding on the input data. Storing the cue vectors in this module allows the target tracking model to combine the information from the cue vectors for more accurate feature extraction and target tracking when processing input data. After obtaining the target tracking model, night vision image sequences and thermal imaging image sequences are input. The target tracking model performs joint target tracking based on the cue vectors and outputs the target tracking trajectory, achieving high-precision target localization and motion tracking driven by multimodal features.

[0063] By using dual-modal fusion feature input and combining the base class and new class cue vectors of historical samples, the dynamic information of the target can be captured simultaneously in night vision mode and infrared mode, enabling stable perception and continuous tracking of dynamic targets in complex low-light environments, effectively improving the trajectory positioning accuracy, tracking accuracy and reliability of dynamic targets.

[0064] Furthermore, the initial target tracking model is subjected to two-dimensional cue training using the base class dataset and the new class dataset to obtain a first cue vector and a second cue vector. This includes: loading an initial cue vector from the semantic encoding module of the initial target tracking model as a common cue vector, copying it to generate a first initial cue vector and a second initial cue vector; based on the first initial cue vector and the second initial cue vector, the initial target tracking model is subjected to two-dimensional cue training using the base class dataset and the new class dataset to obtain base class model state parameters and base class optimized cue vectors, as well as new class model state parameters and new class optimized cue vectors; associating the base class optimized cue vectors using the base class model state parameters to obtain a first cue vector; and associating the new class optimized cue vectors using the new class model state parameters to obtain a second cue vector.

[0065] Furthermore, based on the first and second initial cue vectors, the initial target tracking model is trained using a base class dataset and a new class dataset in a two-dimensional cue-based manner to obtain base class model state parameters and base class optimized cue vectors, as well as new class model state parameters and new class optimized cue vectors. This includes: inputting the base class dataset into the initial target tracking model, keeping the model's main parameters and the second initial cue vector parameters frozen and not updated, allowing only the first initial cue vector to participate in backpropagation, and after multiple rounds of optimization, obtaining the base class optimized cue vectors and base class model state parameters; inputting the new class dataset into the initial target tracking model, keeping the model's main parameters and the first initial cue vector parameters frozen and not updated, allowing only the second initial cue vector to participate in backpropagation, and after multiple rounds of optimization, obtaining the new class optimized cue vectors and new class model state parameters.

[0066] Specifically, in the initial target tracking model, the semantic encoding module is an intermediate encoding unit positioned between the convolutional feature extraction network and the target tracking prediction head. It performs semantic-level mapping and compressed representation of the high-dimensional multimodal fusion features extracted by the convolutional neural network. That is, the semantic encoding module is located between the convolutional layer and the fully connected layer, or is set independently as part of the fully connected layer. Its input is the high-dimensional feature vector output by the convolutional layer, and its output is a low-dimensional semantic representation vector describing target category attributes, historical motion patterns, or contextual prior information. The semantic encoding module can consist of one or more trainable linear mapping layers, such as a fully connected encoding layer and a non-linear activation layer, used to map the feature vector output by the convolutional layer to a fixed-dimensional semantic space. Assuming the convolutional layer output features are flattened into a 1×N dimensional vector, the semantic encoding module maps this feature vector into a 1×d dimensional semantic encoding vector using a weight matrix W∈R^(N×d) and a bias term b, where d is a preset semantic embedding dimension, for example, d=128 or 256. This semantic encoding vector is stored in the semantic encoding module as the initial cue vector, and is used to carry target category information or statistical priors of historical trajectories.

[0067] After the initial target tracking model training is completed, the 1×d-dimensional semantic encoding vector is directly read from the semantic encoding module as a common cue vector. This common cue vector serves as an explicit trainable parameter within the model and is stored together with the parameters of the convolutional layers and the target tracking prediction head. However, it can be loaded, copied, frozen, or updated separately during subsequent two-dimensional cue training phases. By performing parameter-level copying of this common cue vector, a first initial cue vector V1 and a second initial cue vector V2 are generated. Both are identical in dimensional structure and parameter scale, being 1×d-dimensional row vectors, but occupying independent storage locations in the model parameter space. They are respectively bound to the base class cue branch and the new class cue branch, thus supporting independent gradient calculation and independent optimization during backpropagation. This ensures that the model can independently adapt to the feature distribution of the base class and new class datasets during training. The first initial cue vector is mainly used for optimization when processing the base class dataset. It learns the prior information of the base class target features during training and continuously adjusts its parameters through backpropagation to better represent the features of the base class targets, thereby providing the model with guidance information about the base class targets in subsequent target tracking tasks. The second initial cue vector is mainly used for optimization when processing new types of datasets. It learns prior information about the features of new target classes and adjusts the parameters through backpropagation to adapt to the features of new target classes, thus providing effective guidance for the model when tracking new target classes.

[0068] The base class dataset is input into the initial target tracking model. During training, the main parameters of the initial target tracking model (convolutional and fully connected layer parameters) and the parameters of the second initial cue vector are kept frozen. Only the first initial cue vector is allowed to participate in backpropagation, i.e., to participate in gradient calculation and optimization. The mean squared error loss function is used for backpropagation during training. The Adam optimization algorithm is used to update the parameters of the first initial cue vector according to the gradient. Through multiple rounds of optimization iterations, the first initial cue vector can effectively learn the prior information of the base class target features to form the base class optimized cue vector. The internal state information of the initial target tracking model recorded in each iteration is used as the state parameters of the base class model. The internal state information of the initial target tracking model refers to the feature maps output by each layer during forward propagation, such as the channel features of the convolutional layer, the dimensionality reduction results of the pooling layer, and intermediate activation values, such as the non-zero distribution after ReLU. This information is used to capture the dynamic representation changes of the new class data in the initial target tracking model. Similarly, the new class dataset is input into the initial target tracking model, the main parameters of the model and the first initial cue vector are frozen, and only the second initial cue vector is allowed to participate in backpropagation. Through multiple rounds of iterative optimization, it is made to encode the prior information of the new target features. At the same time, the internal state information of the initial target tracking model recorded in each round of iteration is used as the state parameters of the new class model so that it can be used for subsequent cue vector association.

[0069] After the two-dimensional cue training is completed, the state parameters of the base class model are associated with the base class optimized cue vector to generate the first cue vector, which is used to provide prior information of the base class target during tracking. The state parameters of the new class model are associated with the new class optimized cue vector to generate the second cue vector, which is used to provide prior information of the new class target during tracking. This completes the two-dimensional cue training, enabling the initial target model to efficiently track common targets (base class) and newly emerging targets (new class) using different cue vectors during tracking.

[0070] For example, if the initial cue vector P0 is a 1×64 row vector, and its elements are initialized to follow a Gaussian distribution with a mean of 0 and a variance of 0.01, first initial cue vectors V1 and V2 are generated by copying P0. These are also 1×64 row vectors with the same element values ​​as P0. The two initial cue vectors are stored independently in memory, supporting independent gradient calculation and optimization. The base class dataset is input into the initial target tracking model, freezing the model's main parameters and the parameters of the second initial cue vector V2. Only the first initial cue vector V1 is allowed to participate in backpropagation. The mean squared error loss function is used to measure the difference between the model's predicted result and the actual target tracking trajectory. The Adam optimization algorithm is used to update the parameters of V1 based on the gradient. After 50 rounds of optimization iterations, V1 gradually learns the prior information of the target features in the base class data, forming the optimized base class cue vector. Simultaneously, during the forward propagation process, the intermediate state parameters of the model are recorded as base class model state parameters. These intermediate state parameters refer to the feature representations output by the main network at each layer. Similarly, by training with a new dataset, we obtain new class optimization cue vectors and new class model state parameters. We then associate the base class model state parameters with the base class optimization cue vectors to generate a first cue vector. We then associate the new class model state parameters with the new class optimization cue vectors to generate a second cue vector. The association method can be a simple concatenation operation, which enables the target tracking model to better adapt to tracking tasks of different types of targets and improves the stability and adaptability of dynamic target tracking.

[0071] By optimizing the cue vectors for base class and new class targets respectively, the target tracking model can adaptively select cue vectors according to the target's category during the tracking process, effectively improving the target positioning accuracy, reliability, and adaptive tracking performance for newly appearing targets.

[0072] Example 2, based on the same inventive concept as the dynamic target tracking method based on night vision sight image processing in the previous examples, such as... Figure 2 As shown, this application provides a dynamic target tracking system based on night vision sight image processing, wherein the dynamic target tracking system based on night vision sight image processing includes: Image sequence acquisition module 11 is used to simultaneously acquire night vision images and thermal imaging images at the same time using the night vision detector and thermal imaging detector of the night vision sight within a preset window, and perform image time alignment and pixel-level registration according to a pre-calibrated spatial mapping relationship to obtain a night vision image sequence and a thermal imaging image sequence; Image sequence processing module 12 is used to traverse the night vision image sequence and thermal imaging image sequence to perform image preprocessing and multimodal feature fusion to obtain a multimodal fusion feature map; Joint target tracking module 13 is used to call a pre-built target tracking model, and call the first cue vector or the second cue vector of the target tracking model based on the multimodal fusion feature map to perform joint target tracking on the night vision image sequence and the thermal imaging image sequence to obtain the target tracking trajectory.

[0073] Furthermore, the image sequence acquisition module 11 is also used to: perform intrinsic parameter calibration on the night vision detector and the thermal imaging detector; after completing the intrinsic parameter calibration, obtain the extrinsic parameter relationship between the night vision detector and the thermal imaging detector based on the infrared-visible light joint calibration board; and use the extrinsic parameter relationship as a pre-calibration spatial mapping relationship.

[0074] Furthermore, the image sequence processing module 12 is also used to: perform low-light image enhancement preprocessing on the night view image sequence to obtain an enhanced night view image sequence; perform non-uniformity calibration and thermal noise suppression processing on the thermal imaging image sequence to obtain an enhanced thermal imaging image sequence; and perform one-to-one mapping and multimodal feature temporal processing on the enhanced night view image sequence and the enhanced thermal imaging image sequence to obtain the multimodal fusion feature map.

[0075] Furthermore, the image sequence processing module 12 is also used to: perform brightness separation processing on the night view image sequence based on the Retinex model to obtain the illuminance component sequence; perform adaptive contrast stretching on the illuminance component sequence, and combine it with a frequency domain denoising algorithm to suppress random noise, thereby obtaining an enhanced night view image sequence.

[0076] Furthermore, the image sequence processing module 12 is also used to: perform one-to-one mapping joint multimodal feature processing on the enhanced night vision image sequence and the enhanced thermal imaging image sequence to obtain an initial multimodal fusion feature map sequence; perform sequential trend superposition processing on the first initial multimodal fusion feature map and the second initial multimodal fusion feature map in the initial multimodal fusion feature map sequence to obtain a first superimposed multimodal fusion feature map; use the first superimposed multimodal fusion feature map to perform sequential trend superposition processing on the third initial multimodal fusion feature map in the initial multimodal fusion feature map sequence to obtain a second superimposed multimodal fusion feature map, and so on, to obtain multimodal fusion feature maps.

[0077] Furthermore, the image sequence processing module 12 is also used to: perform approximate analysis on the first initial multimodal fusion feature map and the second initial multimodal fusion feature map from two dimensions, night vision features and thermal imaging features, respectively, to obtain night vision feature similarity and thermal imaging feature similarity; normalize the night vision feature similarity and thermal imaging feature similarity using the max-min normalization method, and fill the normalization result into the initially empty two-dimensional diagonal matrix to construct a trend superposition matrix; use the trend superposition matrix to perform trend superposition on the second initial multimodal fusion feature map to obtain a first superimposed multimodal fusion feature map.

[0078] Furthermore, the joint target tracking module 13 is also used to: acquire multiple historical multimodal fusion feature maps, as well as multiple sample night view image sequences, multiple sample thermal imaging image sequences, and multiple sample target tracking trajectories, as a training sample set; aggregate the multiple historical multimodal fusion feature maps of the same type to obtain multiple aggregated historical multimodal fusion feature map sets; perform base class and new class data partitioning based on the number of feature maps within the multiple aggregated historical multimodal fusion feature map sets to obtain a base class aggregated historical multimodal fusion feature map set and a new class aggregated historical multimodal fusion feature map set; and perform data partitioning according to the base class aggregated historical multimodal fusion feature map set. The multimodal fusion feature map set and the new class aggregated historical multimodal fusion feature map set are used to map and divide the multiple sample night view image sequences, multiple sample thermal imaging image sequences, and multiple sample target tracking trajectories to obtain a base class data set and a new class data set. The training sample set is used to train the model to obtain an initial target tracking model. The initial target tracking model is then trained with two-dimensional cueing using the base class data set and the new class data set to obtain a first cue vector and a second cue vector. The first cue vector and the second cue vector are stored in parallel in the semantic encoding module of the initial target tracking model to obtain the target tracking model.

[0079] Furthermore, the joint target tracking module 13 is also used to: load an initial cue vector from the semantic encoding module of the initial target tracking model as a common cue vector, copy it to generate a first initial cue vector and a second initial cue vector; based on the first initial cue vector and the second initial cue vector, perform two-dimensional cue-based training on the initial target tracking model using a base class dataset and a new class dataset to obtain base class model state parameters and a base class optimized cue vector, as well as new class model state parameters and a new class optimized cue vector; associate the base class optimized cue vector with the base class model state parameters to obtain a first cue vector; and associate the new class optimized cue vector with the new class model state parameters to obtain a second cue vector.

[0080] Furthermore, the joint target tracking module 13 is also used to: input the base class data set into the initial target tracking model, and under the condition that the main parameters of the model and the parameters of the second initial cue vector are frozen and not updated, only the first initial cue vector is allowed to participate in backpropagation, and after multiple rounds of optimization, obtain the base class optimized cue vector and the base class model state parameters; input the new class data set into the initial target tracking model, and under the condition that the main parameters of the model and the parameters of the first initial cue vector are frozen and not updated, only the second initial cue vector is allowed to participate in backpropagation, and after multiple rounds of optimization, obtain the new class optimized cue vector and the new class model state parameters.

[0081] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The dynamic target tracking method and specific examples based on night vision scope image processing in the aforementioned embodiment one are also applicable to the dynamic target tracking system based on night vision scope image processing in this embodiment. Through the foregoing detailed description of the dynamic target tracking method based on night vision scope image processing, those skilled in the art can clearly understand the dynamic target tracking system based on night vision scope image processing in this embodiment. Therefore, for the sake of brevity, it will not be described in detail here.

[0082] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0083] Obviously, those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A dynamic target tracking method based on night vision sight image processing, characterized in that, include: Within a preset window, the night vision detector and thermal imaging detector of the night vision scope are used to simultaneously acquire night vision images and thermal imaging images at the same time. Based on the pre-calibrated spatial mapping relationship, image time alignment and pixel-level registration are performed to obtain night vision image sequences and thermal imaging image sequences. Image preprocessing and multimodal feature fusion are performed by traversing the night vision image sequence and thermal imaging image sequence to obtain a multimodal fusion feature map; A pre-built target tracking model is invoked, and the first or second cue vector of the target tracking model is invoked based on the multimodal fusion feature map to perform joint target tracking on the night view image sequence and the thermal imaging image sequence to obtain the target tracking trajectory.

2. The dynamic target tracking method based on night vision sight image processing as described in claim 1, characterized in that, Based on the pre-calibrated spatial mapping relationship, image temporal alignment and pixel-level registration are performed to obtain night vision image sequences and thermal imaging image sequences, including: The internal parameters of the night vision detector and the thermal imaging detector were calibrated. After completing the internal parameter calibration, the external parameter relationship between the night vision detector and the thermal imaging detector is obtained based on the infrared-visible light joint calibration board; The extrinsic parameter relationship is used as a pre-calibrated spatial mapping relationship.

3. The dynamic target tracking method based on night vision sight image processing as described in claim 1, characterized in that, Image preprocessing and multimodal feature fusion are performed on the night vision image sequence and thermal imaging image sequence to obtain a multimodal fused feature map, including: The night view image sequence is subjected to low-light image enhancement preprocessing to obtain an enhanced night view image sequence; Non-uniformity calibration and thermal noise suppression processing are performed on the thermal imaging image sequence to obtain an enhanced thermal imaging image sequence; The enhanced night vision image sequence and the enhanced thermal imaging image sequence are subjected to one-to-one mapping and joint multimodal feature temporal processing to obtain the multimodal fusion feature map.

4. The dynamic target tracking method based on night vision sight image processing as described in claim 3, characterized in that, The night view image sequences are subjected to low-light image enhancement preprocessing to obtain enhanced night view image sequences, including: Based on the Retinex model, brightness separation processing is performed on the night view image sequence to obtain the illuminance component sequence; An enhanced night view image sequence is obtained by adaptive contrast stretching of the illuminance component sequence and combining it with a frequency domain denoising algorithm to suppress random noise.

5. The dynamic target tracking method based on night vision sight image processing as described in claim 3, characterized in that, The enhanced night vision image sequence and the enhanced thermal imaging image sequence are subjected to one-to-one mapping and joint multimodal feature temporal processing to obtain the multimodal fusion feature map, including: The enhanced night vision image sequence and the enhanced thermal imaging image sequence are subjected to one-to-one mapping and joint multimodal feature processing to obtain an initial multimodal fusion feature map sequence; Perform sequential trend overlay processing on the first initial multimodal fusion feature map and the second initial multimodal fusion feature map in the initial multimodal fusion feature map sequence to obtain the first overlaid multimodal fusion feature map; The first superimposed multimodal fusion feature map is used to perform a time-series trend superposition process on the third initial multimodal fusion feature map in the initial multimodal fusion feature map sequence to obtain the second superimposed multimodal fusion feature map, and so on, to obtain the multimodal fusion feature map.

6. The dynamic target tracking method based on night vision sight image processing as described in claim 5, characterized in that, Perform time-series trend overlay processing on the first initial multimodal fusion feature map and the second initial multimodal fusion feature map in the initial multimodal fusion feature map sequence to obtain the first overlaid multimodal fusion feature map, including: The first initial multimodal fusion feature map and the second initial multimodal fusion feature map are approximately analyzed from two dimensions: night vision features and thermal imaging features, respectively, to obtain the night vision feature similarity and thermal imaging feature similarity. The night vision feature similarity and thermal imaging feature similarity are normalized using the maximum-minimum normalization method, and the normalization result is filled into an initially empty two-dimensional diagonal matrix to construct a trend superposition matrix. The trend superposition matrix is ​​used to superimpose the second initial multimodal fusion feature map to obtain a first superimposed multimodal fusion feature map.

7. The dynamic target tracking method based on night vision sight image processing as described in claim 1, characterized in that, A pre-built target tracking model is invoked, and based on the multimodal fusion feature map, the first or second cue vector of the target tracking model is invoked to perform joint target tracking on the night view image sequence and the thermal imaging image sequence to obtain the target tracking trajectory, including: Multiple historical multimodal fusion feature maps, as well as multiple sample night view image sequences, multiple sample thermal imaging image sequences, and multiple sample target tracking trajectories, are obtained as training sample sets. The multiple historical multimodal fusion feature maps are aggregated in the same category to obtain multiple aggregated historical multimodal fusion feature map sets; Based on the size of the number of feature maps within the multiple aggregated historical multimodal fusion feature map sets, base class and new class data partitioning is performed to obtain the base class aggregated historical multimodal fusion feature map set and the new class aggregated historical multimodal fusion feature map set; Based on the base class aggregated historical multimodal fusion feature map set and the new class aggregated historical multimodal fusion feature map set, the multiple sample night view image sequences, multiple sample thermal imaging image sequences and multiple sample target tracking trajectories are mapped and divided to obtain the base class data set and the new class data set; The initial target tracking model is obtained by training the model using the training sample set. The initial target tracking model is trained using the base class dataset and the new class dataset to obtain a first cue vector and a second cue vector. The first cue vector and the second cue vector are stored in parallel to the semantic encoding module of the initial target tracking model to obtain the target tracking model.

8. The dynamic target tracking method based on night vision sight image processing as described in claim 7, characterized in that, The initial target tracking model is trained using the base class dataset and the new class dataset to obtain a first cue vector and a second cue vector, including: The initial cue vector is loaded from the semantic encoding module of the initial target tracking model as a common cue vector, and then copied to generate the first initial cue vector and the second initial cue vector; Based on the first initial cue vector and the second initial cue vector, the initial target tracking model is trained in two dimensions using the base class dataset and the new class dataset to obtain the base class model state parameters and the base class optimized cue vector, as well as the new class model state parameters and the new class optimized cue vector. The first suggestion vector is obtained by associating the base class optimization suggestion vector with the base class model state parameters. The new class optimization hint vector is obtained by associating the new class model state parameters with the new class optimization hint vector.

9. The dynamic target tracking method based on night vision sight image processing as described in claim 8, characterized in that, Based on the first and second initial cue vectors, the initial target tracking model is trained using a base class dataset and a new class dataset in a two-dimensional cue-based manner to obtain the base class model state parameters and the base class optimized cue vector, as well as the new class model state parameters and the new class optimized cue vector, including: The base class dataset is input into the initial target tracking model. Under the condition that the main parameters of the model and the parameters of the second initial cue vector are frozen and not updated, only the first initial cue vector is allowed to participate in backpropagation. After multiple rounds of optimization, the base class optimized cue vector and the base class model state parameters are obtained. The new dataset is input into the initial target tracking model. While keeping the main parameters of the model and the parameters of the first initial cue vector frozen and not updated, only the second initial cue vector is allowed to participate in backpropagation. After multiple rounds of optimization, the new optimized cue vector and the new model state parameters are obtained.

10. A dynamic target tracking system based on night vision sight image processing, characterized in that, The step of implementing the dynamic target tracking method based on night vision sight image processing according to any one of claims 1 to 9, wherein the dynamic target tracking system based on night vision sight image processing comprises: The image sequence acquisition module is used to simultaneously acquire night vision images and thermal imaging images at the same time using the night vision detector and thermal imaging detector of the night vision scope in a preset window, and perform image time alignment and pixel-level registration according to the pre-calibrated spatial mapping relationship to obtain night vision image sequence and thermal imaging image sequence. The image sequence processing module is used to traverse the night view image sequence and thermal imaging image sequence to perform image preprocessing and multimodal feature fusion to obtain a multimodal fusion feature map; The joint target tracking module is used to call a pre-built target tracking model, and based on the multimodal fusion feature map, call the first or second cue vector of the target tracking model to perform joint target tracking on the night view image sequence and the thermal imaging image sequence to obtain the target tracking trajectory.

Images