Mountainous area night animal tracking method and system based on multispectral and thermal imaging data fusion

Abstract

The application provides a mountainous area night animal tracking method and system based on multispectral and thermal imaging data fusion, and relates to the technical field of night animal tracking, and comprises the following steps: synchronously acquiring a multispectral image sequence and a thermal imaging image sequence in a mountainous area at night; performing multiscale Retinex enhancement processing on a registered near-infrared image and a thermal image respectively to obtain a strengthened candidate target image; extracting a multi-source texture signature of a fusion local binary pattern feature vector and a thermal gradient direction histogram for each candidate target, and combining geometric features obtained through morphological skeletonization to construct a multi-dimensional feature set; realizing target correlation between adjacent frames by using a nearest neighbor greedy matching algorithm to form a preliminary trajectory segment; and finally eliminating abnormal trajectories, solving trajectory intersection and occlusion conflicts through a sparse dynamic time warping algorithm, and generating an accurate and complete animal moving trajectory. The application can solve the conflicts caused by trajectory intersection and occlusion, and output continuous and accurate single animal moving trajectories.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of animal tracking, in particular to a mountainous area night animal tracking method and system based on multispectral and thermal imaging data fusion. BACKGROUND

[0002] In the field of mountainous area biodiversity monitoring and pest control, effective tracking of wild animals at night is a key challenge. In the prior art, single visible light imaging is basically ineffective in the night environment due to insufficient light; the tracking method based on thermal imaging can utilize the thermal radiation characteristics of animals, but in the complex environment of the mountains, it is easily affected by ground residual heat, rock and other heat interference sources, resulting in high false alarm rate, and it is difficult to distinguish different individuals with similar thermal characteristics; and the candidate target extraction accuracy is insufficient, and the unique texture and posture (skeleton) features of the animal individual are not deeply mined, resulting in low individual re-identification accuracy in complex situations such as trajectory intersection and occlusion, and easy to produce trajectory break or identity switching error.

[0003] Therefore, there is an urgent need for a mountainous area night animal tracking method and system based on multispectral and thermal imaging data fusion to solve the above technical problems. SUMMARY

[0004] The purpose of the present application is to provide a mountainous area night animal tracking method and system based on multispectral and thermal imaging data fusion to improve the above problems. In order to achieve the above purpose, the technical scheme adopted by the present application is as follows:

[0005] In a first aspect, the present application provides a mountainous area night animal tracking method based on multispectral and thermal imaging data fusion, comprising:

[0006] obtaining multi-source image data, the multi-source image data comprising multispectral image sequence data and thermal imaging image sequence data at night in the mountainous area, the multispectral image sequence data comprising visible light image frames and near-infrared image frames of multiple wavebands, and the thermal imaging image sequence data comprising infrared thermal distribution image frames varying with time;

[0007] performing enhancement processing based on the multi-source image data, wherein the uniform distribution of background brightness and heat is suppressed to obtain a strengthened candidate target image, and the candidate target image is an animal image;

[0008] performing texture feature encoding processing and morphological skeletonization processing based on the candidate target image, respectively extracting texture signatures and skeleton geometric features of each candidate region to obtain a multi-dimensional feature set of the animal;

[0009] performing matching processing based on the multi-dimensional feature set and a nearest neighbor greedy matching algorithm, connecting the candidate target images between adjacent frames to obtain a set of cross-frame connected trajectory segments;

[0010] Based on the connected track segment set after the frame, biological threshold filtering and sparse dynamic time warping processing are performed, wherein abnormal tracks are removed by preset speed, acceleration and heat intensity range constraints, and conflict resolution is performed based on time sequence consistency to obtain a moving track set of the animal.

[0011] In a second aspect, the application further provides a mountainous area night animal tracking system based on multispectral and thermal imaging data fusion, comprising:

[0012] An acquisition unit is configured to acquire multi-source image data, wherein the multi-source image data comprises multispectral image sequence data and thermal imaging image sequence data in a mountainous area at night, the multispectral image sequence data comprises visible light image frames and near-infrared image frames of multiple wavebands, and the thermal imaging image sequence data comprises infrared thermal distribution image frames varying with time;

[0013] A strengthening unit is configured to perform enhancement processing based on the multi-source image data, wherein the enhancement processing is performed by suppressing uniform distribution of background brightness and heat to obtain a strengthened candidate target image, and the candidate target image is an animal image;

[0014] A processing unit is configured to perform texture feature encoding processing and morphological skeletonization processing based on the candidate target image, to extract texture signatures and skeleton geometric features of each candidate region respectively, and to obtain a multi-dimensional feature set of the animal;

[0015] A connection unit is configured to perform matching processing based on the multi-dimensional feature set and a nearest neighbor greedy matching algorithm, to connect the candidate target images between adjacent frames, and to obtain a connected track segment set after the frame;

[0016] A filtering unit is configured to perform biological threshold filtering and sparse dynamic time warping processing based on the connected track segment set after the frame, wherein abnormal tracks are removed by preset speed, acceleration and heat intensity range constraints, and conflict resolution is performed based on time sequence consistency to obtain a moving track set of the animal.

[0017] The application has the following beneficial effects:

[0018] The present application processes near-infrared and thermal images respectively by a multi-scale Retinex algorithm, significantly suppresses background interference, and uses multiplicative fusion to accurately highlight real candidate targets; further, the near-infrared LBP feature representing animal fur texture, the thermal gradient direction histogram feature representing body surface heat distribution characteristics, and the skeleton geometry feature describing target shape and posture are creatively combined to extract a multi-dimensional feature set with strong discrimination to realize more reliable individual representation; finally, by introducing a biological kinematic threshold filtering and sparse dynamic time warping algorithm, abnormal trajectories that are physically impossible are effectively eliminated, and conflicts caused by trajectory intersection and occlusion can be intelligently solved based on time sequence consistency, so that continuous and accurate single animal movement trajectories are finally output.

[0019] Other features and advantages of the present application will be set forth in the following description, and in part will become apparent from the description, or can be learned by practice of the present application as hereinafter described. The objects and other advantages of the present application will be realized and attained by means of the instrumentalities particularly pointed out in the written description and claims hereof as well as the appended drawings. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some of the embodiments of the present application, and therefore should not be considered as limiting the scope. For those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0021] Figure 1 Flowchart of the mountainous area night animal tracking method of multispectral and thermal imaging data fusion described in the embodiments of the present application;

[0022] Figure 2 Structure diagram of the mountainous area night animal tracking system of multispectral and thermal imaging data fusion described in the embodiments of the present application.

[0023] In the figure: 701, acquisition unit; 702, strengthening unit; 703, processing unit; 704, connection unit; 705, filtering unit. DETAILED DESCRIPTION

[0024] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present application.

[0025] It should be noted that similar reference numerals and letters represent similar items in the following drawings, and therefore, once an item is defined in one drawing, it need not be further defined and explained in subsequent drawings. Meanwhile, in the description of the present application, the terms “first”, “second”, and the like are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0026] Embodiment 1

[0027] The present embodiment provides a mountainous area night animal tracking method based on multispectral and thermal imaging data fusion.

[0028] Referring to Figure 1 , the present method includes steps S1, S2, S3, S4 and S5.

[0029] Step S1, acquiring multi-source image data, the multi-source image data including multispectral image sequence data and thermal imaging image sequence data in a mountainous area at night, the multispectral image sequence data including visible light image frames and near-infrared image frames of multiple wavebands, and the thermal imaging image sequence data including infrared thermal distribution image frames changing over time;

[0030] It can be understood that this step is performed by deploying a multispectral camera and a thermal imaging camera in the field for data acquisition. The multispectral camera can acquire images in the near-infrared waveband, because the animal fur has high reflection characteristics in the near-infrared region, and can still form images with high contrast with the background vegetation and soil under weak moonlight or complete absence of visible light. At the same time, the thermal imaging camera captures medium-long wave infrared (8 μm-14 μm) data, directly sensing the surface temperature distribution of each object in the scene, thereby effectively highlighting the thermal radiation difference of homeothermic animals relative to the environment.

[0031] The present step provides an information complementary raw data source for subsequent processing: the multispectral image (especially near-infrared) can provide rich texture and shape details, and the thermal imaging data can reliably indicate the position of the living body, and the combination of the two lays a solid foundation for stably finding potential animal targets in complex and dark mountainous night scenes.

[0032] Step S2, performing enhancement processing based on the multi-source image data, wherein the enhanced candidate target image is obtained by suppressing the uniform distribution of background brightness and heat, and the candidate target image is an animal image;

[0033] It can be understood that the present step generates a binary image (black and white image) with significantly improved signal-to-noise ratio by performing enhancement processing on the multi-source image data, wherein the connected white area accurately identifies the position and shape outline of the potential animal target with high confidence in the image, laying a solid foundation for subsequent feature extraction of each target, effectively reducing the processing burden and misjudgment risk of the subsequent steps. In the present step, step S2 includes step S21, step S22 and step S23.

[0034] Step S21, performing feature point-based spatial registration based on the near-infrared image in the multispectral image sequence data and the thermal image in the thermal imaging image sequence data, to obtain a registered near-infrared-thermal image pair;

[0035] It can be understood that the feature points in the present step are rock edge points detected by using a scale-invariant feature transform (SIFT) algorithm, which are used as coordinate origins to perform equal scaling or stretching of the near-infrared image and the thermal imaging image, thereby solving the problem of geometric inconsistency of images caused by differences in physical position, optical characteristics and imaging principles of different imaging devices.

[0036] Step S22, performing multi-scale Retinex enhancement and background suppression processing based on the registered near-infrared-thermal image pair, wherein a Gaussian wrap function with a preset scale parameter is applied to the near-infrared image and the thermal image respectively to calculate the illumination component estimate, and the reflection component image of the near-infrared image and the thermal reflection component image of the thermal image are obtained by subtracting the illumination component logarithmic domain from the original image logarithmic domain.

[0037] It can be understood that this step first reads the registered near-infrared-thermal image pair, converts it to floating-point number format and converts it to a logarithmic domain to convert the multiplicative relationship to an additive relationship. For each image, three different scale Gaussian kernel parameters (such as Gaussian kernel parameters 15, 80, 250) are preset, corresponding to detail, region and background scales respectively. Through convolution operation, three different scale illumination component estimates are obtained, and then weighted fusion is performed according to the preset weight to obtain the final illumination component map. In the logarithmic domain, the original image is subtracted from the illumination component map to obtain the reflection component map. Finally, linear stretching and exponential operation are performed to restore to the real number domain, and the near-infrared reflection component image and the thermal reflection component image are output respectively. This process realizes effective suppression of large-area uniform background through multi-scale illumination component estimation and weighted fusion mechanism, while retaining target detail information.

[0038] Step S23, performing multiplicative fusion and adaptive threshold segmentation processing based on the reflection component image and the thermal reflection component image, wherein the reflection component image of the near-infrared image and the thermal reflection component image are subjected to pixel-level multiplication operation, and the fusion result image after pixel-level multiplication operation is subjected to image binarization processing to obtain an enhanced candidate target image.

[0039] It can be understood that this step first performs pixel-level normalization processing on the near-infrared reflection component image and the thermal reflection component image obtained through multi-scale Retinex processing, to ensure that both are in the same numerical range. Then, pixel-level multiplication operation is performed: for each spatial position, the normalized near-infrared reflection component value and the thermal reflection component value are multiplied to obtain a fusion result. This multiplicative fusion operation fully utilizes the characteristic that the animal target simultaneously presents high response in the near-infrared band and the thermal infrared band, and enhances the saliency of the dual-modality consistent region through multiplication operation. Subsequently, histogram statistics are performed on the fusion image, and the Otsu threshold method is used to automatically calculate the best segmentation threshold: by traversing all possible thresholds, the inter-class variance is calculated, and the threshold that maximizes the inter-class variance is selected as the segmentation point. Finally, the threshold is applied to the binarization processing of the fusion image, and the pixels greater than the threshold are set to 255 (target, i.e. animal), and the pixels less than the threshold are set to 0 (background), to generate the final binarized candidate target image. Through the continuous processing of multiplicative fusion and adaptive threshold segmentation in this step, the real target region is effectively highlighted, providing high-quality input data for subsequent feature extraction.

[0040] Step S3, performing texture feature encoding processing and morphological skeletonization processing based on the candidate target image, extracting the texture signature and the skeleton geometric feature of each candidate region respectively, to obtain a multi-dimensional feature set of the animal;

[0041] It can be understood that this step constructs a feature representation with strong discriminability through multi-modal feature fusion, and provides a reliable identity identification basis for subsequent target association. In this step, step S3 includes steps S31, S32, S33 and S34.

[0042] Step S31, performing connected region division processing on the enhanced candidate target image, wherein the enhanced candidate target image is divided into connected regions, each candidate target region is assigned a unique identifier, and the corresponding near-infrared image block and thermal image block in the multi-source image data are extracted to obtain a candidate target region set;

[0043] It can be understood that this step first performs full-image scanning on the binary candidate target image, and uses an 8-connected region marking algorithm (two-pass scanning method) to perform connected region detection. In the first pass, the pixels are traversed by row, and when a foreground pixel (white pixel) is encountered, the marking status of the four adjacent pixels above, above, above, and above the left of the foreground pixel is checked: if all four are background, a new label is assigned; if there is a marked adjacent pixel, the minimum value is taken as the current pixel label, and the equivalence relationship is recorded. In the second pass, the label numbers are unified according to the equivalence relationship table to ensure that each connected region obtains a unique identifier. After marking is completed, the bounding rectangle coordinates of each connected region are calculated, and the corresponding image blocks are extracted from the registered near-infrared image and thermal image according to the coordinate information. To improve the robustness of subsequent feature extraction, the extracted image blocks are normalized in size and adjusted to a fixed size (such as 64x64 pixels), while preserving the position, area, and other metadata of each target. The final structured data set contains the unique ID of each target, the normalized near-infrared image block, the thermal image block, and the spatial attribute information. This step realizes target individual separation and multi-modal data association through connected region analysis, and provides standardized input data for subsequent feature extraction.

[0044] Step S32, performing multi-source texture feature encoding processing based on the candidate target region set, wherein the local binary pattern feature vector of the near-infrared image block corresponding to each candidate target region is calculated, the thermal gradient direction histogram feature vector of the thermal image block corresponding to each candidate target region is calculated, and the local binary pattern feature vector and the thermal gradient direction histogram feature vector are spliced to obtain the texture signature of each candidate target;

[0045] Understandably, this step first extracts Local Binary Pattern (LBP) features from the near-infrared image patch corresponding to each candidate target region. A circular neighborhood sampling method is used, with each pixel as the center, P=16 sampling points are uniformly collected on a circle with a radius R=3. The sampling values ​​are obtained through bilinear interpolation. The center point value is compared with the neighboring point values ​​to generate an 8-bit binary pattern. Rotation-invariant uniform pattern (rotation-invariant LBP) features are calculated, resulting in a 59-dimensional texture histogram. Simultaneously, thermal gradient orientation histogram (Thermal-HOG) features are calculated for the thermal image patch: first, the Sobel operator is used to calculate the temperature gradient magnitude and direction, quantizing the direction into 9 intervals (0°-180°). The gradient direction is weighted and statistically analyzed within each cell unit (8×8 pixels), and trilinear interpolation is used to smooth the position and orientation dimensions. Finally, the contrast of the cell units within the patch is normalized. The obtained 59-dimensional LBP features and 36-dimensional Thermal-HOG features are L2 normalized and then concatenated into a 95-dimensional fused feature vector. This step constructs a texture signature with strong discriminative power by fusing visible light texture and thermal distribution features, providing a robust feature representation for subsequent target recognition.

[0046] Step S33: Binarize the near-infrared image block and thermal image block in each candidate target region set, and perform thinning algorithm iterative processing on the binary image of each candidate target region obtained by the processing to obtain a skeleton with a single pixel width. The skeleton trunk length, skeleton branch number, skeleton length and the relative position of the highest temperature point on the skeleton are used as the skeleton geometric feature vector of the corresponding candidate target.

[0047] Understandably, this step first performs adaptive threshold binarization on the normalized near-infrared image patch and thermal image patch, using a local window to calculate the optimal threshold and generate a binary image. Then, morphological skeletonization is performed on the binary image, through an iterative thinning algorithm: in each iteration, image pixels are scanned, and boundary points that meet the deletion criteria (such as southeast and northwest boundary points) are deleted while maintaining topological connectivity. This process continues until a skeleton structure with a single pixel width is obtained. Next, the geometric features of the skeleton are extracted: the main skeleton path (longest branchless path) is identified through depth-first search, and its pixel length is calculated as the main skeleton length; the number of branch points and the length of each branch are counted; simultaneously, the highest temperature point in the thermal image patch is located, and the shortest Euclidean distance from this point to the main skeleton and its relative position ratio along the skeleton are calculated. Finally, these features are quantized into a geometric feature vector: main skeleton length, number of branches, average branch length, and relative position of the highest temperature point. This step abstracts the target shape into a topological structure through skeletonization, and the extracted geometric features are highly robust to changes in target pose.

[0048] Step S34, the skeleton geometry feature vector and the texture signature of the candidate target region are standardized and vector fusion processed to obtain a multi-dimensional feature set containing the skeleton geometry feature vector and the texture signature of the candidate target.

[0049] It can be understood that this step first standardizes the 95-dimensional texture signature vector (containing LBP and Thermal-HOG features) and the 4-dimensional skeleton geometry feature vector respectively. The Z-score standardization method is adopted, and the mean and standard deviation of each feature dimension are calculated separately. Then the standardized feature vectors are fused by weighting, and the weight coefficients are assigned according to the feature importance (texture feature weight 0.7, geometry feature weight 0.3). Then the fused feature vector is normalized to the interval [0, 1] by using the maximum and minimum scaling. Finally, the processed feature vector is spliced in a fixed dimension order to form a unified 99-dimensional multi-dimensional feature vector, and a mapping relationship table between the feature vector and the target identifier is constructed. Through feature standardization and weighted fusion processing in this step, the dimensional difference of different feature quantities is eliminated, the discriminative feature components are highlighted, and a multi-dimensional feature set suitable for subsequent similarity calculation is generated.

[0050] Step S4, based on the multi-dimensional feature set and the nearest neighbor greedy matching algorithm, a matching process is performed to connect the candidate target images between adjacent frames to obtain a connected track segment set after cross-frame connection;

[0051] It can be understood that this step realizes target cross-frame correspondence through feature-driven data association, and provides a preliminary spatio-temporal association result for subsequent trajectory optimization. In this step, step S4 includes step S41, step S42, step S43 and step S44.

[0052] Step S41, based on the multi-dimensional feature set, a trajectory segment initialization process is performed, wherein the multi-dimensional feature set corresponding to the first frame of multi-source image data is created as a trajectory segment containing only one target point, and an initial trajectory segment set is obtained, the target point being a point containing all features in the multi-dimensional feature set;

[0053] Understandably, this step first reads the processing result of the first frame image (the frame with the earliest acquisition time) and obtains a multi-dimensional feature set of all candidate targets in that frame (each target corresponds to a 99-dimensional feature vector). An independent trajectory segment data structure is created for each detected target. This structure contains the following core fields: trajectory ID (an automatically generated unique identifier, such as UUID), start timestamp (the acquisition time of the current frame), and trajectory point sequence (initially containing only the current target point). Each trajectory point records detailed attributes: spatial coordinates (the pixel position of the target center in the original image), time offset (milliseconds relative to the start timestamp), and multi-dimensional feature vector (a numerical array storing the 99-dimensional features). Simultaneously, the trajectory status flag is initialized to "active". Finally, a set of initial trajectory segments containing the number of target points from the first frame is generated and stored as an ordered list or hash table structure, with the trajectory ID as the index key. This step transforms statically detected targets into extensible trajectory objects through structured encapsulation, establishing an initial tracking context for subsequent inter-frame matching.

[0054] Step S42: Perform inter-frame feature distance matrix calculation based on the multi-dimensional feature set corresponding to the candidate target images of two adjacent frames. The inter-frame feature distance matrix is ​​obtained by calculating the Euclidean distance between the multi-dimensional feature vector of each candidate target in the previous frame and the multi-dimensional feature vector of each candidate target in the next frame.

[0055] Understandably, this step first involves the system acquiring M trajectory points from the previous frame (each point corresponding to a 99-dimensional feature vector) and N detected targets from the (k+1)th frame (each target also corresponding to a 99-dimensional feature vector). Next, an M×N inter-frame feature distance matrix is ​​constructed, where the distance calculation uses the weighted Euclidean distance formula: ;

[0056] in, For the first O The weighted Euclidean distance between the p-th target and the p-th target, where n is the total dimension of the feature distance, a is the feature dimension index, and w a Let be the weight coefficient of the a-th feature. For the first O The numerical value of each target on the a-th dimension feature Let be the value of the p-th target in the a-th dimension feature.

[0057] Wherein, the weight coefficient is preset according to the feature component discriminant (texture feature weight 0.4, hot texture weight 0.3, skeleton geometry feature weight 0.3). In the calculation process, the system will dynamically exclude obviously unreasonable matching pairs (such as target pairs whose spatial distance exceeds the maximum motion speed constraint), and set their distance to be infinite. The finally generated distance matrix contains feature similarity information and spatial constraint information, which provides a quantitative basis for subsequent greedy matching. This step improves the accuracy and robustness of cross-frame target association through multi-dimensional feature weighted distance calculation.

[0058] Step S43, based on the inter-frame feature distance matrix and the nearest neighbor greedy matching algorithm, the matching processing is carried out, and the matching pair set and the unmatching candidate target set between adjacent frames are obtained.

[0059] It can be understood that this step finds the candidate target pair with the smallest feature distance in the distance matrix by circulation, and when the minimum distance is less than the preset species-related maximum matching threshold, it is determined that the target pair is the same animal individual, and a matching connection is established. Then the matched target is removed from the distance matrix, and the process is iterated until there is no candidate target pair that meets the threshold condition. This step realizes optimal matching through the greedy strategy and effectively handles the target temporary occlusion problem. Wherein, the nearest neighbor greedy matching algorithm is based on the nearest neighbor algorithm. For a target in the current frame, find the target with the closest feature distance (most similar) in the next frame. If the distance is less than a certain threshold, it is considered that they are the same individual. The greedy algorithm is used as the execution strategy. The specific steps are to find the smallest distance in the current matrix (i.e. the most matched target pair), confirm whether the distance is less than the preset threshold. If yes, confirm the matching of the pair. And immediately remove the matched target from the matrix (i.e. cross out the corresponding row and column), and no longer consider the matching possibility of the target with other targets. In the new matrix, repeat the above steps until no matching pair that meets the condition is found. The technical effect is to generate accurate inter-frame target correspondence, which provides guarantee for trajectory continuity.

[0060] Step S44, based on the matching pair set and the unmatching candidate target set between adjacent frames, the trajectory segment updating processing is carried out, and the connected trajectory segment set after cross-frame is obtained.

[0061] It can be understood that this step first loads the matching pair set and the unmatched target set between adjacent frames into the system. For each matching pair, the system performs a track extension operation: according to the track ID in the matching pair, the corresponding track is located in the track segment set, the newly matched target point (including spatial coordinates, timestamp, feature vector) is added to the end of the point sequence of the track, and the termination timestamp and motion state of the track are updated. For the unmatched historical track, its state is marked as "temporarily ended" but still remains in the set, and the last visible timestamp is recorded to provide an opportunity for subsequent possible re-association. Among them, for the unmatched new target, a brand new track segment is created, a unique track ID is generated, and the track point sequence is initialized. The final output of the updated track segment set contains three types of tracks: extended active tracks, temporarily ended tracks, and newly initialized tracks. This step realizes the continuity maintenance of cross-frame target association by dynamically managing the track life cycle, and provides a complete spatio-temporal data basis for subsequent track optimization.

[0062] Step S5, based on the cross-frame connected track segment set, biological threshold filtering and sparse dynamic time warping processing are performed, wherein abnormal tracks are removed by presetting speed, acceleration and heat intensity range constraints, and conflict resolution is performed based on time sequence consistency to obtain a set of animal movement tracks.

[0063] It can be understood that in this step, step S5 includes step S51, step S52, step S53 and step S54.

[0064] Step S51, based on the displacement of the target points between consecutive frames in each cross-frame connected track segment, the instantaneous speed sequence and the acceleration sequence are calculated, and the average heat intensity value in the thermal imaging data corresponding to each track point in each cross-frame connected track segment is calculated.

[0065] It can be understood that this step first performs time series data analysis on each track segment to extract the spatial displacement between consecutive frames. The Euclidean distance (in pixels) between adjacent track points is calculated to obtain the instantaneous speed sequence. The acceleration sequence is calculated by differentiating the speed sequence. At the same time, the thermal image block corresponding to each track point is subjected to radiation calibration processing, the pixel gray value is converted into absolute temperature value, the average heat intensity value in the target region is calculated and the timestamp is recorded. In order to eliminate measurement noise, the speed sequence and the acceleration sequence are respectively smoothed by using a sliding window mean filter (window size is 5 frames). Finally, the track dynamics parameter sequence including timestamp, instantaneous speed, acceleration and average heat intensity is generated, which provides a quantitative analysis basis for subsequent biological reasonableness test.

[0066] Step S52, compare the instantaneous speed sequence, acceleration sequence and average heat intensity value with the preset threshold range of animal movement data respectively, delete the trajectory segment not in the threshold range, and obtain a set of trajectory segments filtered by biological threshold;

[0067] It can be understood that the preset species biological parameter threshold table is first loaded (for example, Cervidae: speed range 0.5 m / s-8.0 m / s, acceleration range 0 m / s 2 -3.5 m / s 2 , body surface temperature range 30℃-40℃). Each trajectory segment is subjected to three-criteria test: the first criterion is speed test: the mean and maximum of the speed sequence of the whole trajectory are calculated, and if the mean is lower than the minimum activity speed (0.5 m / s) or the maximum exceeds the burst speed (8.0 m / s), it is marked as abnormal; the second criterion is acceleration test: the extreme value of the acceleration sequence is detected, and the trajectory appearing more than the physiological limit of the animal (3.5 m / s 2 ) is removed; the third criterion is heat intensity test: the average temperature of the trajectory is calculated, and the trajectory exceeding the reasonable body temperature range of homeothermic animals (30℃-40℃) is removed. Any parameter exceeding the standard will result in the removal of the whole trajectory. This step effectively removes the noise trajectory that does not conform to the animal behavior rule through multi-dimensional biological constraint verification.

[0068] Step S53, the set of trajectory segments filtered by biological threshold is subjected to trajectory conflict segment detection processing, in which whether there is a trajectory intersection, proximity or overlap of target points between two trajectory segments in the space-time domain is detected, and if so, the corresponding trajectory segment is marked as a conflict trajectory segment to be regularized;

[0069] It can be understood that this step first detects the space-time proximity of the filtered trajectory segments, and calculates the spatial overlap and time overlap of any two trajectory segments. For trajectory pairs with time intersection, sampling is performed at a fixed time interval (such as every second), and the Euclidean distance between trajectory points at the same time is calculated. When the distance between consecutive sampling points is less than the adaptive distance threshold (such as 10 pixels), it is marked as a trajectory proximity segment; when the distance between trajectory points is less than the average size of the target, it is marked as a trajectory overlap segment. The change of movement direction is also detected, and when the angle between the vectors of the two trajectories is less than 45 degrees and the average distance continues to decrease, it is marked as a potential intersection segment. The system uses trajectory envelope algorithm to construct a conflict feature vector containing time stamp, spatial coordinate and movement vector, and finally generates a conflict trajectory segment list and records the conflict type, start and end time and spatial position. This step accurately identifies the trajectory conflict segment that needs subsequent regularizing processing through multi-dimensional space-time relationship analysis.

[0070] Step S54, calculate the minimum cumulative distance of all to-be-normalized conflict trajectory segments based on a dynamic time warping algorithm, if the minimum cumulative distance is less than a preset consistency threshold, determine that the corresponding trajectory segment belongs to the same individual, and merge the corresponding trajectory segment to obtain a moving trajectory set of each animal.

[0071] It can be understood that this step first extracts the multi-dimensional feature sequence in the overlapping time window of each conflict trajectory segment, including texture features, skeleton geometric features and thermal features, and uniformly samples to the same length. Then the cumulative cost matrix of dynamic time warping is constructed, and the optimal bending path is found by recursive calculation, wherein the calculation formula is as follows: ;

[0072] Wherein D(i,j) is the element of the i-th row and the j-th column of the cumulative distance matrix, that is, the minimum cumulative alignment cost between the i-th point of the first sequence and the j-th point of the other sequence; cost(i,j) is the feature difference (Euclidean distance) between the i-th point of the first sequence and the j-th point of the other sequence, D(i-1,j) is the minimum cumulative alignment cost between the i-1-th point of the first sequence and the j-th point of the other sequence, D(i,j-1) is the minimum cumulative alignment cost between the i-th point of the first sequence and the j-1-th point of the other sequence, and D(i-1,j-1) is the minimum cumulative alignment cost between the i-1-th point of the first sequence and the j-1-th point of the other sequence.

[0073] The cost function in this step is calculated by using the Euclidean distance calculation formula. When the minimum cumulative distance obtained is lower than the preset threshold (such as 0.25), the system performs a trajectory merging operation: taking the earliest appearing trajectory as the reference, using a sliding window interpolation algorithm to fuse the trajectory point coordinates, and reconstructing the continuous speed and acceleration sequence. The merged trajectory inherits all the properties of the merged trajectory, while updating the trajectory confidence score. This step effectively solves the identity switching problem caused by trajectory intersection and occlusion through multi-feature time sequence alignment technology.

[0074] Embodiment 2:

[0075] As shown in Figure 2 , the embodiment provides a mountainous area night animal tracking system for multispectral and thermal imaging data fusion, referring to Figure 2 The system comprises an acquisition unit 701, a reinforcement unit 702, a processing unit 703, a connection unit 704 and a filtering unit 705.

[0076] The acquisition unit 701 is used to acquire multi-source image data, which includes multispectral image sequence data and thermal imaging image sequence data of mountainous nighttime. The multispectral image sequence data includes visible light image frames and near-infrared image frames of multiple bands, and the thermal imaging image sequence data includes infrared thermal distribution image frames that change over time.

[0077] The enhancement unit 702 is used to perform enhancement processing based on the multi-source image data, wherein an enhanced candidate target image is obtained by suppressing the uniform distribution of background brightness and heat, and the candidate target image is an animal image;

[0078] Processing unit 703 is used to perform texture feature encoding and morphological skeletonization processing based on the candidate target image, and extract the texture signature and skeleton geometric features of each candidate region to obtain a multi-dimensional feature set of the animal.

[0079] The connection unit 704 is used to perform matching processing based on the multidimensional feature set and the nearest neighbor greedy matching algorithm, and to connect the candidate target images between adjacent frames across frames to obtain a set of trajectory segments after cross-frame connection.

[0080] The filtering unit 705 is used to perform biological threshold filtering and sparse dynamic time warping processing on the set of trajectory segments after cross-frame connection. Abnormal trajectories are eliminated by pre-set speed, acceleration and heat intensity range constraints, and conflict resolution is performed based on time series consistency to obtain the set of animal movement trajectories.

[0081] It should be noted that the specific methods by which each module performs operations in the system described in the above embodiments have been described in detail in the embodiments related to the method, and will not be elaborated here.

[0082] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0083] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for tracking animals at night in mountainous areas by fusing multispectral and thermal imaging data, characterized in that, include: Acquire multi-source image data, which includes multispectral image sequence data and thermal imaging image sequence data of mountainous nighttime. The multispectral image sequence data includes visible light image frames and near-infrared image frames of multiple bands, and the thermal imaging image sequence data includes infrared thermal distribution image frames that change over time. Enhancement processing is performed based on the multi-source image data, wherein enhanced candidate target images are obtained by suppressing the uniform distribution of background brightness and heat, and the candidate target images are animal images; Based on the candidate target image, texture feature encoding and morphological skeletonization are performed to extract the texture signature and skeleton geometric features of each candidate region, thereby obtaining a multidimensional feature set of the animal. Based on the multidimensional feature set and the nearest neighbor greedy matching algorithm, matching processing is performed to connect candidate target images between adjacent frames to obtain a set of trajectory segments after cross-frame connection. Based on the set of trajectory segments after cross-frame connection, biological threshold filtering and sparse dynamic time warping are performed. Abnormal trajectories are eliminated by pre-set constraints on velocity, acceleration and thermal intensity ranges, and conflict resolution is performed based on time series consistency to obtain the set of animal movement trajectories. Specifically, based on the candidate target image, texture feature encoding and morphological skeletonization processes are performed to extract the texture signature and skeleton geometric features of each candidate target region, including: The enhanced candidate target image is subjected to connected component segmentation. Specifically, the enhanced candidate target image is segmented into connected components, and a unique identifier is assigned to each candidate target region. The near-infrared image block and thermal image block in the corresponding multi-source image data are extracted to obtain a set of candidate target regions. Multi-source texture feature encoding is performed based on the set of candidate target regions. Specifically, the local binary mode feature vector of the near-infrared image block corresponding to each candidate target region and the thermal gradient direction histogram feature vector of the thermal image block corresponding to each candidate target region are calculated. The local binary mode feature vector and the thermal gradient direction histogram feature vector are then concatenated to obtain the texture signature of each candidate target. The near-infrared image patch and thermal image patch in each candidate target region set are binarized, and the resulting binary image of each candidate target region is subjected to a thinning algorithm iterative processing to obtain a skeleton with a single pixel width. The skeleton trunk length, skeleton branch number, skeleton length, and the relative position of the highest temperature point on the skeleton are used as the skeleton geometric feature vector of the corresponding candidate target. The skeleton geometric feature vector and texture signature of the candidate target region are standardized and fused to obtain a multi-dimensional feature set containing the skeleton geometric feature vector and texture signature of the candidate target. The biological threshold filtering and sparse dynamic time warping processing based on the trajectory segment set after cross-frame connection includes: Based on the displacement of the target points in consecutive frames in the trajectory segment after each cross-frame connection, the instantaneous velocity sequence and acceleration sequence are calculated, and the average thermal intensity value is calculated based on the thermal imaging data corresponding to each trajectory point in the trajectory segment after each cross-frame connection. The instantaneous velocity sequence, acceleration sequence, and average thermal intensity value are compared with the preset threshold range of animal motion data, and trajectory segments that are not within the threshold range are deleted to obtain a set of trajectory segments after biological threshold filtering. The trajectory segment set filtered by biological thresholds is subjected to trajectory conflict segment detection processing. Specifically, in the spatiotemporal domain, it is detected whether there are segments where the trajectory of the target point intersects, approaches or overlaps between any two trajectory segments. If so, the corresponding trajectory segment is marked as a conflict trajectory segment to be regularized. The minimum cumulative distance of all conflict trajectory segments to be normalized is calculated based on the dynamic time warping algorithm. If the minimum cumulative distance is less than the preset consistency threshold, it is determined that the corresponding trajectory segments belong to the same individual, and the corresponding trajectory segments are merged to obtain the movement trajectory set of each animal.

2. The method for tracking animals at night in mountainous areas by fusing multispectral and thermal imaging data according to claim 1, characterized in that... Enhancement processing is performed based on the multi-source image data, wherein enhanced candidate target images are obtained by suppressing the uniform distribution of background brightness and heat, including: Based on the near-infrared image in the multispectral image sequence data and the thermal image in the thermal imaging image sequence data, spatial registration based on feature points is performed to obtain a registered near-infrared-thermal image pair. Multi-scale Retinex enhancement and background suppression processing are performed on the registered near-infrared-thermal image pair. Specifically, Gaussian wrap function with preset scale parameters is applied to the near-infrared image and the thermal image respectively to calculate their illumination component estimates. The reflection component image of the near-infrared image and the thermal reflection component image of the thermal image are obtained by subtracting the logarithmic domain of the illumination component from the logarithmic domain of the original image. Multiplicative fusion and adaptive threshold segmentation are performed based on the reflection component image and the thermal reflection component image. Specifically, pixel-level multiplication is performed on the reflection component image of the near-infrared image and the thermal reflection component image, and image binarization is performed on the fusion result image after pixel-level multiplication to obtain the enhanced candidate target image.

3. The method for tracking animals at night in mountainous areas by fusing multispectral and thermal imaging data according to claim 1, characterized in that... Based on the multidimensional feature set and the nearest neighbor greedy matching algorithm, matching processing is performed to connect candidate target images between adjacent frames across frames, including: The trajectory segment initialization process is based on a multidimensional feature set. Specifically, the initial trajectory segment set is obtained by creating a trajectory segment containing only one target point from the multidimensional feature set corresponding to the first frame of multi-source image data. The target point is a point that contains all features in the multidimensional feature set. The inter-frame feature distance matrix is ​​calculated based on the multi-dimensional feature set corresponding to the candidate target images of two adjacent frames. The inter-frame feature distance matrix is ​​obtained by calculating the Euclidean distance between the multi-dimensional feature vector of each candidate target in the previous frame and the multi-dimensional feature vector of each candidate target in the next frame. Matching is performed based on the inter-frame feature distance matrix and the nearest neighbor greedy matching algorithm to obtain a set of matched pairs and a set of unmatched candidate targets between adjacent frames. The trajectory segment update process is performed based on the set of matching pairs between adjacent frames and the set of unmatched candidate targets to obtain the set of trajectory segments after cross-frame connection.

4. A mountain nocturnal animal tracking system based on the fusion of multispectral and thermal imaging data, characterized in that, include: The acquisition unit is used to acquire multi-source image data, which includes multispectral image sequence data and thermal imaging image sequence data of mountainous nighttime. The multispectral image sequence data includes visible light image frames and near-infrared image frames of multiple bands, and the thermal imaging image sequence data includes infrared thermal distribution image frames that change over time. An enhancement unit is used to perform enhancement processing based on the multi-source image data, wherein an enhanced candidate target image is obtained by suppressing the uniform distribution of background brightness and heat, and the candidate target image is an animal image; The processing unit is used to perform texture feature encoding and morphological skeletonization processing based on the candidate target image, and extract the texture signature and skeleton geometric features of each candidate region to obtain a multi-dimensional feature set of the animal. The connection unit is used to perform matching processing based on the multidimensional feature set and the nearest neighbor greedy matching algorithm, and to connect the candidate target images between adjacent frames to obtain a set of trajectory segments after cross-frame connection. The filtering unit is used to perform biological threshold filtering and sparse dynamic time warping processing on the set of trajectory segments after cross-frame connection. Abnormal trajectories are eliminated by pre-set speed, acceleration and heat intensity range constraints, and conflict resolution is performed based on time series consistency to obtain the set of animal movement trajectories. The processing unit includes: The enhanced candidate target image is subjected to connected component segmentation. Specifically, the enhanced candidate target image is segmented into connected components, and a unique identifier is assigned to each candidate target region. The near-infrared image block and thermal image block in the corresponding multi-source image data are extracted to obtain a set of candidate target regions. The first processing subunit is used to perform multi-source texture feature encoding processing based on the candidate target region set. The local binary mode feature vector of the near-infrared image block corresponding to each candidate target region and the thermal gradient direction histogram feature vector of the thermal image block corresponding to each candidate target region are calculated. The local binary mode feature vector and the thermal gradient direction histogram feature vector are concatenated to obtain the texture signature of each candidate target. The second processing subunit is used to binarize the near-infrared image blocks and thermal image blocks in each candidate target region set, and to perform thinning algorithm iterative processing on the binary image of each candidate target region obtained by the processing to obtain a skeleton with a single pixel width. The skeleton trunk length, skeleton branch number, skeleton length and the relative position of the highest temperature point on the skeleton are used as the skeleton geometric feature vector of the corresponding candidate target. The third processing subunit is used to standardize and fuse the skeleton geometric feature vector and texture signature of the candidate target region to obtain a multi-dimensional feature set containing the skeleton geometric feature vector and texture signature of the candidate target. The filtering unit includes: The first filtering subunit is used to calculate the instantaneous velocity sequence and acceleration sequence based on the displacement of the target points in the continuous inter-frame trajectory segment after each cross-frame connection, and to calculate the average thermal intensity value based on the thermal imaging data corresponding to each trajectory point in the trajectory segment after each cross-frame connection. The second filtering subunit is used to compare the instantaneous velocity sequence, acceleration sequence and average heat intensity value with the preset threshold range of animal motion data, delete the trajectory segments that are not in the threshold range, and obtain the set of trajectory segments after biological threshold filtering. The third filtering subunit is used to perform trajectory conflict segment detection processing on the trajectory segment set after biological threshold filtering. Specifically, it detects whether there are segments in the spatiotemporal domain where the trajectory of the target point intersects, approaches or overlaps between two trajectory segments. If so, the corresponding trajectory segment is marked as a conflict trajectory segment to be regularized. The fourth filtering subunit is used to calculate the minimum cumulative distance of all conflict trajectory segments to be normalized based on the dynamic time warping algorithm. If the minimum cumulative distance is less than a preset consistency threshold, it is determined that the corresponding trajectory segments belong to the same individual, and the corresponding trajectory segments are merged to obtain the movement trajectory set of each animal.

5. The mountain nocturnal animal tracking system based on multispectral and thermal imaging data fusion according to claim 4, characterized in that, The strengthening unit includes: The first acquisition subunit is used to perform spatial registration based on feature points based on the near-infrared image in the multispectral image sequence data and the thermal image in the thermal imaging image sequence data to obtain a registered near-infrared-thermal image pair. The second acquisition subunit is used to perform multi-scale Retinex enhancement and background suppression processing based on the registered near-infrared-thermal image pair. Specifically, Gaussian wrapping functions with preset scale parameters are applied to the near-infrared image and the thermal image respectively to calculate their illumination component estimates. The reflection component image of the near-infrared image and the thermal reflection component image of the thermal image are obtained by subtracting the logarithmic domain of the illumination component from the logarithmic domain of the original image. The third acquisition subunit is used to perform multiplicative fusion and adaptive threshold segmentation processing based on the reflection component image and the thermal reflection component image. Specifically, the reflection component image of the near-infrared image and the thermal reflection component image are multiplied at the pixel level, and the fusion result image after pixel-level multiplication is binarized to obtain the enhanced candidate target image.

6. The mountain nocturnal animal tracking system based on multispectral and thermal imaging data fusion according to claim 4, characterized in that, The connection unit includes: The first connection subunit is used to perform trajectory segment initialization processing based on a multidimensional feature set. The initial trajectory segment set is obtained by creating a trajectory segment containing only one target point from the multidimensional feature set corresponding to the first frame of multi-source image data. The target point is a point that contains all features in the multidimensional feature set. The second connection subunit is used to perform inter-frame feature distance matrix calculation based on the multi-dimensional feature set corresponding to the candidate target images of two adjacent frames. The inter-frame feature distance matrix is ​​obtained by calculating the Euclidean distance between the multi-dimensional feature vector of each candidate target in the previous frame and the multi-dimensional feature vector of each candidate target in the next frame. The third connection subunit is used to perform matching processing based on the inter-frame feature distance matrix and the nearest neighbor greedy matching algorithm to obtain a set of matching pairs and a set of unmatched candidate targets between adjacent frames. The fourth connection subunit is used to perform trajectory segment update processing based on the set of matching pairs and the set of unmatched candidate targets between adjacent frames, so as to obtain the set of trajectory segments after cross-frame connection.

Images