An insect swarm three-dimensional trajectory reconstruction method based on adaptive re-detection

Abstract

The application discloses an insect cluster three-dimensional trajectory reconstruction method based on adaptive re-detection, which combines the redundancy of multiple perspective data and the re-detection technology, and can adaptively adjust the detection threshold in the view where the matching measurement value is not found, search for the individual that may be missed in the detection stage, when the matching measurement value is lacked in the tracking process. This effectively improves the individual missing problem caused by noise or setting a fixed detection threshold, thereby improving the detection effect in the noisy image, reducing the data missing problem in the tracking process and the influence of missing and false detection on trajectory reconstruction, and improving the integrity and consistency of the data.

Description

Technical Field

[0001] This invention belongs to the field of trajectory tracking technology and relates to a method for reconstructing the three-dimensional trajectory of insect swarms based on adaptive re-detection. Background Technology

[0002] Nature exhibits a wide variety of group movement phenomena, including but not limited to flocks of birds soaring, schools of fish swimming, swarms of mosquitoes gathering, and busy colonies of bees. These biological groups demonstrate astonishingly high levels of coordinated movement, effectively responding to environmental changes and predator threats through complex interaction patterns. This collective intelligence not only reflects the ingenious design of the biological world but also offers profound insights for humankind. Therefore, accurately reconstructing the three-dimensional movement trajectories of these biological groups is a crucial step in exploring their intrinsic coordination mechanisms and the patterns of group behavior. This is not only significant for deepening our understanding of life sciences but also promotes progress and development in multiple fields such as biological intelligence, robot swarm control, and virtual reality.

[0003] Especially in the field of biological research, researchers are increasingly interested in the study of insect colonies in the wild. Compared to laboratory environments, field colony studies are particularly important because they provide more realistic data. However, field colony studies face many challenges. Compared to birds, insect colonies in the wild are subject to limitations such as flight altitude and habitat location, resulting in more complex backgrounds and increasing the difficulty of detection and tracking. The quality of trajectory reconstruction is closely related to the quality of both the video and the algorithm. Most existing target detection methods perform poorly in complex backgrounds and high-noise environments, and fixed detection thresholds can easily lead to missed detections and false detections, thus seriously affecting the subsequent target tracking and reconstruction results. Previous studies have mainly focused on optimizing target detection and tracking algorithms separately, failing to fully consider the interaction between the two. As the first step in trajectory reconstruction, the accuracy of target detection directly affects the subsequent tracking and reconstruction results. Summary of the Invention

[0004] In order to overcome the shortcomings of the prior art, this invention provides a method for reconstructing the three-dimensional trajectory of insect clusters based on adaptive re-detection.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] A method for reconstructing the 3D trajectory of an insect swarm based on adaptive re-detection includes the following steps:

[0007] Step 1: Acquire image sequences or videos from multiple camera perspectives as input, and extract the measurement values ​​of the target of interest in the view;

[0008] Step 2: Extract the set of measurements from multiple views as direct input, and use the particle filter tracking algorithm for 3D tracking to obtain the 3D trajectory data of the insect swarm;

[0009] Step 3: During the tracking process, a tracker is assigned to each target. If the tracker obtains the corresponding matching measurement value in all views, the 3D position of the target is reconstructed based on all matching measurement values. If some views are missing matching measurement values, a re-detection method is used. Temporary 3D points are constructed based on the known matching measurement values ​​in the views where matching measurement values ​​have been obtained. The temporary 3D points are projected onto the views where matching measurement values ​​are missing. The detection threshold around the projection of the temporary 3D points is adjusted to search for matching measurement values ​​in the views where matching measurement values ​​are missing. Then, the 3D position of the target is reconstructed based on the matching measurement values ​​in all views to ensure the continued execution of the tracker.

[0010] Furthermore, in step 3, each tracked target corresponds to a tracker for independent tracking and reconstruction of the three-dimensional trajectory, completing the three-dimensional position reconstruction and velocity calculation of the target over time.

[0011] Furthermore, in step 3, the prerequisite for the target's three-dimensional position reconstruction is that the tracker must have matching measurement data in each view.

[0012] Furthermore, in step 3, the premise for using the re-detection method is that the tracker fails to find matching measurements in all views, but can obtain matching measurements in at least two views.

[0013] Furthermore, in step 3, when a matching measurement is found using the re-detection method in a view with missing matching measurements, the found matching measurement is verified using epipolar constraints.

[0014] Furthermore, in step 2, the particle filter tracking algorithm includes: predicting the target's position at time t based on the target's position in the previous two frames, generating N sampled particles of fixed size to represent the posterior probability distribution of the three-dimensional position of each target at time t, updating the weights based on the matching measurement values, resampling the high-weight particles by the tracker, and updating the target's three-dimensional position at time t.

[0015] Furthermore, in step 1, the method for extracting the measured values ​​is as follows: using multiple views as input, a background model is constructed using sliding window technology, a preliminary foreground image is obtained using background subtraction, tree outlines are extracted and filled to generate a tree background image, the preliminary foreground image is subtracted from the tree background image to obtain an optimized foreground image, the optimized foreground image is segmented to identify connected regions, and the measured values ​​are obtained.

[0016] Furthermore, the measurements include the centroid and target area coordinate data.

[0017] Furthermore, in step 1, the input includes at least a sequence of images or videos from three camera perspectives.

[0018] In summary, the advantages of this invention are:

[0019] This invention processes image sequences of insect swarms flying from multiple perspectives, extracts measurement information of targets of interest through a target detection algorithm, and obtains the three-dimensional trajectory of the biological swarm using a tracking algorithm combined with a re-detection method. By leveraging the redundancy of data from multiple perspectives and the re-detection technique, when matching measurements are lacking during the tracking process, the detection threshold can be adaptively adjusted in views where no matching measurements are found, and individuals that may have been missed during the detection phase can be searched for, improving the continuity and accuracy of target tracking and effectively mitigating the impact of missed and false detections on trajectory reconstruction during the tracking process. Attached Figure Description

[0020] Figure 1 This is a flowchart of the insect cluster three-dimensional trajectory reconstruction method of the present invention. Detailed Implementation

[0021] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0022] This invention provides a method for reconstructing the 3D trajectory of insect swarms based on adaptive re-detection, consisting of two main parts: target detection and target tracking. The first step is to extract high-level observation data, or measurements, from the raw input through target detection. The second step uses the extracted measurement set from multiple viewpoints as direct input to a particle filter tracking algorithm based on adaptive re-detection. The particle filter algorithm is used for spatial tracking, predicting and sampling to obtain matching measurements between the target and multiple views. The system requires that the target have associated matching measurements on all views for tracking to continue. If this requirement is not met, as long as the target has matching measurements on two views, the system will activate the re-detection algorithm to find the unmatched measurements and construct a 3D point with smaller errors.

[0023] The hardware architecture upon which the insect swarm trajectory reconstruction algorithm in this embodiment is based mainly includes three high-speed storage workstations, three portable displays, three outdoor power banks, three high-speed cameras, a camera synchronization digital signal trigger, and a T-shaped calibration rod; the images acquired by the cameras are stored in the workstations as the raw data for the algorithm.

[0024] Before implementing the method for reconstructing the three-dimensional trajectory of insect clusters, camera calibration is first performed from multiple perspectives. The specific calibration method includes: using a fixed-length horizontal bar fixed to a long rod to form a T-shaped calibration bar; waving the calibration bar within the common field of view of the cameras to ensure that different postures and positions are captured by multiple cameras; using EasyWand software to digitize the image positions of the two endpoints on the horizontal bar in all views, and then using these corresponding points to estimate the intrinsic and extrinsic parameters of the cameras, thereby improving the accuracy and reliability of camera calibration and ensuring the accuracy of subsequent three-dimensional trajectory reconstruction.

[0025] The target detection method for the 3D trajectory reconstruction of insect clusters includes acquiring image sequences or videos of multi-organism clusters from multiple camera perspectives as input, and using a target detection algorithm to obtain the measurement values ​​of the target of interest, including the centroid and target region coordinate data.

[0026] Specifically, the object detection algorithm includes: taking an image sequence or video as input, constructing a background model in the view using a sliding window technique, and obtaining a preliminary foreground image using background subtraction. Next, the Canny edge detection algorithm is used to extract the tree contours, and the contours are filled to generate a binarized tree background image. Then, the preliminary foreground image is subtracted from the tree background image to eliminate interference from the tree region, resulting in a processed optimized foreground image. Based on this, the optimized foreground image is segmented to identify multiple connected regions, and the centroid of each region is calculated to determine the center position and region of the target of interest, ultimately obtaining the measurement value of the target in the view.

[0027] The target tracking method includes extracting a set of measurements from multiple views as direct input to the target tracking algorithm, using a particle filtering algorithm for three-dimensional spatial tracking, and finally obtaining three-dimensional trajectory data of the insect swarm.

[0028] Specifically, the particle filter algorithm involves extracting a set of measurements from multiple views as observation input and generating a dedicated particle filter tracker for each target. Based on the target's position in the previous two frames, the algorithm predicts the position at the next time step t and generates N fixed-size sampled particles, representing the posterior probability distribution of the target's 3D position at time t. Each sampled particle represents a possible state of the target, and a weight value is calculated based on the similarity and dynamic relationship between the measurements associated with the target at time t-1 and the measurements associated with the sampled particles at time t. After state prediction and observation-based weight updates, the 3D position of the target at time t is effectively estimated.

[0029] Particle filter trackers perform prediction and tracking in three-dimensional space, combining time and stereo correlation. Based on the appearance similarity and dynamic consistency of the motion model in the observation data, they find the measurement values ​​that match the target in multiple views and use these measurement values ​​to update the target's three-dimensional position at that moment, thereby achieving accurate tracking of the target.

[0030] To ensure tracking stability, the tracker must acquire the target's measurements in each view in order to continuously track and accurately reconstruct the target's three-dimensional position.

[0031] When the tracker fails to find a matching measurement in all views, but finds a matching measurement in at least two views, a re-detection method is used. The re-detection method involves adaptively adjusting the detection threshold in the views where no matching measurement was found for the target, to search for measurements that might have been missed during the target detection phase to continue subsequent tracking.

[0032] When the particle filter tracker finds matching measurements in at least two views, it selects the two found matching measurements and constructs a temporary 3D point using triangulation. Then, using the camera's intrinsic and extrinsic parameters, this 3D point is reprojected onto the view where no matching measurements were found, and target detection is performed again in the area surrounding the reprojected point. If no matching measurements are found in the initial detection, the system adaptively lowers the target detection threshold, relaxes the detection conditions, and searches for possible matching measurements in that view. During the re-detection process, if a possible matching measurement is found, it is verified using epipolar constraints. If satisfied, the 3D position and velocity of the tracked target at that moment are constructed and updated using matching measurements from all views; if no matching measurement is found or the epipolar constraints are not satisfied, the particle filter tracker terminates its tracking.

[0033] Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort should fall within the scope of protection of the present invention.

Claims

1. A method for reconstructing the three-dimensional trajectory of insect swarms based on adaptive re-detection, characterized in that, Includes the following steps: Step 1: Input an image sequence or video from at least three camera views and extract the measurements of the target of interest in the view; Step 2: Extract the set of measurements from multiple views as direct input, and use the particle filter tracking algorithm for 3D tracking to obtain the 3D trajectory data of the insect swarm; Step 3: During the tracking process, a dedicated particle filter tracker is generated for each target. If the tracker obtains corresponding matching measurements in all views, the 3D position of the target is reconstructed based on all matching measurements. If some views lack matching measurements, a re-detection method is used. Temporary 3D points are constructed based on the known matching measurements in the views where matching measurements have been obtained. These temporary 3D points are projected onto the views where matching measurements are missing. The detection threshold around the projection of the temporary 3D points is adjusted to search for matching measurements in the views where matching measurements are missing. The 3D position of the target is then reconstructed based on the matching measurements in all views to ensure the continued execution of the tracker. The prerequisite for using the re-detection method is that the tracker fails to find matching measurements in all views, but can obtain matching measurements in at least two views. When a matching measurement is found using the re-detection method in a view with missing matching measurements, the found matching measurement is validated using polar constraints.

2. The method for reconstructing the three-dimensional trajectory of an insect swarm based on adaptive re-detection according to claim 1, characterized in that, In step 3, each tracked target is independently tracked and its three-dimensional trajectory is reconstructed by a tracker, completing the three-dimensional position reconstruction and velocity calculation of the target over time.

3. The method for reconstructing the three-dimensional trajectory of an insect swarm based on adaptive re-detection according to claim 1, characterized in that, In step 3, the prerequisite for reconstructing the target's 3D position is that the tracker must have matching measurement data in each view.

4. The method for reconstructing the three-dimensional trajectory of an insect swarm based on adaptive re-detection according to claim 1, characterized in that, In step 2, the particle filter tracking algorithm includes: predicting the target's position at time t based on the target's position in the previous two frames, generating N sampled particles of fixed size to represent the posterior probability distribution of the three-dimensional position of each target at time t, updating the weights based on the matching measurement values, resampling the high-weight particles by the tracker, and updating the target's three-dimensional position at time t.

5. The method for reconstructing the three-dimensional trajectory of an insect swarm based on adaptive re-detection according to claim 1, characterized in that, In step 1, the method for extracting the measured values ​​is as follows: using multiple views as input, a background model is constructed using sliding window technology, a preliminary foreground image is obtained using background subtraction, tree outlines are extracted and filled to generate a tree background image, the preliminary foreground image is subtracted from the tree background image to obtain an optimized foreground image, the optimized foreground image is segmented to identify connected regions, and the measured values ​​are obtained.

6. The method for reconstructing the three-dimensional trajectory of an insect swarm based on adaptive re-detection according to claim 5, characterized in that, The measured values ​​include the coordinates of the centroid and the target area.

7. The method for reconstructing the three-dimensional trajectory of an insect swarm based on adaptive re-detection according to claim 1, characterized in that, In step 1, the input includes at least three camera viewpoints of image sequence or video.

Images