Insect cluster three-dimensional trajectory reconstruction method based on adaptive redetection

Abstract

The invention discloses an insect cluster three-dimensional trajectory reconstruction method based on adaptive re-detection, which combines redundancy of multi-view data and a re-detection technology, and can adaptively adjust a detection threshold in a view in which a matching measurement value is not found when the matching measurement value is lacked in a tracking process, so that the accuracy of the three-dimensional trajectory reconstruction of the insect cluster is improved. Individuals possibly missed in the detection stage are searched. The problem of individual missing detection caused by noise or setting of a fixed detection threshold is effectively improved, so that the detection effect in a noisy image is improved, the problem of data missing in the tracking process and the influence of missing detection and false detection on trajectory reconstruction are reduced, and the integrity and consistency of data are improved.

Description

Technical Field

[0001] The invention belongs to the technical field of trajectory tracking and relates to a method for reconstructing three-dimensional trajectories of insect clusters based on adaptive re-detection. Background Art

[0002] There are various group movement phenomena in nature, including but not limited to the soaring of birds, the swimming of fish, the gathering of mosquitoes, and the busyness of bees. These biological clusters show amazingly highly coordinated movements, and they effectively respond to environmental changes and threats from predators through complex interaction patterns. This collective wisdom not only reflects the exquisite design of the biological world, but also brings profound inspiration to mankind. Therefore, accurately reconstructing the three-dimensional motion trajectory of these biological clusters has become a key step in exploring their internal coordination mechanisms and the laws of cluster behavior. This is not only of great significance to deepening our understanding of life sciences, but also can promote the progress and development of many fields such as biological intelligence, robot group control, and virtual reality.

[0003] Especially in the field of biological research, researchers are increasingly interested in insect swarms in the wild. Compared with laboratory environments, wild swarm research is particularly important because it can provide more realistic data. However, swarm research in the wild environment faces many challenges. Compared with birds, wild insect swarms have more complex backgrounds due to restrictions such as flight altitude and habitat location, which increases the difficulty of detection and tracking. The quality of trajectory reconstruction is closely related to the quality of video and algorithm, and most existing target detection methods perform poorly when dealing with complex backgrounds and high-noise environments. Fixed detection thresholds are prone to missed detections and false detections, which seriously affect subsequent target tracking and reconstruction effects. Previous studies have mainly focused on the separate optimization of target detection and tracking algorithms, and failed to fully consider the interaction between the two. As the first step in trajectory reconstruction, the accuracy of target detection directly affects the subsequent tracking and reconstruction results. Summary of the invention

[0004] In order to overcome the deficiencies of the prior art, the present invention provides a method for reconstructing three-dimensional trajectories of insect swarms based on adaptive re-detection.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A method for reconstructing three-dimensional trajectories of insect swarms based on adaptive re-detection, comprising the following steps:

[0007] Step 1: Take image sequences or videos from multiple camera views as input and extract the measurements of the objects of interest in the view;

[0008] Step 2: Extract the measurement value sets from multiple views as direct input, use the particle filter tracking algorithm for three-dimensional tracking, and obtain the three-dimensional trajectory data of the insect cluster;

[0009] Step 3: During the tracking process, a tracker is assigned to each tracking target for tracking; if the tracker obtains the corresponding matching measurement values ​​in all views, the 3D position of the target is reconstructed according to all matching measurement values; if matching measurement values ​​are missing in some views, a re-detection method is used to construct temporary 3D points based on the known matching measurement values ​​in the view where matching measurement values ​​have been obtained, and the temporary 3D points are projected to the view where matching measurement values ​​are missing. The detection threshold around the projection of the temporary 3D points is adjusted to search for matching measurement values ​​in the view where matching measurement values ​​are missing, and then the 3D position of the target is reconstructed according to the matching measurement values ​​in all views to ensure the continued execution of the tracker.

[0010] Furthermore, in step 3, each tracking target corresponds to a tracker to independently track and reconstruct the three-dimensional trajectory, completing the three-dimensional position reconstruction and velocity calculation of the target in the time series.

[0011] Furthermore, in step 3, the premise for reconstructing the target's 3D position is that the tracker must have matching measurement data in each view.

[0012] Furthermore, in step 3, the re-detection method is used on the premise that the tracker fails to find matching measurement values ​​in all views, but can obtain matching measurement values ​​in at least two views.

[0013] Furthermore, in step 3, when a matching measurement value is searched for using a re-detection method in a view where a matching measurement value is missing, the searched matching measurement value is verified using an epipolar constraint.

[0014] Furthermore, in step 2, the particle filter tracking algorithm includes: predicting the position of the target at the next time t based on the position of the target in the previous two frames, generating N sampling particles of a fixed size, representing the posterior probability distribution of the three-dimensional position of each target at the sampling time t, updating the weights based on the matching measurement values, and the tracker resampling high-weight particles to update the three-dimensional position of the target at time t.

[0015] Furthermore, in step 1, the method for extracting the measurement value is: taking multiple views as input, constructing a background model through a sliding window technique, obtaining a preliminary foreground image using a background subtraction method, extracting tree outlines and filling in to generate a tree background image, subtracting the preliminary foreground image from the tree background image to obtain an optimized foreground image, segmenting the optimized foreground image to identify connected areas, and obtaining the measurement value.

[0016] Furthermore, the measurement values ​​include centroid and target area coordinate data.

[0017] Furthermore, in step 1, the input includes at least image sequences or videos from three camera perspectives.

[0018] In summary, the present invention is beneficial in that:

[0019] The present invention processes image sequences of insect swarm flying under multiple viewing angles, extracts measurement information of targets of interest through a target detection algorithm, and obtains the three-dimensional trajectory of the biological swarm using a tracking algorithm combined with a re-detection method; through the redundancy of data from multiple viewing angles and re-detection technology, when there is a lack of matching measurements during the tracking process, the detection threshold can be adaptively adjusted in the view where no matching measurements are found, and individuals that may be missed in the detection stage can be searched, thereby improving the continuity and accuracy of target tracking and effectively improving the influence of missed detection and false detection on trajectory reconstruction during the tracking process. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a flowchart of the method for reconstructing three-dimensional trajectories of insect swarms of the present invention. DETAILED DESCRIPTION

[0021] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0022] The present invention provides a method for reconstructing three-dimensional trajectories of insect swarms based on adaptive re-detection, which consists of two major parts: target detection and target tracking. The first step of the method is to extract high-level observation data, measurements, from the original input through target detection. The second step of the method is to extract a measurement set from multiple viewpoints as the direct input of a particle filter tracking algorithm based on adaptive re-detection, use the particle filter algorithm to track in space, and predict and sample to obtain matching measurement values ​​of the target and multiple views. The system requires that the target must have associated matching measurement values ​​on all views before it can continue tracking. If the requirement is not met, as long as the target has matching measurement values ​​on two views, the system will enable the re-detection algorithm to find the unmatched measurement values ​​and construct a three-dimensional point with a smaller error.

[0023] The hardware architecture based on the insect swarm trajectory reconstruction algorithm of this embodiment mainly includes three high-speed storage workstations, three portable displays, three outdoor mobile power supplies, three high-speed cameras, a camera synchronization digital signal trigger and a T-shaped calibration rod; the images collected by the camera will be stored in the workstation as the raw data of the algorithm.

[0024] Before implementing the insect swarm 3D trajectory reconstruction method, camera calibration is first performed under multi-viewing angles. The specific calibration method includes: using a fixed-length horizontal bar fixed on a long pole to form a T-shaped calibration bar; waving the calibration bar within the common field of view of the camera to ensure that different postures and positions are captured by multiple cameras; using EasyWand software to digitize the image positions of the two end points on the horizontal bar in all views, and then using these corresponding points to estimate the camera's intrinsic and extrinsic parameters, thereby improving the accuracy and reliability of the camera calibration and ensuring the accuracy of subsequent 3D trajectory reconstruction.

[0025] The target detection method of the insect swarm three-dimensional trajectory reconstruction method includes obtaining an image sequence or video of a multi-organism swarm under multiple camera perspectives as input, and using a target detection algorithm to obtain measurement values ​​of the target of interest, the measurement values ​​including the center of mass and target area coordinate data.

[0026] Specifically, the target detection algorithm includes taking an image sequence or video as input, constructing a background model in the view through a sliding window technique, and obtaining a preliminary foreground image using a background subtraction method. Next, the Canny edge detection algorithm is used to extract the outline of the tree, and the outline is filled to generate a binary tree background image. Subsequently, the preliminary foreground image is subtracted from the tree background image to eliminate the interference of the tree area on the detection, and the processed optimized foreground image is obtained. On this basis, the optimized foreground image is segmented, multiple connected regions are identified, and the center of mass of each region is calculated to determine the center position and area of ​​the target of interest, and finally the measurement value of the target in the view is obtained.

[0027] The target tracking method includes extracting a set of measurement values ​​from multiple views as direct input to the target tracking algorithm, using a particle filter algorithm to perform three-dimensional tracking in space, and finally obtaining three-dimensional trajectory data of the insect cluster.

[0028] Specifically, the particle filter algorithm includes extracting a set of measurement values ​​from multiple views as observation input and generating a dedicated particle filter tracker for each target. Based on the position of the target in the previous two frames, the position at the next moment t is predicted, and N fixed-size sampling particles are generated to represent the posterior probability distribution of the target's three-dimensional position at time t. Each sampling particle represents a possible state of the target, and the weight value is calculated based on the similarity and dynamic relationship between the measurement value associated at time t-1 and the measurement associated with the sampling particle at time t. After state prediction and observation-based weight update, the three-dimensional position of the target at time t is effectively estimated.

[0029] The particle filter tracker performs prediction and tracking in three-dimensional space, combining time and stereo correlation. It finds measurement values ​​that match the target in multiple views based on the appearance similarity in the observed data and the dynamic consistency of the motion model, and uses these measurement values ​​to update the three-dimensional position of the target at that moment, thereby achieving accurate tracking of the target.

[0030] To ensure tracking stability, the tracker must obtain the target's measurement values ​​in each view so that it can continuously track and accurately reconstruct the target's three-dimensional position.

[0031] When the tracker fails to find matching measurements in all views, but can find matching measurements in at least two views, a re-detection method is used. The re-detection method consists in adaptively adjusting the detection threshold in the view where no matching measurements are found for the target, in order to search for measurements that may have been missed during the target detection phase to continue the subsequent tracking.

[0032] When the particle filter tracker can find matching measurements in at least two views, it selects the two matching measurements that have been found and uses triangulation to construct a temporary 3D point. Then, using the camera's intrinsic and extrinsic matrix, the 3D point is reprojected to the view where no matching measurement is found, and the target is detected again in the surrounding area of ​​the reprojected point. If no matching measurement is found in the initial detection, the system will adaptively lower the threshold of target detection, relax the detection conditions, and search for possible matching measurements in the view. During the re-detection process, if a possible matching measurement is found, it is verified using the epipolar constraint. If it meets the requirements, the matching measurements in all views are used to construct and update the 3D position and velocity of the tracked target at that moment; if no matching measurement is found or the epipolar constraint is not met, the tracking of the particle filter tracker is terminated.

[0033] Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

Claims

1. A method for reconstructing three-dimensional trajectories of insect swarms based on adaptive re-detection, characterized in that: The following steps are involved: Step 1: Take image sequences or videos from multiple camera views as input and extract the measurements of the objects of interest in the view; Step 2: Extract the measurement value sets from multiple views as direct input, use the particle filter tracking algorithm for three-dimensional tracking, and obtain the three-dimensional trajectory data of the insect cluster; Step 3: During the tracking process, a tracker is assigned to each tracking target for tracking; if the tracker obtains the corresponding matching measurement values ​​in all views, the 3D position of the target is reconstructed according to all matching measurement values; if matching measurement values ​​are missing in some views, a re-detection method is used to construct temporary 3D points based on the known matching measurement values ​​in the view where matching measurement values ​​have been obtained, and the temporary 3D points are projected to the view where matching measurement values ​​are missing. The detection threshold around the projection of the temporary 3D points is adjusted to search for matching measurement values ​​in the view where matching measurement values ​​are missing, and then the 3D position of the target is reconstructed according to the matching measurement values ​​in all views to ensure the continued execution of the tracker.

2. The method for reconstructing three-dimensional trajectories of insect swarms based on adaptive re-detection according to claim 1, characterized in that: In step 3, each tracking target corresponds to a tracker to independently track and reconstruct the three-dimensional trajectory, completing the three-dimensional position reconstruction and velocity calculation of the target in the time series.

3. The method for reconstructing three-dimensional trajectories of insect swarms based on adaptive re-detection according to claim 1, characterized in that: In step 3, the 3D position of the target is reconstructed on the premise that the tracker must have matching measurement data in each view.

4. The method for reconstructing three-dimensional trajectories of insect swarms based on adaptive re-detection according to claim 1, characterized in that: In step 3, the re-detection method is used on the premise that the tracker fails to find matching measurements in all views, but can obtain matching measurements in at least two views.

5. The method for reconstructing three-dimensional trajectories of insect swarms based on adaptive re-detection according to claim 1, characterized in that: In step 3, when matching measurements are searched for using the re-detection method in the view where matching measurements are missing, the matched measurements are verified using the epipolar constraints.

6. The method for reconstructing three-dimensional trajectories of insect swarms based on adaptive re-detection according to claim 1, characterized in that: In step 2, the particle filter tracking algorithm includes: predicting the position of the target at the next time t based on the position of the previous two frames, generating N fixed-size sampling particles, representing the posterior probability distribution of the three-dimensional position of each target at the sampling time t, updating the weights based on the matching measurement values, and the tracker resampling high-weight particles to update the three-dimensional position of the target at time t.

7. The method for reconstructing three-dimensional trajectories of insect swarms based on adaptive re-detection according to claim 1, characterized in that: In step 1, the method for extracting the measurement value is: taking multiple views as input, constructing a background model through sliding window technology, obtaining a preliminary foreground image using background subtraction method, extracting tree outlines and filling to generate a tree background image, subtracting the preliminary foreground image from the tree background image to obtain an optimized foreground image, segmenting the optimized foreground image to identify connected areas, and obtaining the measurement value.

8. The method for reconstructing three-dimensional trajectories of insect swarms based on adaptive re-detection according to claim 7, characterized in that: The measurements include centroid and target area coordinate data.

9. The method for reconstructing three-dimensional trajectories of insect swarms based on adaptive re-detection according to claim 1, characterized in that: In step 1, the input includes at least three image sequences or videos from camera perspectives.

Images