A multi-objective three-dimensional zebrafish motion trajectory tracking method

Abstract

The present invention provides a multi-object zebrafish three-dimensional motion trajectory tracking method. This method respectively shoots the social interaction behaviors of multiple zebrafish in the fish tank from the top view and the side view. For the top view image, the instance mask is detected by using the deep learning method and refined to obtain the preliminary skeleton result. The skeleton result is screened and processed, and the tracked skeleton head endpoints are extracted. The heuristic data association algorithm is used to achieve inter-frame matching, the SVM and HOG features are used to achieve matching between long trajectory segments, and the association algorithm based on motion constraints is used to connect long and short trajectory segments to complete the two-dimensional tracking of the top view. For the side view image, the deep learning detection method is used to determine the feasible region, and then the optimal matching point that matches the target in the top view is found through the objective optimization function and the particle optimization method, and triangulation is performed to complete the three-dimensional tracking task. Compared with the existing method, the data integrity of the trajectory tracking method of the present invention can be increased by 5% - 15%, and the tracking mis-matching rate can be reduced by 50%.

Description

Technical Field

[0001] The present invention relates to the field of animal group trajectory tracking, and particularly to a method for tracking the three-dimensional motion trajectories of multiple target zebrafish based on computer vision. Background Art

[0002] Automated animal trajectory tracking methods play an important role in the analysis of experimental behaviors in life sciences. Common model animals include mice, fruit flies, zebrafish, etc. Among them, zebrafish are favored because of their short growth cycle, easy laboratory breeding and observation.

[0003] At present, scientists have invented a variety of automated tracking methods for zebrafish motion trajectories, mainly including invasive tracking methods represented by color marking and RFID implantation, and non-invasive tracking methods based on vision methods. Among them, non-invasive tracking methods are widely used because of their easy operation and little interference with experimental animals. However, this vision-based non-invasive zebrafish tracking method generally can only track the three-dimensional motion trajectories of a single target or the two-dimensional motion trajectories of multiple targets. Therefore, there is still no reliable method to achieve the automated tracking of the three-dimensional multi-target zebrafish at the skeleton level. This is mainly because zebrafish are a type of model animal with a fast moving speed and large angular velocity changes, and it is difficult for the motion model to accurately predict their positions after a period of time. In addition, there are often cross-occlusions of multiple targets in the video acquisition field of view, and the shape of zebrafish in the view becomes larger, making it difficult to detect feature points. These difficulties lead to the deficiencies of low data integrity, high frequency of occurrence of tracking result mis-matching events in the three-dimensional multi-target zebrafish trajectory tracking, and it cannot be applied to the life science experimental scenario.

[0004] Based on the above problems, it is necessary to design a method for tracking the three-dimensional motion trajectories of multiple target zebrafish with high data integrity and stable tracking results. Summary of the Invention

[0005] Based on the above-mentioned defects and deficiencies existing when the existing non-invasive model animal tracking method based on vision is applied to the tracking of the three-dimensional motion trajectories of multiple target zebrafish, the present invention provides a method for tracking the three-dimensional motion trajectories of multiple target zebrafish that can overcome the interference of the cross-occlusion situation of multiple targets in the video acquisition field of view, with high data integrity and stable tracking results.

[0006] The technical solution adopted by the present invention to solve the above technical problems is: A method for tracking the three-dimensional motion trajectories of multiple target zebrafish, including the following steps:

[0007] S1, synchronously collect videos of the social interaction behaviors of zebrafish from the top view angle and the side view angle;

[0008] S2, using a deep learning-based image segmentation method to perform instance segmentation on several video images taken from a bird's-eye view, obtaining a binary mask image for each zebrafish instance. The mask is then refined using an image refinement method to obtain a preliminary zebrafish skeleton.

[0009] S3, screening and processing the preliminary skeleton results obtained in step S2, eliminating skeleton results with segmentation errors in cross-scenes, and saving the remaining corrected and corrected skeleton results as positive samples, and then performing head detection on all positive samples to determine the head endpoint of the zebrafish in the skeleton results;

[0010] S4, using a heuristic data association algorithm based on minimum Euclidean distance to perform inter-frame matching on all temporal continuous positive samples obtained in step S3, to obtain the initial non-intersecting trajectory splicing segments of each target zebrafish, and defining the segments with a time length equal to 1 frame as short trajectory segments, and the segments with a length greater than 1 frame as long trajectory segments;

[0011] S5: An improved HOG feature map is extracted from the rigid body region of the upper half of the zebrafish's back. HOG features of multiple targets within the longest non-overlapping time period are selected and used to train an SVM. The trained model is used to infer the probability that all long trajectory segments belong to different zebrafish individuals. The Hungarian matching algorithm is then used to connect the multi-target long trajectory segments based on the trajectory segment matching probability and temporal relationship, completing two-dimensional tracking.

[0012] S6, uses a motion constraint-based data association algorithm to connect short trajectory segments with long trajectory segments to complete two-dimensional multi-target tracking;

[0013] S7, detecting the body region of each zebrafish instance in the side-view image acquired in step S1 by using the FasterRCNN method, and finding the optimal matching point that matches the top-view target in the body region of each zebrafish instance by using the target optimization function and the particle optimization method;

[0014] S8, triangulates the optimal matching point found with the head and skeleton results in the overhead image of the corresponding zebrafish instance, calculates the three-dimensional skeleton coordinates of the corresponding zebrafish, and inherits the two-dimensional identity information to three dimensions to achieve three-dimensional multi-target zebrafish skeleton-level tracking.

[0015] Furthermore, in step S3, the preliminary skeleton results obtained in step S2 are screened and processed using a skeleton analysis and processing strategy. The skeleton analysis and processing strategy specifically includes the following five constraints:

[0016] (a) In a single zebrafish segmentation example, if the number of skeleton branches is greater than 3, the skeleton result is considered a negative sample and removed;

[0017] (b) For the skeleton results with 3 skeleton branches in a single zebrafish segmentation instance, correct the skeletons, that is, remove the bone spurs and save them as positive samples; where: the bone spurs are the shortest skeleton branches;

[0018] (c) Two instances with skeleton intersections are negative samples and are removed;

[0019] (d) If the masks of two instances are in contact, and one of the instances is determined to be a negative sample by the above conditions, then the other instance is also determined to be a negative sample and is removed;

[0020] (e) Other instance samples are all positive samples and are retained.

[0021] Further, in the skeleton analysis and processing strategy, the number and length of the skeleton branches are determined as follows: Filter the binary images of the skeleton results obtained in step S2 with the set filter kernels respectively, so as to screen out the endpoints and intersection points of the skeletons; Then, use the Freeman chain code form to encode starting from each endpoint, along the direction of the skeleton, and when an intersection point is encountered, it is considered that the skeleton branch ends, so as to determine the number of skeleton branches and the length of each skeleton branch in a single zebrafish segmentation instance.

[0022] Further, in step S3, the specific method for head detection is: Perform filtering operations on the skeleton results saved as positive samples after screening and processing respectively to find the two endpoints of the skeleton, select the window matrices centered on these two endpoints respectively to take values on the mask, calculate the eigenvalues of the covariance matrix of the window, and the one with the larger eigenvalue is defined as the head endpoint of the zebrafish skeleton.

[0023] Further, in step S4, the specific content of frame - to - frame matching for all temporally continuous positive samples according to the heuristic data association algorithm is:

[0024] O i,t-1 and O j,t respectively represent any zebrafish individual in the (t - 1) - th frame and the t - th frame, D(O i,t-1 ,O j,t ) represents the Euclidean distance between O i,t-1 and O j,t , and the targets are matched with the minimum Euclidean distance, and the matching method is as follows:

[0025]

[0026] where: m(O i,t-1 ,O j,t ) = 1 means that the i - th target and the j - th target are connected and matched; If in the (t - 1) - th frame, there are two target individuals O i,t-1 and O k,t-1May match with the target O in the t-frame j,t If a match is made, the trajectory segment is disconnected at this time. The matching method is as follows:

[0027]

[0028] When the fish cross, the number of instances detected in the frames before and after the crossing moment may change. At this time, there may be target individuals O i,t-1 and O k,t-1 in the (t - 1)-frame j,t matching with the target O in the t-frame

[0029]

[0030] In the above formula, the difference between D(O i,t-1 , O j,t ) and D(O k,t-1 , O j,t ) is greater than Thr d , and the target O j,t is matched with the target O i,t-1 ; Thr d is the minimum distance threshold.

[0031] Furthermore, in step S5, the improved HOG feature map is extracted in the following way:

[0032] S11, Scale the zebrafish image sample obtained by segmenting in step S2 to a specified scale size and rotate it to the same direction; then mark the foreground pixel points as 1 and the background pixel points as 0 to obtain a binary image;

[0033] S12, Convert the original image input in step S11 into a HOG feature map through the HOG algorithm;

[0034] S13, Set a 9*9 pixel area as a Cell, coarsen the binary image into N Cells. If the number of pixel points with pixel value 1 in a Cell is greater than a given threshold, it is considered that the Cell is highly information-related, otherwise the Cell is low-information-related;

[0035] S14, Set a 4*4 Cell area as a Block. When all Cells in a Block are low-information-related, define the Block as a useless block, otherwise it is a useful block;

[0036] S15, Match the pixel areas of the above-marked useful blocks with the HOG feature map one by one, and spirally output a certain number of Blocks marked as useful blocks to obtain the final improved HOG feature map.

[0037] Further, the data association algorithm based on motion constraints in step S6 is specifically as follows:

[0038] Select a short track segment m, define its occurrence time as T, and select, from the track segments of each target, the long track segment n that occurred closest to T before time T as the target to be matched; then the matching weight between the track segment m and any long track segment n to be matched is as follows:

[0039]

[0040] In the formula: Vt mn represents the time difference between the start time of the track segment m and the end time of the long track segment n, in frames; represents the distance factor; represents the angle factor; The calculation formula of is as follows:

[0041]

[0042]

[0043] In the formula: d mj represents the distance between the start time coordinate of the short track segment m and the j-th end time coordinate in the long track segment, and a mj represents the absolute value of the difference between the target angle at the start time of the short track segment m and the target angle at the j-th end time in the long track segment; according to the calculated weight J, select the long track segment with the largest weight to connect with the short track segment.

[0044] Further, in step S7, the target optimization function is specifically:

[0045] g(x) = δRepro(x t , x) + Dis(X last , X tri ), δ ∈ [0, 1]

[0046] In the above formula: Repro(x t , x) represents the reprojection error, using pixel distance; x represents the particle point coordinates in the side view; x t represents the head matching point coordinates in the top view; Dis(X last , X tri ) represents the distance constraint error, using Euclidean distance; X last represents the three-dimensional coordinate point of this target at the previous moment, and X tri represents the three-dimensional coordinate point after the current particle triangulation; δ represents the balance weight between the reprojection error and the distance constraint error, with a value of 0.4;

[0047] The specific particle optimization method is:

[0048] Select a target to be matched frame by frame in the top view, and then design n particle points within the fish body area in the side view of the frame to calculate the objective optimization function values of the particles respectively, and move the non-optimal particles towards the optimal particle direction. The moving distance and the moving angle γ follow the following two distributions:

[0049] φ ∼ U((1 - μ)l, (1 + μ)l);

[0050] γ ∼ U(-σ, σ);

[0051] In the above formulas: l represents the pixel distance between the particle and the optimal particle point; σ represents the direction random variable, with a value of 15°; μ represents the distance random weight, with a value of 0.4;

[0052] Optimization stop target: When reaching five iteration times or the difference in pixel distances of the optimal particles between two adjacent optimizations is less than the set threshold, the optimal matching point that matches the top view target is obtained.

[0053] The present invention has the following advantages and effects compared with the prior art:

[0054] 1. The method described in the present invention synchronously collects the social interaction behaviors of multiple zebrafish from the top view and side view perspectives respectively; in the obtained top view image, uses the deep learning method to detect the single zebrafish instance mask and perform refinement processing to obtain the preliminary skeleton result, and reasonably applies the skeleton analysis and processing strategy to screen and process several skeleton results, retains the correct or correctable skeleton results, and eliminates the completely wrong skeleton results in the cross-scene; secondly, uses the heuristic data association algorithm to achieve inter-frame matching, uses SVM and HOG features to achieve matching between long trajectory segments, and uses the motion constraint association method to achieve the connection of long and short trajectory segments, so as to complete the two-dimensional tracking of the top view; then uses the deep learning-based detection method for the side view image to determine the feasible region, and then designs the optimization function and particle optimization method to find the best matching point that matches the top view target and perform triangulation, so as to complete the three-dimensional trajectory tracking task of multi-target zebrafish; this method can overcome the deficiencies of the existing multi-mode animal trajectory tracking methods, such as low data integrity, high frequency of tracking result mis-matching events, and inability to be applied to life science experimental scenarios; further, the experimental results show that the top view data integrity of the trajectory tracking method described in the present invention can be improved by 5% - 15% compared with the existing method, and the tracking mis-matching rate can be reduced by 50%.

[0055] 2. In the multi-object zebrafish three-dimensional motion trajectory tracking method of the present invention, since the image segmentation method of deep learning performs poorly in the scenario of multiple zebrafish crossing, and the accuracy of the segmentation results varies greatly; the present invention screens and processes a number of skeleton results through a skeleton analysis and processing strategy including five constraint conditions, eliminates the skeleton results with poor segmentation performance in the crossing scenario, and avoids the deficiency that the instance segmentation results of individual zebrafish are incorrect due to cross occlusion when performing instance segmentation on a number of video images captured by the overhead camera, thereby creating a premise for improving the data integrity in the detection link and ensuring the stability of the tracking results.

[0056] 3. In the multi-object zebrafish three-dimensional motion trajectory tracking method of the present invention, by scaling the image segmentation samples in step S2 to a fixed size and rotating them to the same direction, then calculating the HOG feature map, defining valid blocks and invalid blocks according to the information holding relevance, and spirally outputting a certain number of feature regions marked as valid blocks from the center of the image sample, an improved HOG feature map is obtained. By adopting this step, on the one hand, the HOG features have rotational invariance, and on the other hand, only the feature vectors related to the dorsal texture of the zebrafish individuals are retained, the background factors are eliminated, the feature dimension is reduced, and the generalization ability of the output features is enhanced, which can effectively improve the reliability, correctness and integrity of the connection of long trajectory segments in the overhead image, and thus can effectively ensure the high integrity of the multi-object zebrafish trajectory tracking data. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0058] Figure 1 It is a schematic flow chart of the multi-object zebrafish three-dimensional motion trajectory tracking method of the present invention.

[0059] Figure 2 It is a schematic diagram of the instrument device used in the multi-object zebrafish three-dimensional motion trajectory tracking method of the present invention.

[0060] Figure 3 It is a schematic diagram of the preliminary skeleton results of zebrafish obtained after segmenting and refining the video images captured from the overhead angle in step S2 of the method of the present invention.

[0061] Figure 4 It is a schematic diagram of the filter kernel described in step S3 of the present invention.

[0062] Figure 5Schematic diagram of the coding start and end of Freeman chain code in step S3 of the present invention.

[0063] Figure 6 Schematic diagram of the positive sample skeleton result after removing bone spurs in step S3 of the present invention.

[0064] Figure 7 Schematic diagram of head detection of the skeleton result saved as a positive sample in step S3 of the present invention.

[0065] Figure 8 Schematic diagram of the result of head coordinate detection of the linear skeleton result in step S3 of the present invention.

[0066] Figure 9 Schematic diagram of the HOG feature extraction and improvement process in step S5 of the present invention.

[0067] Figure 10 Schematic diagram of the side view image detection result in step S7 of the method of the present invention. In the figure, the square represents the result of the fish body area detected by the deep learning detection method.

[0068] Figure 11 In the particle optimization method in step S7 of the method of the present invention, schematic diagram of the moving distance and moving direction of particle points: In the figure, the star point represents the optimal point in the current iteration process, the dot A represents a certain particle point, and the dot B represents the position where the dot A will move to after optimization.

[0069] Figure 12 It is the time-sequence trajectory diagram after three-dimensional tracking of the method of the present invention with three zebrafish as an example.

[0070] Label description: 1 - device frame; 2 - fish tank; 3 - top view camera; 4 - side view camera; 5 - LED light source. Specific implementation mode

[0071] The following further elaborates on the present invention in conjunction with embodiments. The following embodiments are explanations of the present invention and the present invention is not limited to the following embodiments.

[0072] Embodiment 1: This embodiment provides a multi-target zebrafish three-dimensional motion trajectory tracking method, and its flow schematic diagram is as Figure 1 shown, and the specific steps are as follows:

[0073] S1, synchronously collect videos of the social interaction behaviors of zebrafish from the top view angle and the side view angle;

[0074] Among them: The position setting method between the acquisition camera and the fish tank 2 is as Figure 2As shown; the top-view camera 3 is suspended directly above the fish tank 2, and the lens of the side-view camera 4 is aimed at the center position of the side of the fish tank. The fish tank 2 is a square tank. In this embodiment, two to three adult zebrafish are selected and placed in the fish tank;

[0075] Specifically, the acquisition cameras used in this embodiment are industrial cameras (LUSTER GS3-U3-41C6C-C). Before acquisition, the internal and external parameter matrices of the two cameras need to be calibrated. During acquisition, the LED light source 5 is turned on, and shooting is synchronized from the top view and the side view. The pixel size of the captured video is 2048*2048, the camera frame rate is 30 frames / second, and the synchronization of shooting is ensured by setting the timestamp through an external trigger of the signal generator.

[0076] S2, through the deep learning-based image segmentation method, several video images taken by the overhead camera were segmented to obtain the mask binary image of each zebrafish instance, and the image refinement method was used to refine the mask to obtain the preliminary skeleton results of the zebrafish; some of the preliminary skeleton results are as follows Figure 3 shown.

[0077] Specifically, the deep learning-based image segmentation method described in this embodiment is the Mask RCNN method. Mask RCNN is a general object instance segmentation model that is improved by adding a segmentation branch based on the detection of Faster RCNN. It can not only detect targets in the image, but also provide a high-quality segmentation result for each target; among them: the specific instance segmentation calculation method is the existing technology and will not be described in detail in this invention.

[0078] Furthermore, the image thinning method used in this embodiment is the Zhang thinning algorithm. The Zhang thinning algorithm is a classic morphological thinning method that can thin out the skeleton to pixel width based on the shape of the connected domain. The specific implementation of the algorithm is part of the existing technology and will not be described in detail in this invention.

[0079] However, in this step, the deep learning-based image segmentation method performs poorly for the intersection scene of multiple zebrafish; therefore, the accuracy of the segmentation results varies greatly, and the skeleton results need to be further screened and processed.

[0080] S3, screening and processing the preliminary skeleton results obtained in step S2, eliminating skeleton results with segmentation errors in cross-scenes, and saving the remaining corrected and corrected skeleton results as positive samples, and then performing head detection on all positive samples to determine the head endpoint of the zebrafish in the skeleton results;

[0081] Among them: Since multiple zebrafish will have cross - occlusion in the top - down view, for the skeleton results in the cross - scene, the correctly segmented skeleton results based on the deep - learning method described in step S1 are retained, the incorrect but correctable skeleton results are retained after removing the bone spurs, and the completely incorrect results are discarded, thereby creating a premise for improving the data integrity in the detection link and ensuring the stability of the tracking results.

[0082] Specifically, the skeleton analysis and processing strategy proposed in the present invention is used to screen and process several preliminary skeleton results obtained in step S2. The skeleton analysis and processing strategy specifically includes the following five constraint conditions:

[0083] (a) In a single zebrafish segmentation instance, if the number of skeleton branches is greater than 3, the skeleton result is determined as a negative sample and excluded.

[0084] (b) For the skeleton results with the number of skeleton branches equal to 3 in a single zebrafish segmentation instance, correction is performed, that is, after removing the bone spurs, it is saved as a positive sample; where: the bone spur is the shortest skeleton branch.

[0085] (c) Two instances with skeleton crossing are negative samples and are excluded.

[0086] (d) If the masks of two instances are in contact, and one of the instances is determined as a negative sample by the above conditions,

[0087] then the other instance is also determined as a negative sample and excluded.

[0088] (e) Other instance samples are all positive samples and are retained.

[0089] Specifically, as Figure 3 shown, when the number of skeleton branches is greater than 3, it indicates that the segmentation result has an error (i.e., there is a situation of cross - occlusion). In the present invention, instance segmentation is actually a process of obtaining the target contour. However, it is difficult to distinguish good and bad samples for the contour respectively. Therefore, from the perspective of the skeleton, an ideal skeleton of a zebrafish is a single line from head to tail. However, sometimes the abdomen of the female fish is larger, resulting in an additional bone spur in the refined segmentation result of the female fish (it can also be said that its skeleton result has three skeleton branches); so both the skeleton with 1 and 3 branches can be screened as positive samples, and any situation where the number of skeleton branches is greater than 3 is an error in the segmentation result and needs to be excluded. In summary, through the above five constraint conditions, it is possible to effectively avoid the deficiency of incorrect segmentation of the skeleton results of a single zebrafish instance due to cross - occlusion when performing instance segmentation on several video images taken by a top - down camera.

[0090] Further, in the above constraints, to determine condition (a) and condition (b), the number of branches of the skeleton and the length of each branch need to be judged in the following way. Specifically, the present invention uses Figure 4 the shown filtering kernel, which is a 5*5 weight matrix with an inner layer value of 100, a middle layer value of 15, and an outer layer value of 1; through this filtering kernel Figure 5 the shown binary skeleton image is filtered. It can be seen from the figure that the kernel weight orders of magnitude of this filtering kernel are different from the inside to the outside. Through this characteristic, the number of skeleton pixel points appearing in each circle of the neighborhood within the range of 5*5 pixels can be judged according to different order of magnitude combinations, so as to screen out the skeleton endpoints and intersection points. In this embodiment, pixel values 148, 149, 150, and 151 are selected as intersection points, and pixel values 116, 117, and 118 are selected as skeleton endpoints; encoding starts from each endpoint in the form of Freeman chain code, along the direction of the skeleton, and when an intersection point is encountered, it is considered that the skeleton branch ends; as Figure 5 shown, by this method, the number of skeleton branches in each zebrafish skeleton instance and the length of each skeleton branch can be recorded; for the skeleton results with the number of skeleton branches greater than 3, they are directly judged as negative samples and excluded. For those with the number of skeleton branches equal to 3, the shortest bone spur (skeleton branch) is removed to ensure the accuracy of subsequent head detection, and then an ideal linear skeleton is obtained. The sample after removing the bone spurs is as Figure 6 shown.

[0091] In the above solution, the Freeman chain code is a coding method that encodes by tracking line segment sub-elements according to the position direction, so as to record the line segment trajectory and length. The specific algorithm of the present invention will not be elaborated. Conditions (c) and (d) in the constraints can be realized by simple coding.

[0092] Secondly, the filtered skeleton results saved as positive samples are filtered again using Figure 4 the shown filtering kernel. At this time, pixel values 116, 117, and 118 represent the endpoints of the skeleton. At this time, the two endpoints only represent the head or tail of the zebrafish. Window matrices centered on the endpoints are respectively selected to take values on the mask, and the eigenvalues of the covariance matrix of the window are calculated. The value-taking process is as Figure 7 shown (in the figure, the fish-shaped contour represents the mask contour of the zebrafish, the dots represent the endpoint coordinates obtained after filtering, the rectangular frame represents the window matrix centered on the endpoint, and the values within the mask contour are set to 1, and others are set to 0). The larger eigenvalue is taken as the head of the zebrafish to be tracked subsequently and is used as the only point for our subsequent tracking. The detected head endpoints are as Figure 8 shown. It should be noted that each head endpoint uniquely specifies a skeleton here. Therefore, when matching and tracking the head endpoints, the skeleton information of each target can be saved at the same time.

[0093] In summary, through the above steps, the skeleton results of each zebrafish instance after screening and processing and the head coordinates of each skeleton result can be obtained. This method can effectively eliminate the deficiencies of low data integrity and high frequency of incorrect matching events in the tracking results caused by target crossing and occlusion in the video acquisition field of view.

[0094] S4. According to the characteristic that the swimming distance of zebrafish in a continuous image sequence is limited, use the heuristic data association algorithm based on the minimum Euclidean distance to perform inter-frame matching on all temporal continuous positive samples obtained in step S3. At the moment when no target crossing or occlusion occurs, obtain the longest initial non-crossing trajectory splicing segment of each target zebrafish as long as possible, and define a trajectory segment with a time length equal to 1 frame as a short trajectory segment, and a trajectory segment with a length greater than 1 frame as a long trajectory segment.

[0095] Among them: in the heuristic data association algorithm, the inter-frame target matching method is as follows:

[0096] O i,t-1 and O j,t respectively represent any zebrafish individual (instance) in the (t - 1)-th frame and the t-th frame; N t-1 and N t are the numbers of zebrafish in the (t - 1)-th frame and the t-th frame respectively; D(O i,t-1 , O j,t ) represents the Euclidean distance between O i,t-1 and O j,t ; match the targets with the minimum Euclidean distance, and the matching method is as follows:

[0097]

[0098] Among them: m(O i,t-1 , O j,t ) = 1 means that the i-th target and the j-th target are connected and matched; if in the (t - 1)-th frame, there are two target individuals O i,t-1 and O k,t-1 that may be matched with the target O j,t in the t-th frame, then the trajectory segment is disconnected at this time, and the matching method is as follows:

[0099]

[0100] In addition, when the fish cross, affected by the deep learning segmentation method, the number of detected instances in the frames before and after the crossing moment may change. The determination criterion for the change in the number of instances is defined as:

[0101]

[0102] The above formula means that if the number of detected targets changes between the front and back frames, then C(O i,t-1 , Oj,t ) = 1; At this time, there may be two target individuals O in the (t - 1)th frame i,t-1 and O k,t-1 matching with the target O in the tth frame j,t At this time, if D(O i,t-1 , O j,t ) and D(O k,t-1 , O j,t ) differ by more than Thr d , O j,t shows a situation of being very close to the target O i,t-1 and very far from the target O k,t-1 , we will match the target O j,t with the target O i,t-1 in the following way:

[0103]

[0104] where: Thr d is the minimum distance threshold; interpreted as: a sufficiently small distance threshold due to the fish being stationary or a special matching distance, set to 30 pixel values in the present invention.

[0105] Through the above steps, initial trajectory segments that are as long as possible and non - crossing and belong to each zebrafish target can be obtained. Due to the instability of the deep - learning image segmentation method for multi - target crossing scenarios and the frame - by - frame processing characteristics of the skeleton post - processing method, there will be a frequent alternation of positive and negative samples in a multi - target crossing scenario for a period of time. Trajectories are connected among positive samples, and negative samples help to disconnect trajectory segments. After step S4 of the present invention, trajectory segments of different lengths will be connected; for the convenience of subsequent description, here a trajectory segment with a time length equal to 1 frame is defined as a short trajectory segment, and a trajectory segment with a length greater than 1 frame is defined as a long trajectory segment.

[0106] S5. Extract an improved HOG feature map for the rigid body region of the upper - half body of the zebrafish back, select the HOG features of multiple targets within the longest non - crossing time period and train an SVM. Use the trained model to infer the probability that all long trajectory segments belong to different zebrafish individuals, and use the Hungarian matching algorithm to connect multi - target long trajectory segments according to the trajectory segment matching probability and temporal relationship to complete 2D tracking;

[0107] where: as Figure 9 shown, the improved HOG feature is used to achieve the connection between long trajectory segments and is extracted in the following way:

[0108] S11. Scale the zebrafish image samples segmented in step S2 to a specified scale size and rotate them to the same direction; then mark the foreground pixel points as 1 and the background pixel points as 0 to obtain a binary image;

[0109] S12. Convert the original image input in step S11 into a HOG feature map through the HOG algorithm. Specifically, the HOG feature is the histogram of oriented gradients feature. By calculating the pixel differences in the horizontal and vertical directions of each pixel, the gradient feature of each pixel point is calculated, and then the final HOG feature is obtained through normalization in the form of regional statistics. Converting the image into a HOG feature map using the HOG algorithm is well-known in the art and will not be elaborated in this invention.

[0110] S13. Set a 9*9 pixel area as a Cell, and coarsen the binary map into N Cells. If the number of pixel points with a pixel value of 1 in a Cell is greater than a given threshold, then this Cell is considered highly information-related; otherwise, this Cell is low information-related.

[0111] S14. Set a 4*4 Cell area as a Block. When all Cells in a Block are low information-related, define this Block as a useless block; otherwise, it is a useful block.

[0112] S15. Match the pixel areas of the above-marked useful blocks with the HOG feature map one by one, and spirally output a certain number of Blocks marked as useful blocks to obtain the final improved HOG feature. The feature extraction process is as Figure 9 shown.

[0113] In the above method for extracting the improved HOG feature, rotating the zebrafish instance image samples to the same direction and then scaling them to the specified scale size can ensure that the dimensions of the HOG features obtained are consistent even when the target sizes are different. Secondly, designing useful feature blocks and useless feature blocks according to the information relevance can obtain an improved HOG feature with a smaller dimension and richer data. At the same time, adopting the above steps can make the HOG feature have rotational invariance on the one hand, and on the other hand, only retain the feature vectors related to the dorsal texture of the individual, eliminate the background factors, reduce the dimension of the features, and enhance the generalization ability of the output features, which can effectively improve the reliability, correctness, and integrity of the connection of long trajectory segments.

[0114] Furthermore, the present invention selects the multi-object HOG features within the longest non-crossing time period to train the Support Vector Machine (SVM), and uses the trained model to infer the probability that each long trajectory segment belongs to different zebrafish individuals. Then, the Hungarian matching algorithm is used to connect the trajectory segments according to the matching probability and temporal relationship of the trajectory segments. Among them: SVM is a discriminant model in machine learning models. By means of supervised learning, it finds the optimal classification hyperplane, and can achieve the classification of high-dimensional samples by combining non-linear kernels. This method is a classic classification method and will not be elaborated in this invention.

[0115] The method of combining HOG features with SVM has good global accuracy. Even if there are occasional classification errors in long trajectory segments, the probability of correct classification exceeds 70%. Therefore, this method can accurately identify long trajectory segments. However, if the above method is used for short trajectory segments, classification errors may occur, resulting in incorrect tracking of identity information.

[0116] S6. To avoid the situation where classification errors lead to incorrect tracking of identity information, the present invention uses a data association algorithm based on motion constraints to connect short trajectory segments with long trajectory segments to complete two-dimensional multi-object tracking.

[0117] Specifically: Select a short trajectory segment m, define its occurrence time as T, and select the long trajectory segment n that is closest to T before time T from the trajectory segments of each target as the target to be matched. The matching method for the trajectory segment m and any long trajectory segment n to be matched is as follows:

[0118]

[0119] In the formula: Vt mn represents the time difference between the start time of the trajectory segment m and the end time of the long trajectory segment n, in frames; represents the distance factor; represents the angle factor; The calculation formula of is as follows:

[0120]

[0121] In the formula: d mj represents the distance between the start time coordinate of the trajectory segment m and the j-th end time coordinate in the long trajectory segment, a mj represents the absolute value of the difference between the target angle at the start time of the trajectory segment m and the target angle at the j-th end time in the long trajectory segment. Correspondingly, d mn represents the distance between the start coordinate of the short trajectory segment m and the end coordinate of the long trajectory segment n, a mn Similarly, according to the calculated weight J, select the long trajectory segment with the largest weight to connect with the short trajectory segment to complete the two-dimensional multi-object tracking task.

[0122] S7. Use the Faster RCNN method to detect the fish body region (ROI) of each zebrafish instance in the side view image collected in step S1, and the detection results are as Figure 10As shown in the figure; among them: the head points of the side view target that can match the top view target must be within the ROI; in addition, Faster RCNN is an image detection method based on a deep learning convolutional network, proposed by Girshick in 2016, using a two-stage network based on Region Proposal Network (RPN), which can not only ensure the detection accuracy, but also further improve the detection speed compared with previous detection methods; this method is a classic method and will not be elaborated in this invention;

[0123] On this basis, the present invention finds the optimal matching point that matches the top view target within the fish body area of each zebrafish instance by designing an objective optimization function and a particle optimization method;

[0124] Among them: the objective optimization function is:

[0125] g(x) = δRepro(x t , x) + Dis(X last , X tri ), δ ∈ [0, 1]

[0126] In the above formula: Repro(x t , x) represents the reprojection error, using pixel distance; x represents the particle point coordinates of the side view; x t represents the head matching point coordinates of the top view; Dis(X last , X tri ) represents the distance constraint error, using Euclidean distance; X last represents the three-dimensional coordinate point of this target at the previous moment, and X tri represents the three-dimensional coordinate point after triangulation of the current particle; δ represents the balance weight of the reprojection error and the distance constraint error, with a value of 0.4;

[0127] The specific particle optimization method is as follows:

[0128] Select a target to be matched in the top view frame by frame, and then randomly define 8 particles within the fish body area of the side view of this frame; calculate the objective optimization function values of each particle respectively, and the particle with the smallest objective optimization function value is the optimal particle, and let the other 7 particles move in the direction of the optimal particle in the current iteration round; the moving distance and the moving angle γ follow the following two distributions:

[0129] φ ∼ U((1 - μ)l, (1 + μ)l);

[0130] γ ∼ U(-σ, σ);

[0131] where \(l\) represents the pixel distance of the particle from the optimal point; \(\sigma\) represents the random direction quantity, which is taken as \(15^{\circ}\) in this design; \(\mu\) represents the random weight of the distance, which is taken as \(0.4\) in this example. The moving direction and distance of the particle are as Figure 11 shown.

[0132] Optimization stop criterion: When the five iteration times are reached or the difference in the pixel distances of the optimal particles between two adjacent optimizations is less than the set threshold, which is \(5\) in this example, the optimal matching point that matches the top-down target is obtained.

[0133] S8. Triangulate the found optimal matching point with the head and skeleton results in the top-down image of the corresponding zebrafish instance, calculate the three-dimensional skeleton coordinates of the corresponding zebrafish, and inherit the two-dimensional identity information into three dimensions to achieve three-dimensional multi-target zebrafish skeleton-level tracking. The spatio-temporal changes in three-dimensional tracking are as Figure 12 shown, where Figures (a), (b), and (c) are the three-dimensional spatio-temporal trajectories of each fish, and Figure (d) is the overlapping graph of the trajectories of the three fish.

[0134] Furthermore, in order to demonstrate the advancement of this method and quantify the tracking results, the present invention selects a situation with more cross-scenes to show the comparison. The specific data are shown in Tables 1 and 2.

[0135] Table 1 Data Complexity Indicator Table

[0136]

[0137] Table 2 Tracking Evaluation Indexes

[0138]

[0139] Table 1 is the evaluation table of the occlusion complexity of the tracking data selected by the present invention. Among them, NC, OF / TF, CT / TT, AC, and MI respectively represent the number of occlusion events, the ratio of the number of occluded frames to the total number of frames, the ratio of the number of targets participating in occlusion in the total duration to the total number of targets, the duration of each occlusion, and the number of mirror targets. The arrow after the index in the table indicates the positive or negative relationship between the index and the occlusion complexity.

[0140] Table 2 is a comparison between the present invention and an advanced current method for tracking model animals, idtracker.ai. In the table, MOTA is the multi-target tracking accuracy index, Frag represents the number of trajectory segments in tracking, and ID_Sw represents the number of identity exchanges. The arrow after the index in the table indicates the correlation between the index and the quality of the tracking method. The experimental results show that this method can at least track the three-dimensional trajectories of three zebrafish, the integrity of the data is increased by an average of 10% (MOTA), and the false matching rate of the data is reduced by 50% compared with the existing method (ID_Sw).

[0141] In addition, it should be noted that for the specific embodiments described in this specification, the shapes of their components, the names taken, etc. may be different. Any equivalent or simple changes made according to the structure, features, and principles described in the inventive concept of this invention patent are included within the protection scope of this invention patent. Those skilled in the technical field to which this invention pertains can make various modifications, supplements, or use similar means for substitution to the specific embodiments described, as long as they do not deviate from the structure of this invention or exceed the scope defined by this claims book, they should all fall within the protection scope of this invention.

Claims

1. A method for tracking the three-dimensional movement trajectories of multiple zebrafish, characterized in that, The steps are as follows: S1, synchronously collect video of the social interaction behavior of zebrafish from the top-down view angle and the side view angle; S2, perform instance segmentation on several video images taken from the top-down view angle respectively through an image segmentation method based on deep learning to obtain the binary mask image of each zebrafish instance, and use the image thinning method to thin the mask respectively to obtain the preliminary skeleton results of the zebrafish; S3, screen and process several preliminary skeleton results obtained in step S2, eliminate the skeleton results with segmentation errors in the cross scene, and save the remaining correct and corrected skeleton results as positive samples, and then perform head detection on all positive samples respectively to determine the head endpoints of the zebrafish in the skeleton results; S4, use a heuristic data association algorithm based on the minimum Euclidean distance to perform inter-frame matching on all temporally continuous positive samples obtained in step S3 to obtain the initial non-crossing trajectory splicing segments of each target zebrafish, and define those with a time length equal to 1 frame as short trajectory segments and those with a length greater than 1 frame as long trajectory segments; S5, extract the improved HOG feature map of the rigid body area of the upper half of the zebrafish's back, select the HOG features of multiple targets within the longest non-crossing time period and train the SVM, use the trained model to infer the probability that all long trajectory segments belong to different zebrafish individuals, and use the Hungarian matching algorithm to connect the multi-target long trajectory segments according to the trajectory segment matching probability and temporal relationship to complete 2D tracking; S6, use a data association algorithm based on motion constraints to connect the short trajectory segments with the long trajectory segments to complete 2D multi-target tracking; S7, use the Faster RCNN method to detect the body area of each zebrafish instance in the side view image collected in step S1 respectively, and find the optimal matching point that matches the top-down target within the body area of each zebrafish instance through the target optimization function and the particle optimization method; S8, perform triangulation on the found optimal matching point with the head and skeleton results in the top-down view image of the corresponding zebrafish instance, calculate the three-dimensional skeleton coordinates of the corresponding zebrafish, and inherit the two-dimensional identity information to the three-dimensional to achieve three-dimensional multi-target zebrafish skeleton-level tracking; Among them: the data association algorithm based on motion constraints in step S6 is specifically: Select a short trajectory segment m, define its occurrence time as T, and select the long trajectory segment n that is closest to T before time T from the trajectory segments of each target as the target to be matched; then the matching weight between the trajectory segment m and any long trajectory segment n to be matched is as follows: Where: Vt mn represents the time difference between the start time of the trajectory segment m and the end time of the long trajectory segment n, with the unit of frame; represents the distance factor; represents the angle factor; The calculation formula of where: d mj represents the distance between the starting time coordinate of the short trajectory segment m and the j-th ending time coordinate in the long trajectory segment, a mj represents the absolute value of the difference between the target angle at the starting time of the short trajectory segment m and the target angle at the j-th ending time in the long trajectory segment; according to the calculated weight J, the long trajectory segment with the largest weight is selected to be connected to the short trajectory segment.

2. The multi-objective zebrafish three-dimensional motion trajectory tracking method according to claim 1, characterized in that, In step S3, several preliminary skeleton results obtained in step S2 are screened and processed through a skeleton analysis and processing strategy, and the skeleton analysis and processing strategy specifically includes the following five constraint conditions: (a) In a single zebrafish segmentation instance, if the number of skeleton branches is greater than 3, the skeleton result is determined as a negative sample and eliminated; (b) For the skeleton results with the number of skeleton branches equal to 3 in a single zebrafish segmentation instance, remove the bone spurs and save them as positive samples; among them: the bone spur is the shortest skeleton branch; (c) Two instances with skeleton crossing are negative samples and are eliminated; (d) If the masks of two instances are in contact and one of the instances is determined to be a negative sample by the above conditions, then the other instance is also determined to be a negative sample and is excluded; (e) All other instance samples are positive samples and are retained.

3. The multi-object zebrafish three-dimensional motion trajectory tracking method according to claim 2, characterized in that In the skeleton analysis and processing strategy, the number and length of the skeleton branches are determined as follows: Filter the binary images of several skeleton results obtained in step S2 through the set filter kernels to screen out the endpoints and intersection points of the skeletons; then, encode from each endpoint in the form of Freeman chain codes, along the direction of the skeleton, and when an intersection point is encountered, it is considered that the skeleton branch ends, thereby determining the number of skeleton branches in a single zebrafish segmentation instance and the length of each skeleton branch.

4. The multi-objective zebrafish three-dimensional motion trajectory tracking method according to claim 3, wherein In step S3, the specific method for head detection is as follows: Perform filtering operations on the several skeleton results saved as positive samples obtained through screening and processing to find the two endpoints of the skeleton, respectively select window matrices centered on these two endpoints to take values on the mask, and calculate the eigenvalues of the covariance matrix of the window. The endpoint with the larger eigenvalue is defined as the head endpoint of the zebrafish skeleton.

5. The multi-object zebrafish three-dimensional motion trajectory tracking method according to claim 1, characterized in that In step S4, the specific content of frame-by-frame matching of all temporally continuous positive samples according to the heuristic data association algorithm is as follows: O i,t-1 and O j,t respectively represent any individual zebrafish in the (t - 1)-th frame and the t-th frame. D(O i,t-1 , O j,t ) represents the Euclidean distance between O i,t-1 and O j,t . The target is matched with the minimum Euclidean distance, and the matching method is as follows: where: m(O i,t-1 ,O j,t ) = 1 indicates that the i-th target and the j-th target are connected and matched; if in the (t - 1)th frame, there are two target individuals O i,t-1 and O k,t-1 that may be matched with the target O j,t in the t-th frame, then the trajectory segment is disconnected at this time, and the matching method is as follows: When fish cross, the number of instances detected in the frames before and after the crossing moment may change. At this time, target individuals O i,t-1 and O k,t-1 in frame t-1 may simultaneously exist j,t and are matched with target O in frame t. The matching method is as follows: In the above formula, D(O i,t-1 ,O j,t ) and the difference between D(O k,t-1 ,O j,t ) is greater than Thr d , the target O j,t is matched with the target O i,t-1 ; Thr d is the minimum distance threshold.

6. The multi-objective zebrafish three-dimensional motion trajectory tracking method according to claim 1, wherein In step S5, the improved HOG feature map is extracted in the following way: S11, Scale the zebrafish image samples segmented in step S2 to a specified scale size and rotate them to the same direction; then mark the foreground pixel points as 1 and the background pixel points as 0 to obtain a binary image; S12, Convert the original image input in step S11 into a HOG feature map through the HOG algorithm; S13, Set a 9*9 pixel area as a Cell, coarsen the binary image into N Cells. If the number of pixel points with a value of 1 in a Cell is greater than a given threshold, then this Cell is considered highly information-related, otherwise this Cell is low information-related; S14, Set a 4*4 Cell area as a Block. When all Cells in a Block are low information-related, define this Block as a useless block, otherwise it is a useful block; S15, Match the pixel areas of the above-marked useful blocks with the HOG feature map one by one, and spirally output a certain number of Blocks marked as useful blocks to obtain the final improved HOG feature map.

7. The multi-objective zebrafish three-dimensional motion trajectory tracking method according to claim 2, characterized in that, In step S7, the specific objective optimization function is as follows: g(x) = δRepro(x t , x) + Dis(X last , X tri ), δ ∈ [0, 1] In the above formula: Repro(x t , x) represents the reprojection error, using pixel distance; x represents the particle point coordinates of the side view; x t represents the head matching point coordinates of the top view; Dis(X last , X tri ) represents the distance constraint error, using Euclidean distance; X last represents the three-dimensional coordinate point of the target at the previous moment, X tri represents the three-dimensional coordinate point after triangulation of the current particle; δ represents the balance weight of the reprojection error and the distance constraint error, with a value of 0.4; The specific particle optimization method is as follows: Select a target to be matched frame by frame in the top view, and then design n particle points in the fish body area in the side view of this frame to calculate the objective optimization function values of the particles respectively, and move the non-optimal particles in the direction of the optimal particle. The moving distance and the moving angle γ follow the following two distributions: φ~U((1 - μ)l, (1 + μ)l); γ~U(-σ, σ); In the above formulas: l represents the pixel distance of the particle from the optimal particle point; σ represents the direction random quantity, with a value of 15°; μ represents the distance random weight, with a value of 0.4; Optimization stop objective: When five iteration times are reached or the difference in the optimal particle pixel distances between two adjacent optimizations is less than the set threshold, the optimal matching point matching the top-down target is obtained.

Images