A hierarchical segmentation and behavior pattern analysis method for fruit fly flight trajectories

Abstract

The present invention provides a method for hierarchical segmentation of fruit fly flight trajectories and analysis of behavioral patterns, comprising a fruit fly flight behavior trajectory segmentation module, a fruit fly individual flight behavior pattern extraction and analysis module, and an interaction pattern extraction and analysis module. The present invention calculates the trajectory's adaptive curvature threshold and segments the trajectory according to changes in spatial features. Based on a customized energy function, unsupervised clustering of variable-length trajectories is achieved, assigning category information to basic behavioral patterns. After a single clustering, a long trajectory splitting method is used to discover component behavioral patterns. Unsupervised clustering is used to classify patterns based on the motion parameters and position parameters of the interaction trajectory segments, analyzing interaction pattern types such as chasing and collision avoidance, or discovering new interaction patterns. The present invention can be used to study fruit fly flight behavior, reducing over-segmentation in the behavior segment extraction process and resolving the issue of component behavioral pattern discovery and interaction pattern discovery relying on manual analysis.

Description

Technical Field

[0001] The present invention belongs to the field of bioethology, and specifically relates to a method for hierarchical segmentation and behavioral pattern analysis of fruit fly flight trajectories. The method assigns comprehensive semantic annotations to fruit fly flight trajectories, which can improve research efficiency and accuracy of results. The method can also be applied to discovering fruit fly flight patterns, basic research on biological neural mechanisms, automatic annotation of fruit fly flight behavior datasets, and research and construction of biomimetic drone swarm behavior. Background Art

[0002] Drosophila, a commonly used model organism in research, boasts high individual reproductive capacity, ease of care, and a rich array of flight behaviors. Because fruit flies, honey bees, albatrosses, and other species exhibit Lévy flight in statistical distribution and share similar behavioral spaces, fruit fly behavior is easily tracked, and experimental environments and conditions are more controllable. Therefore, research on fruit fly flight behavior is representative, and the relevant research ideas and methods can be extended to studying flight behavior patterns in other related organisms. The study of fruit fly flight behavior should be approached from two perspectives: the individual behavioral space and the interactions between groups. Most behaviors and intentions of organisms can be described using a patterned behavioral space. Furthermore, social behavior within a group depends on the occurrence of interactions between individuals. Therefore, the study of social behavior relies on accurate and reliable detection of interactions between individuals.

[0003] With the development of artificial intelligence, the study of fruit fly flight behavior can now be more systematically investigated using machine learning or deep learning methods. Currently, methods for extracting individual fruit fly flight behavior patterns are relatively simple and lack the ability to determine high-level semantic spaces. These methods rely primarily on manual analysis by experts, such as observing fruit fly flight videos or flight trajectories and recording their flight movements. This manual approach is difficult, inefficient, subjective, and lacks comprehensiveness and precision, making it difficult to meet the needs of spatial analysis of flying organism behavior. Current trajectory analysis methods typically employ a trajectory segmentation module, followed by clustering to quickly identify similar trajectory categories. Trajectory segmentation typically uses three criteria: fixed duration, limited change angle, and dwell detection. However, in fruit fly flight trajectory research, these criteria are limited in generating trajectory segments due to the varying durations of different behaviors, leading to over-segmentation. Furthermore, performing unsupervised clustering alone yields limited information. Longer trajectory segments may be composed of multiple basic behavioral patterns, representing higher-level semantic spaces. Existing work lacks an understanding of this issue. In addition, the extraction of interactive motion patterns is also an indispensable part of the study of Drosophila flight behavior, but existing work lacks a module for automatic analysis of Drosophila interactive patterns. Summary of the Invention

[0004] In response to the limitations of the above-mentioned research on fruit fly flight behavior, the purpose of the present invention is to provide a method for hierarchical segmentation and behavior pattern analysis of fruit fly flight trajectories. The method comprises three technical modules: (1) a fruit fly flight behavior trajectory segmentation module, which is used to extract fruit fly flight trajectory segments; (2) a fruit fly individual flight behavior pattern extraction and analysis module, which is used to extract individual flight behavior patterns from fruit fly flight trajectories; and (3) a fruit fly individual flight interaction pattern extraction and analysis module, which is used to detect trajectory segments where fruit flies generate interactive behaviors and, based on the obtained high-level semantic space, to determine the relationship patterns of interactive movements between individuals. Through the individual behavior pattern analysis method comprising three modules, a comprehensive flight behavior pattern analysis of fruit fly trajectories is achieved, providing a systematic behavior analysis method for related research.

[0005] To achieve the above objectives, the present invention designs a method for hierarchical segmentation of fruit fly flight trajectories and analysis of behavioral patterns, which includes the following three technical modules:

[0006] (1) Fruit fly flight behavior trajectory segmentation module: Use the interpolation method to calculate the Bezier continuous curve where the trajectory point is located, calculate the variance fluctuation of the curvature of the trajectory point, screen and obtain the trajectory point set, and then determine the adaptive bending threshold of the trajectory, define the local maximum bending point, distortion point, bending point, and flat point, segment the trajectory, and generate variable-length fruit fly flight behavior trajectory segments.

[0007] (2) Fruit fly individual flight behavior pattern extraction and analysis module: The extracted trajectory segments are clustered in an unsupervised manner in rounds, and the clustering results are evaluated using a customized comprehensive scoring system of height, angle, and length energy functions. In order to obtain the basic behavior patterns corresponding to the fruit fly flight behavior trajectory segments, the long trajectory is judged in a high-level semantic space based on the order and number of basic behavior patterns, thereby discovering new combined action patterns. After each clustering, the long trajectory splitting method is used to perform hierarchical splitting of the trajectory.

[0008] (3) Fruit fly individual flight interaction pattern extraction and analysis module: Based on the fruit fly trajectory dataset, the individual behavior vector of the fruit fly at each moment is analyzed, the interaction pattern is detected at each time point of the fruit fly trajectory, and its interaction behavior pattern is judged based on the position parameters and motion parameters of the interaction trajectory segment.

[0009] Preferably, the specific contents of step (1) of constructing the trajectory segmentation module are:

[0010] (1-1) To obtain a set of fruit fly flight trajectory segments for subsequent behavioral pattern analysis, the spatial features of the fruit fly trajectory are calculated and the original fruit fly trajectory is segmented based on the changes in the spatial features. The spatial features include trajectory curvature (cur) and torsion (tor).

[0011] (1-2) Trajectory data is usually expressed as a set of trajectory points. Its essence can be regarded as time series data containing spatial information. The fruit fly trajectory data is expressed as:

[0012]

[0013] Among them, Traj is a trajectory in the trajectory dataset, are the three-dimensional trajectory points that make up the trajectory, t is the timestamp of the trajectory point, i is the number of the fruit fly individual, T is the time length, and N is the number of fruit flies.

[0014] By calculating the curvature and torsion of the trajectory, spatial features are extracted, and then the trajectory segments in the real data set and the simulation data set are extracted. Since the trajectory data is a discrete point, it is difficult to directly calculate the curvature and torsion. Therefore, the third-order Bessel function is used to calculate the continuous trajectory function and obtain the curvature and torsion:

[0015]

[0016] Among them, B t (t B ) is the third-order Bessel function, P t-1 、P t and P t+1 are three adjacent discrete trajectory points, t B Is P t-1 and P t+1 The interval obtained after normalizing the time between two trajectory points can be used to calculate the curvature of the trajectory point at time t, P e and P f It is a control point calculated by three adjacent trajectory points. The calculation method is as follows:

[0017] Calculate P t-1 With P t and P t With P t+1 The midpoint P m and P n , then calculate P m and P n The midpoint P g , by translating the line segment P m P n Move to the end point P g With P t coincide, and the two endpoints P of the line segment m ' and P n ' is the control point P e and P f .

[0018] The curvature of the trajectory point can be obtained by substituting the continuous function of the trajectory into the formula, where the calculation formula of the curvature of the trajectory point is:

[0019]

[0020] Among them, cur t is the function used to calculate the curvature of the trajectory point at time t, ndh Is P t-1 and P t+1 The time between two trajectory points is normalized to the time corresponding to t. Since the time interval between each trajectory point is the same, t ndh Should be a constant 0.5, B t ′(t ndh ) is the first-order derivative of the continuous trajectory function, B t ″(t ndh ) is the second-order derivative of the continuous trajectory function. The continuous trajectory function can be solved in the following way: The calculation formula of the trajectory point torsion is:

[0021]

[0022] Among them, tor t is the function used to calculate the torsion of the trajectory point at time t, B t ″′(t ndh ) is the third-order derivative of the continuous trajectory function.

[0023] (1-3) By plotting the curve of curvature changing with time, we can see that the process of significant change in the trajectory is exactly the process of forming a curve peak. Therefore, the starting point and end point of the peak can be used to segment the trajectory segment. In addition, the torsion of the intersection of trajectories in different planes is not zero. This intersection is used to segment the trajectory so that the trajectory points in the trajectory segment are located in the same plane. Using these trajectory features, the trajectory points can be divided into four types, including:

[0024]

[0025] Among them, P curmax is the local maximum bending point, P tor is the twist point, P curved is the bending point, P smooth For a smoother point, M cur is the bending threshold. According to the change of trajectory point type, a set of segmentation points can be obtained. The segmentation standard is: P does not appear in the trajectory. curmax In the case of the first occurrence of P curved When P appears in the trajectory, the point is used as the split point. curmax In the case of the first occurrence of P smooth When P appears in the trajectory, the point is used as the split point; tor, use this point as the split point.

[0026] For bending threshold M cur The method for determining is as follows: Due to the difference in curvature between flat points and curved points, if a curved point is added to a flat point set, the curvature variance of the point set will change significantly, and vice versa. Based on this characteristic, the optimal curvature threshold is found. The curvature threshold is used to divide the trajectory point set on a trajectory into two parts. The variance and mean of the two sets are calculated. The specific optimization goal is:

[0027]

[0028] in, and Take M at the bending threshold cur The two trajectory point sets obtained when Var W Take M as the bending threshold cur When, collection The curvature variance, mean Q and mean W Take M as the bending threshold respectively cur When the average curvature of the two sets, score cur The score obtained by selecting the current bending threshold. The lower the score, the better the effect of distinguishing flat points and bending points by the threshold. The bending threshold M cur The range is [cur min ,cur max ], cur min and cur max are the minimum and maximum curvatures in the trajectory, respectively.

[0029] According to the segmentation point set calculated by the segmentation standard, the smoothed fruit fly trajectory data is segmented to obtain the fruit fly flight trajectory segments:

[0030] Frags = {Frag k |k∈{1,...,M},length≥4}

[0031] Among them, Frags is the set of trajectory segments obtained after segmentation, Frag k For the track segment numbered k, the track segment length must be greater than or equal to 4.

[0032] Preferably, the specific content of step (2) of constructing a module for extracting and discovering the flight behavior pattern of individual fruit flies is:

[0033] (2-1) The fruit fly individual flight behavior pattern extraction module uses the customized height, length, and angle energy functions based on the fruit fly flight trajectory segment data obtained in module 1 to calculate the similarity between the fruit fly flight trajectory segments. Through round-by-round unsupervised clustering, the long trajectory splitting method is used after each clustering to obtain the classification results of the fruit fly flight behavior trajectory segments. The representative trajectory segments in each class are analyzed based on expert knowledge. Then, based on the basic flight behavior patterns obtained, the long trajectory segments are judged in the high-level semantic space according to the order and number of basic behavior patterns.

[0034] (2-2) For fruit fly flight trajectory segments of different time series lengths, if the traditional distance function is directly used to calculate the distance, spatial position preprocessing is required. In order to reduce the amount of calculation and improve the accuracy of similarity calculation, a new energy function calculation method is defined in combination with DTW (Dynamic Time Warping), specifically:

[0035]

[0036]

[0037] Among them, Dis Frag (Frag a ,Frag b ) calculates the distance between two trajectory segments, Calculate the distance between two trajectory points, Frag a and Frag b are two different trajectory segments, and Belong to Frag a and Frag b , and The distance is the closest, w1, w2, w3 are the weights in the energy function, They are length energy function, angle energy function, and height energy function, specifically:

[0038] l dis =|l a -l b |

[0039] Among them, l a 、l b To compare the length of the unit trajectory segment where the trajectory point is located, the unit trajectory segment is composed of the trajectory point and its previous moment trajectory point.

[0040]

[0041] Among them, vec a , vec bIt is the direction vector from the previous moment's trajectory point to the comparison trajectory point.

[0042]

[0043] Among them, l a 、l b To compare the length of the unit trajectory segment where the trajectory point is located, θ dis is the direction angle between the two trajectory segments.

[0044] (2-3) The segmented trajectory segments are still long and the behavior categories obtained by a single clustering are small. To solve this problem, a long trajectory segment splitting method is added after unsupervised clustering. After completing a clustering, the shortest trajectory in the trajectory cluster is used as the standard sample, and the trajectory with a length exceeding the threshold L is split into two segments. max The trajectory segments are split and a new round of clustering is performed on the newly generated trajectory segment set. This process is repeated until no new behavior categories are generated or the clustering threshold is reached. Specifically:

[0045] A n =cluster(Frag n )|n∈[1,m]

[0046] Among them, cluster represents the clustering method used, here is K-Medoids, A n It is n The clustering result obtained by sub-trajectory clustering, Frag n is the set of split trajectory segments after the n-1th clustering, m is the clustering number threshold.

[0047] The specific criteria for trajectory segment splitting are:

[0048] Use the custom energy function to calculate the distance between each trajectory point in the standard sample and each trajectory point in the trajectory segment to be split, find the trajectory point with the smallest matching distance with each trajectory point in the standard sample, take the first matching point and the last matching point in the target sample as the splitting point, and split the target sample with a length greater than L. min The trajectory segment is added to the trajectory segment set for the next round of clustering.

[0049] Based on the clustering results obtained in the above steps, a representative trajectory segment is selected from each cluster. The basic behavior pattern corresponding to the trajectory segment is determined by judging the changes in speed and direction of the trajectory segment and the curvature calculated in module 1. Specifically,

[0050] S={type(A m ,v t ,a t ,Vec t )}

[0051] Among them, S is the basic action mode set, type(A m ,v t ,a t ,Vec t ) is a function that calculates individual flight behavior patterns based on expert knowledge analysis and related motion characteristics. m is the clustering result of the bottom trajectory segment obtained by the last clustering, v t and a t They are t The speed and acceleration of the individual at each moment. This function can calculate the behavioral movement pattern corresponding to each category in the single clustering result, thereby realizing the automatic extraction of the basic pattern of variable-length trajectory of the fruit fly individual during flight, and giving category information, such as straight line, turning, spiral, and pause. After introducing dynamic information, straight line can be distinguished by deceleration, acceleration, and constant speed. The size of the turn can be distinguished by arc and angular velocity.

[0052] After extracting the basic patterns of the fruit fly flight trajectory, we add a high-level semantic space judgment method. Based on the hierarchical trajectory splitting process in the long trajectory classification method and the types, occurrence times, and order of the basic action patterns, we can obtain the basic action pattern composition of the trajectory segment. Specifically,

[0053] S senior ={type senior (S, Frag n )|n∈[1,m-1]}

[0054] Among them, S senior The compositional movement pattern of the fruit fly was discovered, type senior (S, Frag n ) can analyze the high-level semantic space composition of fruit flies by given basic action patterns and corresponding trajectory segment sets, combined with the biological research on fruit fly flight behavior patterns, and set it as the judgment space. When the high-level semantic space composition conditions are met, complex behavior categories are discovered for the trajectory segments and category information is given. For example, saccade behavior is composed of straight walking and rapid turns greater than 300° / s in the basic action space; yaw motion is characterized by body rotation and a spiral trajectory.

[0055] Preferably, the specific content of step (3) of constructing a module for extracting and discovering the flight interaction pattern of individual fruit flies is:

[0056] (3-1) The fruit fly individual flight interaction pattern extraction module obtains the neighboring distribution of fruit flies in the fruit fly cluster at each moment, analyzes the individual behavior vector of the fruit flies at each moment, and detects the interaction pattern at each time point of the fruit fly trajectory based on expert knowledge. Based on the three parameter curves of the interaction trajectory pair, namely the speed difference, the angle of the movement direction, and the distance difference curve, the module uses unsupervised clustering to judge the specific interaction pattern or discover new interaction patterns.

[0057] (3-2) Interactions between individuals usually occur within a certain spatial range and last for a period of time. Therefore, it is first necessary to obtain the set of neighboring individuals of each individual at each moment to determine whether interaction occurs.

[0058] The nearest neighbor search is performed on each fruit fly individual at each time point using a KD tree to obtain the neighboring distribution set of each individual that changes over time. Based on the interaction behavior occurrence criteria, the behavior vector of each fruit fly individual at each moment is analyzed to determine the time range in which the individual interacts during the movement process. Specifically,

[0059]

[0060] in, For individual i t The neighboring distribution set at time, M t is the number of individuals existing at that moment.

[0061] This set can be used to find the trajectory segments where individuals interact. The interaction criteria are that the distance to other individuals is less than Eps and the duration exceeds T last , and the angle between the center of mass of individual fruit flies is less than the threshold θ max , here the movement direction calculated in module 2 is used as an approximation; by comprehensively analyzing the above standards, the time periods in which all fruit fly individuals interact are obtained, and the complex interactive behavior patterns are determined for these time periods.

[0062] (3-3) Based on the interaction point detection, the three parameters of the trajectory segment where the interaction occurs are calculated according to the relationship between the position and motion information of the individuals: speed difference, motion direction angle, and distance difference. Specifically:

[0063]

[0064] Among them, V diff is the interaction time between two fruit flies [t s ,t e ] range of speed difference, and is the relationship between individual i and individual j t The speed of time.

[0065]

[0066] Among them, θ diff is the interaction time between two fruit flies [t s ,t e ] range of the movement direction angle set, and is the relationship between individual i and individual j t The direction of motion vector at the moment, the direction is from the current trajectory point to the trajectory point at the next moment.

[0067]

[0068] Among them, dis diff is the interaction time between two fruit flies [t s ,t e ] range of the distance difference set, and is the relationship between individual i and individual j t The spatial position at a moment, Used to calculate the Euclidean distance between two spatial locations.

[0069] (3-4) Two interacting trajectory segments are considered as a certain interaction mode. For similar interaction modes, the changes in the positions and motion information of the interacting individuals are similar. Three parameter curves are generated with time as the horizontal axis and the three parameters as the vertical axis. By comparing the similarity of the parameter curves, the category of the interaction mode can be analyzed, specifically:

[0070] Dist inter (m,n)=k1*dtw(V m ,V n )+k2*dtw(θ m ,θ n )+k3*dtw(dis m ,dis n )

[0071] Where Dist inter (m,n) is the energy function for calculating the two interaction modes, V m and V n For individuals m and individuals n The speed difference parameter curve, θ m and θ n For individuals m and individuals n The motion direction angle parameter curve, dis m and dis n For individuals m and individuals n The distance difference parameter curve, dtw(Curvem ,Curve n ) is used to calculate the DTW distance between two curves, Curve m and Curve n They are two parameter curves of the same type, k1, k2, and k3 are the weights of the distance differences between the three curves in the total distance.

[0072] Based on this energy function, the distances between all interaction patterns are calculated, and K-Medoids is used to cluster the interaction patterns, specifically:

[0073]

[0074] Among them, S inter is the collection of all interaction modes, is a certain type of interaction pattern after clustering, m max is the total number of clusters, By analyzing the three parameter curves of the representative trajectory segments in the cluster, specific interaction modes or new interaction modes can be identified. For example, the distance difference parameter curve of the chasing interaction mode shows a downward trend, the speed difference parameter curve shows an upward trend, and the movement direction angle parameter curve shows a decreasing trend; the speed difference, distance difference, and movement direction angle parameter curves of the mating interaction mode are close to the x-axis and show a flat trend.

[0075] This completes the construction of a pattern discovery and analysis method for individual behavior based on fruit fly trajectory points.

[0076] The application of the fruit fly flight behavior trajectory segmentation module of the module (1) requires calculating the required trajectory spatial features, including curvature and torsion, for a given fruit fly trajectory data set, defining the type of trajectory points based on the spatial features of the trajectory, and extracting the required behavior trajectory segments.

[0077] The application of the Drosophila individual flight behavior pattern extraction module in the module (2) requires a customized energy function to use unsupervised clustering to find a set of trajectory segments with similar spatial shapes. This energy function needs to overcome a series of problems caused by the influence of spatial distance and different sequence lengths. In addition, after unsupervised clustering, a long trajectory splitting method needs to be used to increase the behavioral analysis process in the high-level semantic space.

[0078] The application of the fruit fly individual flight interaction pattern extraction module of the module (3) requires processing a given fruit fly trajectory data set to obtain the adjacent distribution of fruit flies and the individual behavior vectors of fruit flies at each moment in space, and combining expert knowledge to detect the interaction at each time point of the fruit fly trajectory. Based on the position parameters and motion parameters between the interaction trajectory segments, the determination of complex interaction behavior patterns and the discovery of new patterns are achieved.

[0079] The present invention realizes a hierarchical segmentation and behavior pattern analysis method of fruit fly flight trajectory, which can be applied to discovering fruit fly flight patterns, conducting basic research on biological neural mechanisms, and being used for the research and construction of bionic drone swarm behavior.

[0080] The beneficial effects of the present invention are:

[0081] (1) Define a trajectory segment segmentation standard, calculate the curvature and torsion of the trajectory, distinguish the types of trajectory points by threshold method, calculate the average, maximum or median value of the curvature of the flat point set by calculating the variance fluctuation of the trajectory point curvature, and obtain the adaptive threshold of the trajectory. This can reduce the occurrence of over-segmentation during the segmentation process;

[0082] (2) A hierarchical clustering method for individual behavior patterns of fruit flies based on the long trajectory splitting method is proposed. Using multiple rounds of K-Medoids clustering and the long trajectory splitting method, the basic behavior patterns of the bottom-level trajectory segments are identified. From this, the combined action patterns of the upper-level trajectory segments are analyzed from the bottom up to obtain the high-level semantic space. This can solve the problem of low efficiency of manual analysis and difficulty in analyzing the composition of action patterns.

[0083] (3) A method for clustering fruit fly interaction patterns based on fruit fly motion information (speed, angle, and position) is proposed. By parameterizing the relationship between pairs of fruit fly trajectories, an energy function of the interaction pattern is constructed and clustered based on this. The clustering results include four interaction categories: chasing, mating, avoiding, and other unknown patterns. This can provide effective assistance for the discovery and analysis of new fruit fly interaction patterns.

[0084] (4) The invention can be used to study the flight behavior of fruit flies, and provides a method for discovering and analyzing individual behavioral patterns in a cluster. It can be applied to discovering fruit fly flight patterns, basic research on biological neural mechanisms, automatic annotation of fruit fly flight behavior data sets, and research and construction of bionic drone cluster behavior. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] Figure 1 Schematic diagram of the framework for pattern discovery and analysis methods of individual Drosophila behavior.

[0086] Figure 2 Flowchart of the technical method for pattern discovery and analysis of individual Drosophila behavior.

[0087] Figure 3 Schematic diagram of Drosophila trajectory segmentation.

[0088] Figure 4 T-SNE dimensionality reduction plot.

[0089] Figure 5 Schematic diagram of the calculation of the energy function of Drosophila trajectory points.

[0090] Figure 6 An example diagram of the behavioral pattern of individual fruit fly flight components.

[0091] Figure 7 Schematic diagram of the Drosophila interactive behavior assay.

[0092] Figure 8 Schematic diagram of the Drosophila interaction model. DETAILED DESCRIPTION

[0093] The following examples and drawings are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0094] In this example, data annotation was performed using fruit fly trajectory data acquired using a multi-camera-based fruit fly flight behavior data acquisition platform, using this scenario as an example. As a commonly used model organism in basic biological research, fruit flies are known for their high reproductive capacity, ease of care, and diverse flight behaviors, similar to those of other flying organisms. Therefore, the analytical research methods for individual and swarm fruit flies are representative and can be extended to flight behavior studies of other related organisms.

[0095] However, labeling the flight behavior of individual fruit flies is extremely time-consuming and labor-intensive, and manual observation may lead to misjudgment of action categories during analysis. For some more complex trajectories, it is difficult for researchers to classify them based on visual images and extract more information from them.

[0096] In order to improve the efficiency of studying fruit fly flight behavior, enhance the accuracy of behavior classification, and provide researchers with a systematic method, the present invention proposes a pattern analysis method for fruit fly individual behavior, which includes three modules, to automatically extract individual behavior patterns from fruit fly trajectory datasets and solve the problem of data annotation in fruit fly related research. The method framework is as follows: Figure 1 shown.

[0097] The labeling experiment was conducted on a computing server equipped with one HUAWEI Ascend 910 NPU and four HUAWEI Kunpeng 920 CPUs. Each HUAWEI Ascend 910 NPU has 32GB of HBM memory and 1200GB / s of bandwidth, achieving 320 TFLOPS of computing power at FP16 precision. Each HUAWEI Kunpeng 920 CPU has 48 cores and a typical frequency of 2.6GHz / 3.0GHz. The server has 768GB of RAM and 1.5TB of SSD storage. The computing server runs EulerOS 2.0 (SP8) and CANN 5.1.RC2.1, along with Python 3.7.10, PyTorch 1.8.1+ascend.rc2, and MindSpore 1.8.1.

[0098] The application process of the method for pattern discovery and analysis of individual behavior based on fruit fly trajectory points provided by the present invention in the data annotation scenario is as follows. The technical flow chart is as follows: Figure 2 shown.

[0099] Step A: Division of fruit fly flight behavior trajectory segments. First, filter the fruit fly trajectory dataset to eliminate noise interference during the fruit fly trajectory data collection process. Then calculate the curvature and torsion of the trajectory, calculate the optimal curvature threshold for each trajectory segment, define the trajectory point type through the threshold method, generate a segmentation point set based on the change of the trajectory point type, and divide the trajectory. After testing, the curvature threshold is taken as the curvature threshold M. cur , which will change in different trajectories. This method can be used to quickly generate a set of flight behavior trajectory segments for clustering. The visualization of the segmented trajectory is shown in the following figure. Figure 3 shown.

[0100] Step B: Extraction of individual fruit fly flight behavior patterns. Perform round-by-round unsupervised clustering on the fruit fly trajectory segment dataset obtained in step A. After each round of clustering, the long trajectory segment is split to generate a set of trajectory segments for the next round of clustering. The parameters w1, w2, and w3 are set to 0.4, 0.4, and 0.2 respectively, and the clustering round threshold is set to 10. The Silhouette-Coefficient (SC) indicator and Calinski-Harabasz (CH) are used in each clustering round to determine the optimal number of clusters. The scoring weights of the two indicators are set to 0.5 and 0.5 respectively. The clustering results obtained determine the basic action pattern of the underlying trajectory segment through expert knowledge and related motion features, and then analyze the composition of the action pattern of the long trajectory before splitting. According to the correspondence between the obtained trajectory segment and the action pattern, the individual action patterns of the trajectory dataset are annotated. The T-SNE dimensionality reduction diagram of the clustering result is shown as follows: Figure 4 As shown in the figure, the calculation process of the energy function is as follows Figure 5 As shown, the obtained composition action pattern example is as follows Figure 6 shown.

[0101] Step C: Extraction of interaction behavior patterns of individual fruit flies. Use KD tree to perform nearest neighbor search on each fruit fly individual at each time point to obtain the neighbor distribution set of each individual that changes over time. If there are more than 1 individuals in the neighbor distribution set of individual i that have always existed in the time range t and the duration is greater than the threshold T last , then proceed to the next step of judgment, where the relevant parameters of the nearest neighbor range Eps is twice the body length of the fruit fly 5.46mm, the duration threshold T last Set to 0.6s.

[0102] After the above conditions are met, calculate whether the angle between the directions of individuals meets the threshold θ max , in order to detect whether the fruit fly individuals have interactive behavior during this duration, where the angle threshold θ max Set to 90°, the interactive behavior detection diagram is as follows Figure 7 shown.

[0103] For the trajectory segments that meet the conditions, the distance difference parameter set, speed difference parameter set, and motion direction angle parameter set between the interactive trajectory segments are calculated to obtain three parameter curves. inter (m,n) calculate the distance between each interaction pattern, and then use K-Medoids to cluster, and get m max After the interactive mode, use Identify the specific category of each interaction mode and mark the interaction mode that cannot be identified as a new mode, where the parameters k1, k2, and k3 are set to 0.3, 0.3, and 0.4. After obtaining the interaction category, the trajectory dataset is annotated again, annotating the trajectory with the specific interaction category or an unidentified new mode. The example interaction mode diagram is shown below. Figure 8 shown.

Claims

1. A method for hierarchical segmentation of fruit fly flight trajectories and analysis of behavior patterns, characterized in that: The approach consists of three technical modules: (1) Drosophila flight behavior trajectory segmentation module: Use the interpolation method to calculate the Bezier continuous curve where the trajectory points are located, calculate the variance fluctuation of the curvature of the trajectory points, screen and obtain the trajectory point set, and then determine the adaptive bending threshold of the trajectory, define the local maximum bending point, distortion point, bending point, and flat point, segment the trajectory, and generate variable-length Drosophila flight behavior trajectory segments; (2) Fruit fly individual flight behavior pattern extraction and analysis module: The extracted trajectory segments are clustered in an unsupervised manner in rounds, and the clustering results are evaluated using a customized comprehensive scoring system of height, angle, and length energy functions. In order to obtain the basic behavior patterns corresponding to the fruit fly flight behavior trajectory segments, the long trajectory is judged in a high-level semantic space based on the order and frequency of the basic behavior patterns, thereby discovering new combined action patterns. After each clustering, the long trajectory splitting method is used to perform hierarchical splitting of the trajectory. (3) Fruit fly individual flight interaction pattern extraction and analysis module: Based on the fruit fly trajectory dataset, the individual behavior vector of the fruit fly at each moment is analyzed, and the interaction pattern is detected at each time point of the fruit fly trajectory. The interaction trajectory pairs are parameterized into speed difference, movement direction angle, and distance difference change. K-Medoids clustering is used to classify the interaction patterns, and a type determination function is used to determine the specific interaction type or analyze new interaction types.

2. The method according to claim 1, characterized in that In module (1), the spatial features of the fruit fly flight trajectory, including curvature and torsion, are calculated, and the original fruit fly trajectory is segmented according to the above spatial features to extract the fruit fly flight trajectory segments; Trajectory data is usually represented as a set of trajectory points. Its essence can be regarded as time series data containing spatial information. The fruit fly trajectory data is represented as: Among them, Traj is a trajectory in the trajectory dataset, is the three-dimensional trajectory point that constitutes the trajectory, t is the timestamp of the trajectory point, i is the number of the fruit fly individual, T is the time length, and N is the number of fruit flies; By calculating the trajectory curvature and torsion, spatial features are extracted, and then the trajectory segments in the real data set and the simulation data set are extracted. Since the trajectory data is a discrete point, it is difficult to directly calculate the curvature and torsion. Therefore, the third-order Bessel function is used to calculate the continuous trajectory function and obtain the curvature and torsion: t B ∈[0,1],t∈{1,...,T-1} Among them, B t (t B ) is the third-order Bessel function, P t-1 、P t and P t+1 are three adjacent discrete trajectory points, t B Is P t-1 and P t+1 The interval obtained by normalizing the time between two trajectory points is used to calculate the curvature of the trajectory point at time t, P e and P f It is a control point calculated by three adjacent trajectory points, specifically: Calculate P t-1 With P t and P t With P t+1 The midpoint P m and P n , then calculate P m and P n The midpoint P g , by translating the line segment P m P n Move to the end point P g With P t coincide, and the two endpoints P of the line segment m ' and P n ' is the control point P e and P f ; The curvature of the trajectory point is obtained by substituting the continuous function of the trajectory into the formula, where the calculation formula of the curvature of the trajectory point is: Among them, cur t is the function used to calculate the curvature of the trajectory point at time t, ndh Is P t-1 and P t+1 The time between two trajectory points is normalized to the time corresponding to t. Since the time interval between each trajectory point is the same, t ndh Should be a constant 0.5, B t ′(t ndh ) is the first-order derivative of the continuous trajectory function, B t ″(t ndh ) is the second-order derivative of the continuous trajectory function, which is solved as follows: The calculation formula of the trajectory point torsion is: Among them, tor t is the function used to calculate the torsion of the trajectory point at time t, B t ″′(t ndh ) is the third-order derivative of the continuous trajectory function.

3. The method according to claim 2, characterized in that The calculated trajectory spatial features, by plotting the curve of curvature changing with time, show that the trajectory change process is accompanied by the formation of a curve crest. Therefore, the starting and ending points of the crest are used to segment the trajectory. In addition, the torsion of the intersection of trajectories in different planes is not zero. This intersection is used to segment the trajectory so that the trajectory points in the trajectory segment are located in the same plane. Using these trajectory features, the trajectory points are divided into four types, including: Among them, P curmax is the local maximum bending point, P tor is the twist point, P curved is the bending point, P smooth For a smoother point, M cur is the bending threshold, and the segmentation point set is obtained according to the change of trajectory point type. The segmentation standard is: P does not appear in the trajectory curmax In the case of the first occurrence of P curved When P appears in the trajectory, the point is used as the split point. curmax In the case of the first occurrence of P smooth When P appears in the trajectory, the point is used as the split point; tor When , this point is used as the split point; For bending threshold M cur The method for determining is as follows: Due to the difference in curvature between smooth points and curved points, if a curved point is added to the set of smooth points, the curvature variance of the point set will change significantly, and vice versa. Based on this characteristic, the optimal curvature threshold is found. The curvature threshold is used to divide the set of trajectory points on a trajectory into two parts, and the variance and mean of the two sets are calculated. The specific optimization goal is: M cur ∈[cur min ,cur max ] in, and Take M at the bending threshold cur The two trajectory point sets obtained when Var W Take M as the bending threshold cur When, collection The curvature variance, mean Q and mean W Take M as the bending threshold respectively cur When the average curvature of the two sets, score cur The score obtained by selecting the current bending threshold. The lower the score, the better the effect of distinguishing flat points and bending points by the threshold. The bending threshold M cur The range is [cur min ,cur max ], cur min and cur max are the minimum and maximum curvatures in the trajectory, respectively; According to the segmentation point set calculated by the segmentation standard, the smoothed fruit fly trajectory data is segmented to obtain the fruit fly flight trajectory segments: Frags={Frag k |k∈{1,...,M},length≥4} Among them, Frags is the set of trajectory segments obtained after segmentation, Frag k For the track segment numbered k, the track segment length must be greater than or equal to 4.

4. The method according to claim 1, wherein In module (2), based on the fruit fly flight trajectory segment data obtained in module (1), the similarity between the fruit fly flight trajectory segments is calculated using a custom height, length, and angle energy function. Through round-by-round unsupervised clustering, and after each clustering, the long trajectory splitting method is used to obtain the classification results of the fruit fly flight behavior trajectory segments. The representative trajectory segments in each class are analyzed based on the speed and movement direction information. Then, based on the basic flight behavior pattern obtained, the long trajectory segment is judged in the high-level semantic space by the order and number of the basic behavior pattern, and the new combined behavior pattern is analyzed. For fruit fly flight trajectory segments of different time series lengths, a new energy function calculation method is defined in combination with dynamic time warping (DTW), specifically: in, Calculate the distance between two trajectory segments, Calculate the distance between two trajectory points, Frag a and Frag b are two different trajectory segments, and Belong to Frag a and Frag b , and The distance is the closest, w1, w2, w3 are the weights in the energy function, They are length energy function, angle energy function, and height energy function, specifically: l dis ＝|l a -l b | Among them, l a 、l b To compare the length of the unit trajectory segment where the trajectory point is located, the unit trajectory segment is composed of the trajectory point and its previous moment trajectory point; Among them, vec a , vec b The direction vector from the previous moment's trajectory point to the comparison trajectory point; Among them, l a 、l b To compare the length of the unit trajectory segment where the trajectory point is located, θ dis is the direction angle between the two trajectory segments.

5. The method according to claim 4, characterized in that The flight behavior trajectory segments of fruit flies are clustered by calculating the distance between individual fruit flies. The segmented trajectory segments are still long, and the behavior categories obtained by a single clustering are relatively few. To solve this problem, a long trajectory segment splitting process is added after unsupervised clustering. After completing a clustering, the shortest trajectory in the trajectory cluster is used as the standard sample, and the trajectory segments with a length exceeding the threshold L are clustered. max The trajectory segments are split and the newly generated trajectory segments are clustered again. This process is iterated until no new behavior categories are generated or the clustering threshold is reached. Specifically: A n =cluster(Frag n )|n∈[1,m] Among them, cluster represents the clustering method used, here is K-Medoids, A n is the clustering result obtained by the nth trajectory clustering, Frag n is the set of split trajectory segments after the n-1th clustering, and m is the clustering number threshold; The specific criteria for splitting the trajectory segment are: Use the custom energy function to calculate the distance between each trajectory point in the standard sample and each trajectory point in the trajectory segment to be split, find the trajectory point with the smallest matching distance with each trajectory point in the standard sample, take the first matching point and the last matching point in the target sample as the splitting point, and split the target sample with a length greater than L. min The trajectory segment is added to the trajectory segment set of the next round of clustering; According to the clustering results obtained in the above steps, a representative trajectory segment is selected from each cluster. The basic behavior pattern corresponding to the trajectory segment is determined by judging the changes in speed and direction of the trajectory segment and the spatial features such as curvature calculated in module (1). Specifically, S={type(A m ,v t ,a t ,Vec t )} Among them, S is the basic action mode set, type(A m ,v t ,a t ,Vec t ) is a function that calculates the individual flight behavior pattern based on the relevant motion characteristics, A m is the clustering result of the bottom trajectory segment obtained by the last clustering, v t and a t are the speed and acceleration of the individual at time t, respectively. This function can calculate the behavioral motion pattern corresponding to each category in the single clustering result, thereby automatically extracting the basic pattern of the variable-length trajectory of the fruit fly individual during flight, assigning category information, and selecting: straight, turning, spiral, and pause. After introducing dynamic information, the straight line is distinguished by deceleration, acceleration, and uniform speed. The size of the turn is distinguished by arc and angular velocity. After extracting the basic patterns of the fruit fly flight trajectory, we added a high-level semantic space judgment method. Based on the hierarchical trajectory splitting process in the long trajectory classification method, we obtained the basic action pattern composition of the trajectory segment. According to the type, number of occurrences, and order of the basic action pattern, we can further define it as follows: S senior ={type senior (S,Frag n )|n∈[1,m-1]} Among them, S senior The compositional movement pattern of the fruit fly was discovered, type senior (S, Frag n ) Through the given basic action patterns and the corresponding level of trajectory segment types, the high-level semantic space composition of the fruit fly is analyzed and set as the judgment space. When the high-level semantic space composition conditions are met, the complex behavior category is discovered for the trajectory segment, and the category information is given. The saccade behavior is selected, which is composed of straight walking and rapid turns greater than 300° / s in the basic action space; yaw movement, the body rotates, and the trajectory is spiral.

6. The method according to claim 1, characterized in that In module (3), the distribution of individual fruit flies in the fruit fly cluster at each moment is obtained, the behavior vector of the individual fruit flies at each moment is analyzed, the interaction pattern is detected at each time point of the fruit fly trajectory, and based on the three parameter curves of the interaction trajectory pair, namely the speed difference, the angle of the movement direction, and the distance difference curve, the specific interaction pattern is judged and analyzed through unsupervised clustering to analyze the new interaction pattern.

7. The method according to claim 6, characterized in that The nearest neighbor search is performed on the fruit fly individual at each time point using the KD tree to obtain the neighbor distribution set of each individual that changes over time. The behavior vector of the fruit fly individual at each moment is analyzed. According to the interaction behavior occurrence criteria, the time range of the individual interaction during the movement process is determined, specifically: in, is the neighboring distribution set of individual i at time t, M t is the number of individuals existing at that moment; This set is used to find the trajectory segments where individuals interact. The interaction criteria are that the distance to other individuals is less than Eps and the duration is longer than T last , and the angle between the center of mass of individual fruit flies is less than the threshold θ max , here the movement direction calculated in module (2) is used as an approximation; by comprehensively analyzing the above standards, the time periods in which all fruit fly individuals interact are obtained, and the complex interactive behavior patterns are analyzed for these time periods.

8. The method according to claim 6, characterized in that On the basis of realizing the detection of interaction mode, according to the position and motion information relationship between individuals, three parameters of the trajectory segment where interaction occurs are calculated: speed difference, motion direction angle, and distance difference. Specifically: Among them, V diff is the interaction time between two fruit flies [t s ,t e ] range of speed difference, and is the velocity of individuals i and j at time t; Among them, θ diff is the interaction time between two fruit flies [t s ,t e ] range of the movement direction angle set, and is the movement direction vector of individual i and individual j at time t, and the direction is from the current trajectory point to the trajectory point at the next moment; Among them, dis diff is the interaction time between two fruit flies [t s ,t e ] range of the distance difference set, and is the spatial position of individual i and individual j at time t, Used to calculate the Euclidean distance between two spatial locations.

9. The method according to claim 8, characterized in that The three parameters calculated are used to define an interaction mode with two interacting trajectory segments. For similar interaction modes, the changes in the positions and motion information of the interacting individuals are similar. Three parameter curves are generated with time as the horizontal axis and the three parameters as the vertical axis. By comparing the similarities of the parameter curves, the categories of the interaction modes are analyzed, specifically: Dist inter (m,n)=k1*dtw(V m ,V n )+k2*dtw(θ m ,θ n )+k3*dtw(dis m ,this n ) Where Dist inter (m,n) is the energy function for calculating the two interaction modes, V m and V n is the speed difference parameter curve between individual m and individual n, θ m and θ n is the parameter curve of the movement direction angle between individual m and individual n, dis m and dis n is the distance difference parameter curve between individual m and individual n, dtw(Curve m ,Curve n ) is used to calculate the DTW distance between two curves, Curve m and Curve n are two parameter curves of the same type, k1, k2, and k3 are the weights of the distance differences between the three curves in the total distance; Based on this energy function, the distances between all interaction patterns are calculated, and K-Medoids is used to cluster the interaction patterns, specifically: Among them, S inter is the collection of all interaction modes, is a certain type of interaction pattern after clustering, m max is the total number of clusters, By analyzing the three parameter curves of the representative trajectory segments in the clusters, the specific interaction mode or the new interaction mode was identified. The distance difference parameter curve of the chasing interaction mode showed a downward trend, the speed difference parameter curve showed an upward trend, and the movement direction angle parameter curve showed a decreasing trend; the speed difference, distance difference, and movement direction angle parameter curves of the mating interaction mode were close to the x-axis and showed a flat trend.

Images