Zebra fish behavioral analysis method based on deep learning algorithm

Abstract

The invention relates to a zebra fish behavioral analysis method based on a deep learning algorithm, and the method comprises the steps: obtaining zebra fish video data which comprises a plurality of video frames; determining a plurality of fish body key points based on the target zebra fish behavior; using a deep learning algorithm to extract fish body key points in each video frame to obtain key point data; analyzing the dynamic change of the key point position of each fish body by using the key point data, and screening out the target zebra fish behaviors in the video data according to the accumulated value of the dynamic change to obtain a plurality of target behavior units; and calculating a kinematic parameter of each target behavior unit, and performing clustering analysis on the kinematic parameters to obtain each motion mode of the target zebra fish behavior. According to the method, the behavior pattern of the zebra fish can be automatically identified and decoded with high precision, so that the efficiency and the accuracy of behavior analysis are greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of zebrafish behavioral analysis, and in particular to a zebrafish behavioral analysis method based on a deep learning algorithm. Background Art

[0002] With the rapid development of neuroscience and behavioral biology, the demand for accurate analysis of animal behavior is growing. As a widely used model organism, zebrafish plays an important role in neuroscience, developmental biology and pharmacology research. Traditional zebrafish behavior analysis relies on manual observation and recording. Although this method can provide qualitative descriptions, it has limitations such as low efficiency, strong subjectivity, and difficulty in expanding to large-scale studies. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide a zebrafish behavioral analysis method based on a deep learning algorithm, which can automatically and highly accurately identify and decode the behavioral patterns of zebrafish, thereby greatly improving the efficiency and accuracy of behavioral analysis.

[0004] The technical solution adopted by the present invention to solve the technical problem is: to provide a zebrafish behavior analysis method based on a deep learning algorithm, comprising:

[0005] S0 obtains zebrafish video data, where the video data includes multiple video frames;

[0006] S1 determines several key points of the fish body according to the behavior of the target zebrafish to be analyzed;

[0007] S2 uses a deep learning algorithm to sequentially detect the key points of the fish body in each video frame to obtain key point data, wherein the key point data includes the number of time frames, the number of key points, and the coordinates and coordinate confidence of each key point of the fish body;

[0008] S3 uses the key point data to analyze the dynamic changes of the positions of the key points of each fish body, and selects the target zebrafish behavior in the video data according to the cumulative value of the dynamic changes to obtain a plurality of target behavior units;

[0009] S4 calculates the kinematic parameters of each target behavior unit, and performs cluster analysis on the kinematic parameters to obtain various movement patterns of the target zebrafish behavior.

[0010] Further, the use of the key point data to analyze the dynamic changes of the positions of the key points of the fish body includes:

[0011] Calculate the corresponding angle data according to the coordinates of each key point of the fish body, and the angle data is used to represent the positional relationship between each key point of the fish body;

[0012] Performing local mean calculation and time difference operation on the angle data in sequence to obtain dynamic change data of the angle data;

[0013] The dynamically changing data is filtered.

[0014] Furthermore, the corresponding angle data is calculated according to the coordinates of each key point of the fish body, including:

[0015] Taking the set key points of the fish body as the reference coordinate origin, the azimuth of each key point of the fish body is calculated as the angle data, or the angle change of each key point of the fish body relative to the previous key point of the fish body is calculated as the angle data.

[0016] Furthermore, the target zebrafish behavior is the continuous tail movement of the zebrafish, and the key point of the fish body is set to be the key point of the zebrafish's tail, and the reference coordinate origin is the key point of the fish body closest to the zebrafish's head.

[0017] Furthermore, the target zebrafish behavior in the video data is screened out according to the dynamically changing cumulative value to obtain a plurality of target behavior units, including:

[0018] Calculating the accumulated value of the dynamically changing data after filtering to obtain accumulated angle information;

[0019] Performing maximum and minimum value filtering on the accumulated angle information to obtain a curve measurement value;

[0020] A target behavior unit is extracted from the video data, where the target behavior unit is a portion of the curve metric value that is greater than a set threshold value in a continuous time.

[0021] Furthermore, the method further includes the step of screening the extracted target behavior units, specifically including:

[0022] Get the minimum unit interval, minimum unit duration, and minimum and maximum curvature;

[0023] Merge two target behavior units whose interval is smaller than the minimum unit interval;

[0024] The target behavior units whose duration is shorter than the minimum unit duration and whose curve metric value is shorter than the minimum maximum curvature are removed.

[0025] Furthermore, before the step of sequentially performing local mean calculation and time difference operation on the angle data, the step of smoothing the angle data using a spatial and temporal box filter is also included.

[0026] Furthermore, the method also includes the step of performing coordinate and time interpolation processing on the key point data.

[0027] Furthermore, before the step of performing cluster analysis on the kinematic parameters to obtain each movement pattern of the target zebrafish behavior, the step of preprocessing the kinematic parameters is also included, including:

[0028] After performing mean filling and Z value normalization processing on the kinematic parameters in sequence, data with all null values ​​are deleted to obtain parameter processing data;

[0029] The parameter processing data is subjected to dimensionality reduction processing.

[0030] Furthermore, the kinematic parameters include maximum angular velocity, average angular velocity, maximum head yaw angle, and half-beat angle change, wherein the half-beat angle change includes the change in the zebrafish body angle at the end of the first and second half-beats.

[0031] Beneficial Effects

[0032] Due to the adoption of the above-mentioned technical scheme, the present invention has the following advantages and positive effects compared with the prior art: the present invention performs key point detection and behavior segmentation on zebrafish behavior videos, and quantitatively analyzes the segmented behavior characteristics, thereby converting complex behavior sequences into calculable parameters, providing a basis for the recognition and classification of behavior patterns, and then applying an unsupervised clustering algorithm to effectively decode the behavior patterns of zebrafish with restricted movement. This method not only reduces the dependence on expert knowledge, but also greatly improves the efficiency and accuracy of behavior analysis; the method of the present invention is applicable to various behavioral research scenarios, including but not limited to neuroscience research, drug screening, behavioral genetics analysis, etc., and provides researchers with a powerful tool to deeply understand the behavioral characteristics and neural mechanisms of zebrafish. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a schematic diagram of a flow chart of an embodiment of the present invention;

[0034] Figure 2 It is a schematic diagram of the code architecture of an embodiment of the present invention;

[0035] Figure 3 is a schematic diagram of key point angle calculation in an embodiment of the present invention;

[0036] Figure 4 is a schematic diagram of angle data and tail curvature parameters of an embodiment of the present invention;

[0037] Figure 5 is a schematic diagram of a kinematic parameter calculation process according to an embodiment of the present invention;

[0038] Figure 6 This is a display diagram of a zebrafish swimming video frame according to an embodiment of the present invention;

[0039] Figure 7 Schematic diagram of the starting position of the tail curve parameter CurveMeasure curve and Bout according to an embodiment of the present invention;

[0040] Figure 8 is a schematic diagram of a curve showing the change of the angle of the end section of each bout with time according to an embodiment of the present invention;

[0041] Fig. 9 It is a schematic diagram of the change of bout 8 segment angles and curve parameters of various cluster centers over time according to an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0043] The embodiments of the present invention relate to a zebrafish behavioral analysis method based on a deep learning algorithm, comprising:

[0044] S0 obtains zebrafish video data, wherein the video data includes a plurality of video frames;

[0045] S1 determines several key points of the fish body according to the behavior of the target zebrafish to be analyzed;

[0046] S2 uses a deep learning algorithm to sequentially detect the key points of the fish body in each video frame to obtain key point data, wherein the key point data includes the number of time frames, the number of key points, and the coordinates and coordinate confidence of each key point of the fish body;

[0047] S3 uses the key point data to analyze the dynamic changes of the key point positions of each fish body, and selects the target zebrafish behavior in the video data according to the cumulative value of the dynamic changes to obtain a number of target behavior units;

[0048] S4 calculates the kinematic parameters of each target behavior unit, and performs cluster analysis on the kinematic parameters to obtain various movement patterns of the target zebrafish behavior.

[0049] The present implementation is further described below in conjunction with the accompanying drawings and system architecture.

[0050] This implementation method realizes the behavior analysis of behavior-restricted zebrafish videos. The behavior quantification adopts the markerless posture estimation technology, namely key point detection; the behavior recognition adopts the unsupervised clustering method. The algorithm flow design is as follows: Figure 1 shown.

[0051] The code structure of the zebrafish behavior analysis algorithm is as follows Figure 2 The data quantification of behavior was achieved by writing Python programs, and the behavior segmentation and classification was achieved by writing MATLAB.

[0052] The specific process is as follows:

[0053] S1: Tail keypoint detection in zebrafish video

[0054] The deep learning algorithms rtmdetect (Real-Time Multi-Person Detection) and rtmpose (Real-Time Multi-Person Pose Estimation) are used to automatically detect and locate the key points of the zebrafish's tail in the video.

[0055] S101: Create a training dataset

[0056] Capture video frames of zebrafish. These video frames should cover as many behaviors of zebrafish as possible, especially the parts with large tail swings, to ensure the comprehensiveness of model training. Manually annotate 9 key points on the collected video frames, and save the key point position data together with the video frames as the training data set.

[0057] S102: Adjust parameters and train the model

[0058] The model is trained on the training set and the parameters are adjusted to minimize the error function.

[0059] S103: Model Reasoning

[0060] RTMPose outputs the coordinates and confidence of each key point. These key point data provide accurate location information for subsequent behavior analysis. The final key point detection obtains N T×C×V data files, where T is the number of time frames (time), C is the number of channels (channel), including x-coordinates, y-coordinates, and confidence, V is the number of key points detected, and N is the number of videos.

[0061] S2: Process key point data to provide precise parameters for behavior analysis

[0062] In order to improve the accuracy of behavior recognition and facilitate subsequent algorithm calculations, the data is further processed.

[0063] S201: Data interpolation

[0064] The key point data is interpolated in coordinates and time to fill in the missing data points. The third-order B-spline interpolation algorithm is used to ensure the continuity and smoothness of the data. For example, the x and y coordinate interpolation increases the number of key points from 9 to 38; the time t interpolation increases the data rate from 200Hz of the original video to 700Hz.

[0065] S202: Angle calculation

[0066] According to the key point coordinates obtained after interpolation, the angle information of the key point of the zebrafish tail is calculated. Figure 3 As shown, the coordinate axis is rotated 90 degrees counterclockwise, and the angle of each key point is calculated with the first point (the key point close to the head) as the origin as the angle data of the zebrafish tail key point. The angle between each key point and the line segment from the origin can also be further calculated as the angle data of the zebrafish tail key point.

[0067] S203: Data smoothing

[0068] The raw tail angles are smoothed using box filters in space and time. This reduces background noise caused by small bends in the tail measurement due to jitter in the positioning of each point on the tail, especially during the intervals between bouts.

[0069] S3: Identify and segment the basic unit (bout) of zebrafish behavior by analyzing the changes in key point data

[0070] A continuous tail movement of a fish is called a bout. This step will detect all the bouts in a video.

[0071] S301: Calculate the parameters to measure the change of the tail curve

[0072] First, the angle data of the key points of the zebrafish tail are calculated locally to eliminate short-term fluctuations and highlight long-term trends. Subsequently, a time difference operation is applied to the local mean data to determine the dynamic changes in the angular position data. Next, the noise and irregularities in the data are further reduced through smoothing filtering. Based on the filtered data, two cumulative sum calculations are performed to integrate the angle data of multiple key points into a cumulative angle information superCumSegAngle. Finally, the cumulative angle information is filtered for maximum and minimum values ​​to strengthen the overall trend of the curve and suppress any remaining high-frequency disturbances, thereby obtaining the tail curve measurement CurveMeasure for bout detection, as shown in Figure 4 shown.

[0073] S302: bout detection

[0074] Set the threshold of the tail curvature parameter to simplify the processing or adjust the setting based on experience. For each video data, find the critical point where CurveMeasure is greater than the threshold. These critical points are possible Bout starting points. Adjust the start and end indexes to ensure that the data that meets the conditions is included in the bout. Finally, filter the detected Bouts: calculate the length of the interval between bouts, merge those shorter than the specified minimum-interbout-interval into one bout; clear the bouts with a length less than minimum-bout-length; clear the bouts with CurveMeasure less than minimum-max-curvature, and get the final bout detection result.

[0075] S4: Extract kinematic parameters from segmented behavioral units to provide quantitative features for behavioral classification

[0076] The behavior quantification data corresponding to the start and end time of the Bout are divided to obtain the behavior data of a single Bout, and then the kinematic parameters of each Bout are calculated. The data processing process is as follows Figure 5 As shown: Behavior segmentation divides the data set into bouts, T 1 -T B is the time of each bout, B is the number of bouts, and K is the number of kinematic parameters calculated.

[0077] S401: Calculate bout curvature

[0078] The bout curvature is an important parameter to describe the movement of the zebrafish tail during a bout, and can be used to calculate subsequent kinematic parameters, such as the maximum curvature of the beat and the maximum curvature change rate. Using the position information of the key points, the curvature of the zebrafish tail is calculated and estimated by the second-order derivative:

[0079] where F and G describe the x and y midline positions as a function of coordinate s and time t.

[0080] S402: bout kinematic parameter calculation

[0081] For each bout detected, there are 235 kinematic parameters available for calculation, including maximum angular speed, mean angular speed, maximum head yaw, and half-beat angle change (C1 angle, C2 angle): the change in the body angle of the fry at the end of the first and second half-beats, etc.

[0082] S5: Apply unsupervised clustering algorithms to automatically identify behavioral patterns.

[0083] S501: bout kinematic parameter preprocessing

[0084] The above 235 kinematic parameters are not applicable to all behaviors of zebrafish. In some cases, some kinematic parameters cannot be calculated or have no practical significance and need to be preprocessed. First, null values ​​and outliers are processed by mean filling, and then Z-score is performed: z = (x-μ) / σ, where x is a measured value, μ is the mean, and σ is the standard deviation. Finally, the kinematic parameters with all null values ​​are deleted, indicating that the kinematic parameters cannot be calculated or have no distinguishing effect on the behavior of this type of fish.

[0085] S502: Data Dimensionality Reduction

[0086] Dimensionality reduction can retain the most important features of high-dimensional data, remove noise and unimportant features, and facilitate subsequent clustering. The t-SNE (t-distributed Stochastic Neighbor Embedding) method can map data in high-dimensional space to low-dimensional space and retain the local characteristics of the data set.

[0087] S503: Clustering using unsupervised algorithms

[0088] Then observe whether the clustering results have typical movement patterns. If each category contains multiple movement patterns, extract the data of this category and repeat the above steps until each category has a typical movement pattern. You can use k-means or density valley clustering (clusterdv) to analyze the behavior characteristics. For example, call the k-means function clustering, first observe the data distribution, select the input parameter k value and the number of iterations N.

[0089] The following is further explained through specific experimental data.

[0090] We collected videos of different zebrafish swimming under electrical stimulation of different voltages. Each video was about 30 minutes long, with a frame rate of 200Hz. The experimental conditions were electrical stimulation once every minute or so, and the same video recorded the behavioral changes of the same zebrafish under the same electrical stimulation. Since biological experiments require recording whole-brain calcium imaging while recording videos, the zebrafish's head was fixed in the video, and only the movement of the tail could be observed.

[0091] like Figure 6As shown in the figure, the behavior of zebrafish larvae under this condition has the following characteristics compared to free-swimming behavior: 1. The zebrafish's head is fixed, which restricts its movement, and the conclusions obtained from free-swimming behavior cannot be used. 2. The swing of the tail cannot produce displacement, so when the coordinate system is constructed with the zebrafish's head as the origin, the swing angle of the tail will be much larger than when it is free-swimming, and the tail will even bend to coincide with the head. 3. Concentrated electrical stimulation causes it to frequently produce a variety of behaviors, which are very rapid and constantly changing. Figure 6 From frame 994 to frame 999, the movement of the zebrafish's tail changes greatly, and the difference between adjacent frames is only 5ms, which shows how fast its tail moves.

[0092] According to the above technical scheme, the video was analyzed behaviorally, taking the video of fish No. 8 stimulated by 10V voltage as an example. The results of bout detection are as follows Figure 7 , and its accuracy can be confirmed by comparing it with the video.

[0093] Through multiple unsupervised clustering, Bout can be divided into 7 categories. The results are as follows Figure 8 and Fig. 9 The experimental results show that the method of the present invention can effectively distinguish different behavior patterns and successfully decode 7 basic behavior patterns.

[0094] In summary, the present invention was applied to the behavioral video of the head-fixed zebrafish collected in the experiment, and the results proved the effectiveness of the algorithm and system in practical applications.

Claims

1. A zebrafish behavioral analysis method based on a deep learning algorithm, characterized in that: include: S0 obtains zebrafish video data, wherein the video data includes a plurality of video frames; S1 determines several key points of the fish body according to the behavior of the target zebrafish to be analyzed; S2 uses a deep learning algorithm to sequentially detect the key points of the fish body in each video frame to obtain key point data, wherein the key point data includes the number of time frames, the number of key points, and the coordinates and coordinate confidence of each key point of the fish body; S3 uses the key point data to analyze the dynamic changes of the positions of the key points of each fish body, and selects the target zebrafish behavior in the video data according to the cumulative value of the dynamic changes to obtain a plurality of target behavior units; S4 calculates the kinematic parameters of each target behavior unit, and performs cluster analysis on the kinematic parameters to obtain various movement patterns of the target zebrafish behavior.

2. The method according to claim 1, characterized in that The method of analyzing the dynamic changes of the positions of the key points of the fish body by using the key point data includes: Calculate the corresponding angle data according to the coordinates of each key point of the fish body, and the angle data is used to represent the positional relationship between each key point of the fish body; Performing local mean calculation and time difference operation on the angle data in sequence to obtain dynamic change data of the angle data; The dynamically changing data is filtered.

3. The method according to claim 2, characterized in that The method of calculating the corresponding angle data according to the coordinates of each key point of the fish body includes: Taking the set key points of the fish body as the reference coordinate origin, the azimuth of each key point of the fish body is calculated as the angle data, or the angle change of each key point of the fish body relative to the previous key point of the fish body is calculated as the angle data.

4. The method according to claim 3, characterized in that The target zebrafish behavior is the continuous tail movement of the zebrafish, and the key point of the fish body is set to be the key point of the zebrafish tail, and the reference coordinate origin is the key point of the fish body closest to the head of the zebrafish.

5. The method according to claim 2, characterized in that: The target zebrafish behavior in the video data is screened out according to the cumulative value of the dynamic change to obtain a plurality of target behavior units, including: Calculating the accumulated value of the dynamically changing data after filtering to obtain accumulated angle information; Performing maximum and minimum value filtering on the accumulated angle information to obtain a curve measurement value; A target behavior unit is extracted from the video data, where the target behavior unit is a portion of the curve metric value that is greater than a set threshold value in a continuous time.

6. The method according to claim 5, characterized in that The step of screening the extracted target behavior units is also included, including: Get the minimum unit interval, minimum unit duration, and minimum and maximum curvature; Merge two target behavior units whose interval is smaller than the minimum unit interval; The target behavior units whose duration is shorter than the minimum unit duration and whose curve metric value is shorter than the minimum maximum curvature are removed.

7. The method according to claim 2, characterized in that Before the step of sequentially performing local mean value calculation and time difference operation on the angle data, the method further includes the step of smoothing the angle data using a spatial and temporal box filter.

8. The method according to claim 1, characterized in that Before step S3, the method further includes the step of performing coordinate and time interpolation processing on the key point data.

9. The method according to claim 1, characterized in that: Before the step of performing cluster analysis on the kinematic parameters to obtain various movement patterns of the target zebrafish behavior, the step of preprocessing the kinematic parameters is also included, including: After performing mean filling and Z value normalization processing on the kinematic parameters in sequence, data with all null values ​​are deleted to obtain parameter processing data; The parameter processing data is subjected to dimensionality reduction processing.

10. The method according to claim 1, characterized in that The kinematic parameters include maximum angular velocity, average angular velocity, maximum head yaw angle, and half-beat angle change, which includes the change in the zebrafish body angle at the end of the first and second half-beats.

Images