Multi-animal ethology data analysis method and device, electronic equipment and storage medium

Abstract

This invention discloses a method, apparatus, electronic device, and storage medium for multi-animal behavioral data analysis. The method includes: acquiring a video to be analyzed; preprocessing the video; inputting the preprocessed video into a pre-trained pose estimation model to estimate the poses of the animals in the video to obtain pose estimation results; acquiring preset tracking parameters; inputting the tracking parameters into a preset improved idTracker.ai component to enable the improved idTracker.ai component to analyze the video and obtain tracking results for the animals in the video; and combining the pose estimation results with the tracking results to generate visualized animal behavior analysis results. This invention solves the technical problem of poor pose estimation results in multi-animal behavioral data analysis in the prior art.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to a method, apparatus, electronic device, and storage medium for analyzing multi-animal behavioral data. Background Technology

[0002] Multi-animal social behavior research is an important part of neuroscience research, involving many aspects such as communication, cooperation, and aggression in laboratory animals. However, data collection in multi-animal social behavior experiments is costly and easily influenced by human subjective experience, resulting in biases and deficiencies in reliability and validity. Deep learning methods can be used for label-free animal tracking and can annotate key points according to researchers' needs. Transfer learning techniques can effectively utilize pedestrian pose recognition models to estimate the poses of laboratory animals. Existing multi-animal behavioral data collection methods, such as DeepLabCut (Lauer, J., Zhou, M., Ye, S. et al. Multi-animal pose estimation, identification and tracking with DeepLabCut. Nat Methods 19, 496–504 (2022)) and SLEAP (Pereira, TD, Tabris, N., Matsliah, A. et al. SLEAP: A deep learning system for multi-animal pose tracking. Nat Methods 19, 486–495 (2022)), can assist researchers in this type of work.

[0003] idTracker.ai (Romero-Ferrero, F., Bergomi, MG, Hinz, RC et al. idtracker.ai: tracking all individuals in small or large collectives of unmarked animals. Nat Methods 16, 179–182 (2019)) is an active learning-based multi-object tracking algorithm that can be used to track multiple laboratory animals without manual animal labeling. LightweightOpenPose (Osokin, Daniil. Real-time 2d multi-person pose estimation on cpu: Lightweightopenpose. arXiv preprint arXiv:1811.12004 (2018)) was first used for human pose estimation and has been significantly optimized based on OpenPose to improve algorithm efficiency. Improving and combining these two algorithms has great potential for application in the behavioral analysis of laboratory animals.

[0004] However, existing deep learning-based multi-animal social behavior data acquisition and analysis systems have numerous hyperparameters that require manual adjustment by researchers. They often fail to adapt to different video input sizes and animal sizes within the video, resulting in poor pose estimation results. Furthermore, the multi-animal tracking component frequently requires labeling all animals to be tracked, causing significant inconvenience when tracking large numbers of small experimental animals. In addition, existing experimental animal tracking and pose estimation algorithms lack application testing on a large number of small experimental animals, and tracking software often fails to perform pose estimation, presenting users with numerous challenges related to hardware preparation and software installation. Summary of the Invention

[0005] The purpose of this invention is to overcome the above-mentioned technical deficiencies and provide a method, device, electronic device and storage medium for multi-animal behavioral data analysis, thereby solving the technical problem of poor posture estimation results in multi-animal behavioral data analysis in the prior art.

[0006] To achieve the above-mentioned technical objectives, the present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a method for analyzing multi-animal behavioral data, comprising the following steps:

[0008] Acquire the video to be analyzed and preprocess the video to be analyzed;

[0009] The preprocessed video to be analyzed is input into a pre-trained pose estimation model to estimate the pose of the animal in the video to obtain the pose estimation result.

[0010] Obtain preset tracking parameters and input the tracking parameters into a preset improved idTracker.ai component so that the improved idTracker.ai component can analyze the video to be analyzed and obtain the tracking results of the animals in the video to be analyzed.

[0011] The posture estimation results are combined with the tracking results to generate visualized animal behavior analysis results.

[0012] In some embodiments, the preprocessing process includes at least: image scaling, image rotation, image flipping, and image cropping.

[0013] In some embodiments, the training method of the pose estimation model is as follows:

[0014] After obtaining training samples and annotating the training samples with key points, the training samples are divided into training set and test set.

[0015] A neural network model is established, and the parameters of the neural network model are set according to the input size of each sample in the training set and the size of the labeled animal.

[0016] The training set is input into the neural network model after the parameters have been set to train the neural network model and obtain the pose estimation model.

[0017] In some embodiments, the neural network model is a lightweight OpenPose model.

[0018] In some embodiments, the preset tracking parameters include at least the maximum number of tracks, the range of animal region size, the range of animal pixel intensity, the video region of interest, the downsampling rate, and the time interval of the video to be analyzed.

[0019] In some embodiments, combining the posture estimation results with the tracking results to generate visualized animal behavior analysis results includes:

[0020] Calculate the center point of the attitude estimation result and the center point of the tracking result;

[0021] Calculate the Euclidean distance between the center point of the attitude estimation result and the center point of the tracking result, and construct a cost matrix based on the Euclidean distance;

[0022] Based on the cost matrix, the Kuhn-Munkres algorithm is used to optimize the matching results. Based on the optimization results, the posture estimation results and the tracking results are fused to generate visualized animal behavior analysis results.

[0023] In some embodiments, the animal behavior analysis results include at least an animal activity heatmap, an animal behavior state transition map, and an animal behavior state proportion pie chart.

[0024] Secondly, the present invention also provides a multi-animal behavioral data analysis device, comprising:

[0025] The preprocessing module is used to acquire the video to be analyzed and to preprocess the video to be analyzed.

[0026] The pose estimation result acquisition module is used to input the preprocessed video to be analyzed into a pre-trained pose estimation model to perform pose estimation on the animal in the video to be analyzed, so as to obtain the pose estimation result.

[0027] The tracking result acquisition module is used to acquire preset tracking parameters and input the tracking parameters into a preset improved idTracker.ai component, so that the improved idTracker.ai component can analyze the video to be analyzed and obtain the tracking results of the animals in the video to be analyzed.

[0028] The fusion module is used to combine the posture estimation results with the tracking results to generate visualized animal behavior analysis results.

[0029] Thirdly, the present invention also provides an electronic device, comprising: a processor and a memory;

[0030] The memory stores computer programs that can be executed by the processor;

[0031] When the processor executes the computer program, it implements the steps in the multi-animal behavioral data analysis method described above.

[0032] Fourthly, the present invention also provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the steps in the multi-animal behavioral data analysis method described above.

[0033] Compared with existing technologies, the multi-animal behavioral data analysis method, device, electronic equipment, and storage medium provided by this invention perform excellently in dual-mouse behavioral video analysis. They also demonstrate high tracking and posture estimation accuracy in a large number of small laboratory animals, effectively assisting manual data collection and analysis of laboratory animal behavior. Furthermore, the invention employs a front-end and back-end separated internet application design, eliminating the need for local software installation and environment configuration for animal tracking, posture estimation, and behavioral analysis. In addition, this invention minimizes the number of hyperparameters required for user input, making it convenient for researchers in related fields lacking neural network training experience, reducing the time cost of finding the optimal parameter combination. Moreover, this invention enables one-stop multi-animal behavioral analysis, implementing complete interactive logic without requiring users to write code. Attached Figure Description

[0034] Figure 1 This is a flowchart of the multi-animal behavioral data analysis method provided in the embodiments of the present invention;

[0035] Figure 2 This is a flowchart illustrating the software implementation of the multi-animal behavioral data analysis method provided in this embodiment of the invention.

[0036] Figure 3 This is a schematic diagram of the basic structure of the relational database of this invention;

[0037] Figure 4 This is a schematic diagram illustrating the key point annotation of mouse 8kpts according to the present invention;

[0038] Figure 5 This is a schematic diagram of the training of the lightweight OpenPose model of this invention;

[0039] Figure 6 This is a cross-sectional schematic diagram of the pretreatment process of the present invention;

[0040] Figure 7a This is a schematic diagram of the interface showing the tracking and pose estimation results of two mice at 8 key points under a video background.

[0041] Figure 7b This is a schematic diagram of the interface showing the tracking and pose estimation results of two mice at 8 key points under the background of no video.

[0042] Figure 8 This is a schematic diagram of the operation of the scale tool of the present invention;

[0043] Figure 9a This is a heatmap of animal activity in the 8 key behavioral analysis of mice in this invention;

[0044] Figure 9bThis is the animal state transition diagram in the 8 key behavioral analysis of mice in this invention;

[0045] Figure 9c This is a pie chart showing the proportion of animal behavioral states in the 8 key behavioral analysis of mice in this invention.

[0046] Figure 10a This is a schematic diagram of the interface showing the tracking and attitude estimation results of 14 ants and 7 key points under a video background, as presented in this invention.

[0047] Figure 10b This is a schematic diagram of the interface showing the tracking and attitude estimation results of 14 ants and 7 key points in the present invention without video enabled.

[0048] Figure 11 This is a schematic diagram of the functional modules of the multi-animal behavioral data analysis device provided in an embodiment of the present invention;

[0049] Figure 12 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0051] Please see Figure 1 and Figure 2 The multi-animal behavioral data analysis method provided by this invention includes the following steps:

[0052] S100: Obtain the video to be analyzed and preprocess the video to be analyzed;

[0053] S200. Input the preprocessed video to be analyzed into the pre-trained pose estimation model to estimate the pose of the animal in the video to be analyzed, so as to obtain the pose estimation result.

[0054] S300. Obtain preset tracking parameters and input the tracking parameters into a preset improved idTracker.ai component so that the improved idTracker.ai component can analyze the video to be analyzed and obtain the tracking results of the animals in the video to be analyzed.

[0055] S400. Combine the posture estimation results with the tracking results to generate visualized animal behavior analysis results.

[0056] In this embodiment, the system implementing this multi-animal behavioral data analysis method adopts a front-end / back-end separation design and a microservice framework. It has multiple sub-modules and can be used to provide services via the internet, run locally, or have selected modules run independently. The system software front-end primarily uses the React.js framework and Ant Design component library, with JavaScript for data rendering and sending requests to the back-end, HTML5 for page display, and CSS for layout. Data visualization uses D3.js and Echarts. The back-end uses the Flask microservice framework, PyTorch for deep learning, the lightweight SQLite database, Flask-SQLAlchemy as the ORM framework, and OpenCV for video processing. The entire software is packaged using Docker.

[0057] The embodiments of this invention demonstrate excellent performance in dual-mouse behavioral video analysis and high tracking and posture estimation accuracy in a large number of small laboratory animals. It effectively assists in the manual collection and analysis of laboratory animal behavioral data. Furthermore, its front-end and back-end separated internet application design eliminates the need for local software installation and environment configuration for animal tracking, posture estimation, and behavioral analysis. In addition, this invention minimizes the number of hyperparameters required for user input, making it convenient for researchers in related fields lacking neural network training experience. It also reduces the time cost of finding the optimal parameter combination. Moreover, this invention enables one-stop multi-animal behavioral analysis with complete interactive logic, eliminating the need for users to write code.

[0058] In some embodiments, the preprocessing process in step S100 includes at least: image scaling, image rotation, image flipping, and image cropping. Optionally, the video preprocessing function is implemented based on OpenCV.js, using network assembly language technology to provide high-performance real-time video preprocessing functions. The specific implementation method is existing technology and will not be described in detail here.

[0059] Preferably, the video to be analyzed will undergo automatic video encoding format detection during the upload process. If it cannot be played by mainstream browsers, it will be automatically re-encoded into H.264 format to ensure that the video can be played by mainstream browsers.

[0060] In some embodiments, the training method for the pose estimation model in step S200 is as follows:

[0061] After obtaining training samples and annotating the training samples with key points, the training samples are divided into training set and test set.

[0062] A neural network model is established, and the parameters of the neural network model are set according to the input size of each sample in the training set and the size of the labeled animal.

[0063] The training set is input into the neural network model after the parameters have been set to train the neural network model and obtain the pose estimation model.

[0064] In this embodiment, images are first searched from the database as training samples. If no images have been added for the currently selected animal, a certain number are randomly selected from the uploaded videos as training samples. Then, keypoint annotations are performed on the training samples, which serve as the training and testing sets for the model. Keypoint annotation can be implemented using an SVG-based keypoint annotation tool, retaining only images and annotations based on the Microsoft COCO data annotation format, thus reducing data redundancy without compromising pose estimation performance. Specifically, the necessary data required for this embodiment includes videos and related metadata, animal information, images and related metadata, annotation data, datasets and related metadata, experimental parameters, and model parameters. The attributes and interdependencies of each data component are as follows: Figure 3 As shown, a relational database app.sqlite can be constructed based on this, and the data can be persistently stored in the lightweight database app.sqlite and mapped through flask-sqlalchemy. Therefore, when constructing different animal skeleton datasets, existing images can be reused for annotation to save space.

[0065] Furthermore, after obtaining the training and test sets, the key parameter values ​​in the training set are dynamically set according to the video input size and the size of the animals labeled in the video. The output size of the network is adjusted by dynamically changing the intermediate layers of the neural network to dilated convolutional layers or deconvolutional layers. The specific implementation formula is as follows:

[0066]

[0067]

[0068] Where stride is the downsampling rate for the LightweightOpenPose output keypoint heatmap and PAF map, and bbox is... height The height represents the bounding box height in the training samples, and the height represents the image height input to the neural network for pose estimation. height This is the original height of the input video.

[0069] Furthermore, once the parameters are set, the neural network model can be trained using the training set to obtain a fully trained pose estimation model. In addition, to verify the accuracy of the model, a test set can be used to validate the pose estimation model, and the Adam optimizer can be used to optimize the model to further improve the accuracy of the model calculation.

[0070] Optionally, the neural network model is a lightweight OpenPose model. The LightweightOpenPose algorithm is used to estimate the poses of multiple animals in the video to be analyzed. LightweightOpenPose is a lightweight version of OpenPose. Compared with the second-order OpenPose, it has only 15% of the parameters, but the performance is almost the same (the accuracy is reduced by 1%). Most importantly, its model can achieve 26fps on a CPU.

[0071] In step S200, the test8kpts model generated by the dynamically adjusted lightweightOpenPose can be used to perform pose analysis on the region of interest and time interval of the video to be analyzed. After the pose analysis is completed, the pose analysis results are saved to the database and returned.

[0072] It should be noted that, in this embodiment of the invention, videos, animals, images, annotations, experiments, datasets, models, and related information are all stored and retrieved using the lightweight database SQLite.

[0073] In some embodiments, in step S300, a real-time parameter preview is provided in the browser through the OpenCV.js module based on network assembly language to assist the user in setting multi-animal tracking parameters. Specifically, the preset tracking parameters include at least the maximum number of animals to track, the range of animal region size, the range of animal pixel intensity, the video region of interest, the downsampling rate, and the time interval of the video to be analyzed. After the parameters are set, these parameters are input into the improved idTracker.ai component for tracking two mice. After the operation is completed, the tracking results are returned and the estimated tracking accuracy is saved.

[0074] In some embodiments, step S400 specifically includes:

[0075] Calculate the center point of the attitude estimation result and the center point of the tracking result;

[0076] Calculate the Euclidean distance between the center point of the attitude estimation result and the center point of the tracking result, and construct a cost matrix based on the Euclidean distance;

[0077] Based on the cost matrix, the Kuhn-Munkres algorithm is used to optimize the matching results. Based on the optimization results, the posture estimation results and the tracking results are fused to generate visualized animal behavior analysis results.

[0078] In this embodiment, the cost matrix is ​​constructed using the Euclidean distance between the center point of the attitude skeleton and the center point of the tracking, and the Kuhn-Munkres algorithm is used for matching. When the final matching result does not exceed the number of tracking targets, the result is returned to the front end.

[0079] In some embodiments, the animal behavior analysis results include at least an animal activity heatmap, an animal behavior state transition map, and an animal behavior state proportion pie chart.

[0080] In this embodiment, when displaying the results, it is done by [using the video]. <svg>The layer draws a scale bar, mapping SVG coordinates to video pixels, then to physically meaningful length units, and uses a method placed on... <video>Above the components <svg>The component draws the skeleton and uses the requestVideoFrameCallback call to obtain the exact frame for video playback, which is used for synchronization between playback and drawing.

[0081] To better understand the present invention, the technical solution of the present invention will be described in detail below through two specific embodiments:

[0082] In the first specific embodiment, the method provided by the present invention is used for dual mouse 8-key point tracking and pose estimation, specifically:

[0083] Step 1: After installing the necessary components directly or using Docker, expose the corresponding ports on your local machine, or on a local area network (LAN) or wide area network (WAN), and open the corresponding website through the Chrome browser.

[0084] Step 2: Given the experimental animal name "mouse" and after uploading the video to the software front end, the front end sends a request to the back end. The back end checks whether the video encoding format is a video playback format supported by modern browsers. If not, it re-encodes the video using H.264, sets the crf value to 23, and stores the video-related metadata in the app.sqlite database.

[0085] Step 3: In the data annotation interface, select or create a new dataset, mouse8kpts. The front-end requests animal images and the image annotations stored in the dataset from the back-end and returns them. The front-end annotation component displays these annotations. Users can freely crop images from the current animal images and perform necessary key point annotations (such as...). Figure 4 As shown in the figure, in this embodiment, 50 uploaded video images and 200 images extracted from other videos are selected. After the annotation is completed, clicking the save button will cause the software front-end to asynchronously send a request to the back-end, and the back-end will perform update, insertion, and deletion operations on the annotated data. Users can select a dataset in the trainer component, name the model test8kpts, input the keypoint connection method as [nose-left_ear,nose-right_ear,nose-tail_base,tail_base-left_leg,tail_base-right_leg,tail_base-mid_tail,mid_tail-tail_tip] and the mirror keypoint left side as [left_ear,left_leg] and the right side as [right_ear,right_leg], and send a training request through the front-end (e.g., Figure 5 As shown, after receiving the user's training request, the software backend randomly splits the dataset into a 70% training set and a 30% test set.

[0086] Step 4: After receiving the user's training request, the backend dynamically calculates the necessary parameters required for the neural network structure and uses these parameters for training the neural network.

[0087] Step 5: Iteratively train the lightweightOpenPose network using the Adam optimizer for 150 iterations. The model with the highest OKS value on the test set is saved as the optimal model for subsequent pose estimation.

[0088] Step 6: Using the OpenCV.js module, a network assembly language, real-time parameter previews are provided in the browser to assist users in setting multi-animal tracking parameters, including the maximum number of animals tracked, animal region size range, animal pixel intensity range, video region of interest (ROI), downsampling rate, and the time interval of the video to be analyzed. Simultaneously, the pre-trained test8kpts model (e.g., ...) is selected for pose estimation. Figure 6 (As shown).

[0089] Step 7: After the user submits the analysis request, the software backend obtains and checks the validity of these necessary parameters, then inputs them into the improved idTracker.ai component for tracking the two mice. After the operation is complete, the tracking results will be returned and the estimated tracking accuracy will be saved.

[0090] Step 8: Use the test8kpts model generated by the dynamically adjusted lightweightOpenPose to estimate the pose of the video region of interest and the time interval of the video to be analyzed, save the output results into the database and return.

[0091] Step 9: Read the tracking results and pose estimation results, calculate the center point of the pose estimation result frame by frame, construct the cost matrix between the center point of the tracking result and the center point of the tracking result using Euclidean distance, use the Kuhn-Munkres algorithm to find the optimal matching result as the final algorithm output result, and finally store the video name, interval, key point connection method, key point name, and frame-by-frame animal key points and ID tags in the database.

[0092] Step 10: The user queries and selects completed tasks through the front-end SummaryPage component. The software back-end returns relevant playable video resource links and experimental results. The software front-end uses an SVG layer to overlay the results onto the video layer and uses D3.js to render the skeleton frame by frame using the browser's requestVideoFrameCallback. This embodiment provides two preview modes: displaying the video background and not displaying the video background (e.g., ...). Figure 7a and Figure 7b (As shown). This embodiment provides a heatmap of experimental animal activity, and the experimental data of the first mouse in the two-mouse 8-key-point example are shown as follows. Figure 9a As shown. To further utilize the behavioral analysis system provided by this invention, the user uses the scale tool provided in the software front end to obtain the mapping relationship between pixels and actual distances (e.g., Figure 8 As shown), and based on this, combined with preset behavioral discrimination criteria, the mouse state is calculated through the front end, and a mouse state transition diagram and state distribution pie chart are drawn (e.g. Figure 9b and Figure 9c (As shown).

[0093] The final results show that the tracking estimation accuracy of this invention is 99.9%, and the average precision (mAP) value (IoU = 0.5) for attitude estimation calculated using the pycocoapi module is 0.812, with the variance of all keypoints set to the minimum value of 0.025 for human attitude estimation. The results indicate that the tracking part achieves high tracking accuracy, and the attitude estimation part also achieves a good level.

[0094] In the second specific embodiment, to verify the processing capability under different animal species, numbers, sizes, and photographic conditions, and to demonstrate its applicability and generalization ability to different experimental animals and photographic equipment, the method provided in this invention was used for tracking 14 ants at 7 key points and estimating their pose (e.g., Figure 10a and Figure 10b As shown in the figure, its specific execution process is similar to that of the first specific embodiment, and will not be described again here.

[0095] The final results show that the tracking accuracy of the present invention is 98.6%, and the average accuracy (mAP) value (IoU = 0.5) calculated using the pycocoapi module is 0.935, both of which are at a good level, indicating that it can be used for a wide range of experimental animal species. The estimation of 14 experimental animals and animals smaller than mice demonstrates the wide applicability of the present invention.

[0096] The technical solution provided by this invention performs excellently in dual-mouse behavioral video analysis and also demonstrates high tracking and posture estimation accuracy in a large number of small experimental animals. It can effectively assist in the manual collection and analysis of experimental animal behavioral data. Moreover, it adopts a front-end and back-end separated Internet application design, so users do not need to install local software and configure the environment to track animals, estimate posture, and perform behavioral analysis. In addition, this invention minimizes the number of hyperparameters that users need to input, making it convenient for researchers in related fields who lack experience in neural network training to use this invention, reducing the time cost required to find the optimal parameter combination. Furthermore, this invention can perform one-stop multi-animal behavioral analysis, realize complete interactive logic, and eliminate the need for users to write code.

[0097] Another embodiment of the present invention provides a multi-animal behavioral data analysis device; please refer to [link to relevant documentation]. Figure 11 The multi-animal behavioral data analysis device includes a preprocessing module 11, a posture estimation result acquisition module 12, a tracking result acquisition module 13, and a fusion module 14.

[0098] The preprocessing module 11 is used to acquire the video to be analyzed and to preprocess the video to be analyzed.

[0099] The pose estimation result acquisition module 12 is used to input the preprocessed video to be analyzed into the pre-trained pose estimation model to perform pose estimation on the animal in the video to be analyzed, so as to obtain the pose estimation result.

[0100] The tracking result acquisition module 13 is used to acquire preset tracking parameters and input the tracking parameters into a preset improved idTracker.ai component, so that the improved idTracker.ai component can analyze the video to be analyzed and obtain the tracking results of the animals in the video to be analyzed.

[0101] The fusion module 14 is used to combine the posture estimation results with the tracking results to generate visualized animal behavior analysis results.

[0102] In this embodiment, it performs excellently in dual-mouse behavioral video analysis and also demonstrates high tracking and posture estimation accuracy in a large number of small experimental animals. It can effectively assist in the manual collection and analysis of experimental animal behavioral data. Moreover, it adopts a front-end and back-end separated Internet application design, so users do not need to install local software and configure the environment to perform animal tracking and posture estimation and behavioral analysis. In addition, the present invention minimizes the number of hyperparameters that users need to input, making it convenient for users in related fields who lack experience in neural network training to use the present invention, reducing the time cost required to find the best parameter combination. Furthermore, the present invention can perform one-stop multi-animal behavioral analysis, realize complete interactive logic, and does not require users to write code.

[0103] It should be noted that the module referred to in this invention refers to a series of computer program instruction segments that can perform specific functions. It is more suitable than a program for describing the execution process of multi-animal behavioral data analysis. For the specific implementation of each module, please refer to the corresponding method embodiments above, which will not be repeated here.

[0104] In some embodiments, the preprocessing process includes at least: image scaling, image rotation, image flipping, and image cropping.

[0105] In some embodiments, the training method of the pose estimation model is as follows:

[0106] After obtaining training samples and annotating the training samples with key points, the training samples are divided into training set and test set.

[0107] A neural network model is established, and the parameters of the neural network model are set according to the input size of each sample in the training set and the size of the labeled animal.

[0108] The training set is input into the neural network model after the parameters have been set to train the neural network model and obtain the pose estimation model.

[0109] In some embodiments, the neural network model is a lightweight OpenPose model.

[0110] In some embodiments, the preset tracking parameters include at least the maximum number of tracks, the range of animal region size, the range of animal pixel intensity, the video region of interest, the downsampling rate, and the time interval of the video to be analyzed.

[0111] In some embodiments, the fusion module 14 is specifically used for:

[0112] Calculate the center point of the attitude estimation result and the center point of the tracking result;

[0113] Calculate the Euclidean distance between the center point of the attitude estimation result and the center point of the tracking result, and construct a cost matrix based on the Euclidean distance;

[0114] Based on the cost matrix, the Kuhn-Munkres algorithm is used to optimize the matching results. Based on the optimization results, the posture estimation results and the tracking results are fused to generate visualized animal behavior analysis results.

[0115] In some embodiments, the animal behavior analysis results include at least an animal activity heatmap, an animal behavior state transition map, and an animal behavior state proportion pie chart.

[0116] Another embodiment of the present invention provides an electronic device, such as... Figure 12 As shown, the electronic device 10 includes:

[0117] One or more processors 110 and memory 120, Figure 12 The following description uses a processor 110 as an example. The processor 110 and the memory 120 can be connected via a bus or other means. Figure 12 Taking the example of a connection between China and Israel via a bus.

[0118] Processor 110 is used to perform various control logics of electronic device 10. It can be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), microcontroller, ARM (Acorn RISC Machine), graphics processing unit (GPU) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination of these components. Furthermore, processor 110 can also be any conventional processor, microprocessor, or state machine. Processor 110 can also be implemented as a combination of computing devices, such as a combination of DSP and microprocessor, multiple microprocessors, one or more microprocessors combined with DSP and / or any other such configuration.

[0119] The memory 120, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions corresponding to the multi-animal behavioral data analysis method in the embodiments of the present invention. The processor 110 executes various functional applications and data processing of the electronic device 10 by running the non-volatile software programs, instructions, and units stored in the memory 120, thereby realizing the multi-animal behavioral data analysis method in the above-described method embodiments.

[0120] The memory 120 may include a program storage area and a data storage area. The program storage area may store applications required for the operating platform and at least one function; the data storage area may store data created based on the use of the electronic device 10. Furthermore, the memory 120 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 120 may optionally include memory remotely located relative to the processor 110, and these remote memories may be connected to the electronic device 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0121] One or more units are stored in memory 120, and when executed by one or more processors 110, they perform the multi-animal behavioral data analysis method in any of the above method embodiments, for example, the method described above. Figure 1 The method steps S100 to S400.

[0122] Another embodiment of the present invention provides a computer-readable storage medium storing computer-executable instructions that are executed by one or more processors, for example, to perform the above-described instructions. Figure 1 The method steps S100 to S400.

[0123] As an example, computer-readable storage media can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) as external cache memory. By way of illustration and not limitation, RAM can be obtained in many forms such as synchronous RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct Rambus RAM (DRRAM). The memory components or memories disclosed in the operating environment described herein are intended to include one or more of these and / or any other suitable types of memory.

[0124] In summary, the multi-animal behavioral data analysis method, device, electronic equipment, and storage medium provided by this invention demonstrate excellent performance in dual-mouse behavioral video analysis and high tracking and posture estimation accuracy in a large number of small experimental animals. They effectively assist in manual data collection and analysis of experimental animal behavior. Furthermore, the front-end and back-end separated internet application design eliminates the need for local software installation and environment configuration for animal tracking, posture estimation, and behavioral analysis. In addition, this invention minimizes the number of hyperparameters required for user input, making it convenient for researchers in related fields lacking neural network training experience and reducing the time cost of finding the optimal parameter combination. Moreover, this invention enables one-stop multi-animal behavioral analysis, implementing complete interactive logic without requiring users to write code.

[0125] The specific embodiments of the present invention described above do not constitute a limitation on the scope of protection of the present invention. Any other corresponding changes and modifications made in accordance with the technical concept of the present invention should be included within the scope of protection of the claims of the present invention.< / svg> < / video> < / svg>

Claims

1. A method for analyzing multi-animal behavioral data, characterized in that, Includes the following steps: Acquire the video to be analyzed and preprocess the video to be analyzed; The preprocessed video to be analyzed is input into a pre-trained pose estimation model to estimate the pose of the animal in the video to obtain the pose estimation result. Obtain preset tracking parameters and input the tracking parameters into a preset improved idTracker.ai component so that the improved idTracker.ai component can analyze the video to be analyzed and obtain the tracking results of the animals in the video to be analyzed. The posture estimation results are combined with the tracking results to generate visualized animal behavior analysis results; The training method for the pose estimation model is as follows: After obtaining training samples and annotating the training samples with key points, the training samples are divided into training set and test set. A neural network model is established, and the parameters of the neural network model are set according to the input size of each sample in the training set and the size of the labeled animal. The training set is input into the neural network model after the parameters are set to train the neural network model and obtain the pose estimation model. After obtaining the training and testing sets, the key parameter values ​​in the training set are dynamically set according to the video input size and the size of the animals labeled in the video. The output size of the network is adjusted by dynamically changing the intermediate layers of the neural network to dilated convolutional layers or deconvolutional layers. The specific implementation formula is as follows: in, Downsampling of LightweightOpenPose outputs keypoint heatmaps and PAF maps. The height of the bounding box in the training samples. The image height is used as input to the neural network for final pose estimation. This is the original height of the input video.

2. The method for analyzing multi-animal behavioral data according to claim 1, characterized in that, The preprocessing process includes at least: image scaling, image rotation, image flipping, and image cropping.

3. The method for analyzing multi-animal behavioral data according to claim 1, characterized in that, The neural network model is a lightweight OpenPose model.

4. The multi-animal behavioral data analysis method according to claim 1, characterized in that, The preset tracking parameters include at least the maximum number of tracks, the range of animal region size, the range of animal pixel intensity, the video region of interest, the downsampling rate, and the time interval of the video to be analyzed.

5. The method for analyzing multi-animal behavioral data according to claim 1, characterized in that, The step of combining the posture estimation results with the tracking results to generate visualized animal behavior analysis results includes: Calculate the center point of the attitude estimation result and the center point of the tracking result; Calculate the Euclidean distance between the center point of the attitude estimation result and the center point of the tracking result, and construct a cost matrix based on the Euclidean distance; Based on the cost matrix, the Kuhn-Munkres algorithm is used to optimize the matching results. Based on the optimization results, the posture estimation results and the tracking results are fused to generate visualized animal behavior analysis results.

6. The method for analyzing multi-animal behavioral data according to claim 1, characterized in that, The animal behavior analysis results include at least an animal activity heatmap, an animal behavior state transition map, and an animal behavior state proportion pie chart.

7. A multi-animal behavioral data analysis device, characterized in that, include: The preprocessing module is used to acquire the video to be analyzed and to preprocess the video to be analyzed. The pose estimation result acquisition module is used to input the preprocessed video to be analyzed into a pre-trained pose estimation model to perform pose estimation on the animal in the video to be analyzed, so as to obtain the pose estimation result. The tracking result acquisition module is used to acquire preset tracking parameters and input the tracking parameters into a preset improved idTracker.ai component, so that the improved idTracker.ai component can analyze the video to be analyzed and obtain the tracking results of the animals in the video to be analyzed. The fusion module is used to combine the posture estimation results with the tracking results to generate visualized animal behavior analysis results; The training method for the pose estimation model is as follows: After obtaining training samples and annotating the training samples with key points, the training samples are divided into training set and test set. A neural network model is established, and the parameters of the neural network model are set according to the input size of each sample in the training set and the size of the labeled animal. The training set is input into the neural network model after the parameters are set to train the neural network model and obtain the pose estimation model. After obtaining the training and testing sets, the key parameter values ​​in the training set are dynamically set according to the video input size and the size of the animals labeled in the video. The output size of the network is adjusted by dynamically changing the intermediate layers of the neural network to dilated convolutional layers or deconvolutional layers. The specific implementation formula is as follows: in, Downsampling of LightweightOpenPose outputs keypoint heatmaps and PAF maps. The height of the bounding box in the training samples. The image height is used as input to the neural network for final pose estimation. This is the original height of the input video.

8. An electronic device, characterized in that, include: Processor and memory; The memory stores computer programs that can be executed by the processor; When the processor executes the computer program, it implements the steps in the multi-animal behavioral data analysis method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, include: Processor and memory; The memory stores computer programs that can be executed by the processor; When the processor executes the computer program, it implements the steps in the multi-animal behavioral data analysis method as described in any one of claims 1-6.

Images