A method for automatic scoring of primate actions based on behavioral interaction networks

Abstract

A method for automatically scoring primate actions based on a behavioral interaction network includes the following steps: Step 1, acquiring primate video images, scoring and labeling them; the primate video images include normal behavior, post-medication behavior, and sham surgery behavior; Step 2, extracting monkey features from the primate video images using a neural network; training the neural network three times using the monkey features: the first training input is monkey features for normal behavior, the second training input is monkey features for sham surgery behavior, and the third training input is monkey features for post-medication behavior; finally, outputting the primate action score through a GAP layer and a fully connected layer.

Description

Technical Field

[0001] This invention relates to the field of intelligent medical technology, and in particular to a method for automatic scoring of primate actions based on behavioral interaction networks. Background Technology

[0002] In fields such as drug evaluation, primates (such as cynomolgus monkeys) serve as an important assessment model, and behavioral observation of cynomolgus monkeys after drug intervention is a crucial indicator. However, previous behavioral assessments often relied on specialized assessment devices, pre-defined assessment rules, and human scoring, resulting in discrete and difficult-to-quantify scores. Using deep learning-based methods to score primate behavior can provide continuous, more accurate, and objective ratings, while reducing reliance on human and material resources, thus contributing to a more reliable basis for drug evaluation.

[0003] In the field of video understanding, well-trained networks can interpret information in videos and apply it to various downstream tasks, such as evaluating the behavior of subjects in videos. With the development of deep learning and the availability of many large-scale datasets, more and more methods are being used for automatic action scoring of subjects in video data. Among these, 3D convolutional neural networks have achieved excellent performance in extracting spatiotemporal information. However, simple convolutional neural networks have weak interpretability, making it difficult to specifically improve the network, and they rely on large and rich datasets. Currently, there are no publicly available large-scale datasets for primate action scoring, and developing such a dataset is extremely costly. Furthermore, because non-human primates do not cooperate like humans, the quality of the data is often problematic. Summary of the Invention

[0004] This invention proposes a method for automatic scoring of primate actions based on behavioral interaction networks, which is applicable to small-scale primate datasets and has stronger interpretability.

[0005] This invention proposes a method for automatic scoring of primate actions based on behavioral interaction networks. The method includes the following steps:

[0006] Step 1: Collect video images of primates, and score and label them;

[0007] Step 2: Train the neural network using labeled video images;

[0008] In step 1, the primate video images include normal behavior, behavior after medication, and behavior during sham surgery.

[0009] In step 2, the neural network extracts the spatiotemporal features of primate video images. Where T, C, H, and W represent the dimensions in four dimensions: time, number of channels, height of the feature map, and width of the feature map, respectively; global average pooling is performed on the dimension corresponding to T, and the output tensors of the two are added together to obtain the monkey feature M∈R. CxHxW ;

[0010] The neural network was trained three times using monkey features. The first training input was monkey features of normal behavior. The second training input was monkey features of sham surgery behavior. The third training input was monkey features of behavior after medication.

[0011] Primate action scores are output through GAP layers and fully connected layers.

[0012] Furthermore, step 1 also includes the following steps:

[0013] Step 11: Capture video images of the normal behavior of the group of primates. Divide the group of primates into two groups and administer medication and sham surgery to the other group respectively. At the same time, acquire video images of their behavior after medication and during sham surgery.

[0014] Step 12: Filter the video data and obtain video segments involving the same action;

[0015] Step 13: The primate’s movements are scored manually and the scores are labeled on the video segments as training annotation data.

[0016] Furthermore, in step 2, the video segment length is fixed at 2 seconds and input into the neural network.

[0017] Furthermore, in step 2, the monkey features of normal behavior and the monkey features of sham surgery behavior are fed into three different 2D convolutional layers during training, thereby obtaining three tensors: query Q, key K, and value V; the monkey features of post-drug behavior are only input into the convolutional layer that generates the query Q and value V tensors, and the key K tensor is initialized by the weights learned in the previous two training sessions and remains unchanged.

[0018] Furthermore, in step 2, the query Q, key K, and value V interact with the behavioral characteristics of the post-drug intervention behavior and the behavioral characteristics of the two control groups through a multiplication layer. This layer can be represented by the following formula:

[0019]

[0020] Z = sum(Att·V)

[0021] Where Att represents attention weight, Q represents query, K represents key, V represents value, d represents the number of feature channels, and Z represents the behavioral interaction feature matrix used for final scoring;

[0022] The Multiplication layer uses matrix dot product to enable the interaction of features from different monkeys, rather than the interaction of local features from the same monkey.

[0023] Furthermore, in step 12, the number of video segments recording different manifestations of the same action in primates is balanced.

[0024] The beneficial effects of this application are:

[0025] Compared to traditional methods that rely on specially designed cages, this invention offers a contactless, automated scoring method that is simpler, lower in cost, and easier to scale up. Furthermore, traditional methods require manual scoring of monkeys, which is susceptible to subjective bias. This invention, however, uses a deep learning model to score actions, providing more objective and impartial results.

[0026] This invention introduces relational reasoning to improve the accuracy of action recognition in the model. Judging the behavior of a target often requires considering its environment; for example, in pairs figure skating, scoring is based on the interaction between the two skaters and their surroundings. Similarly, in scoring macaque actions, the meaning conveyed by the same action performed in different positions differs, thus necessitating reasoning about the relationship between the target and its surrounding environment, including other objects. In this approach, the Multiplication layer infers the relationship between the macaque and its environment during the initial model training, which contributes to improved model accuracy.

[0027] This invention proposes a multi-training method to improve the accuracy of the model in recognizing monkeys after drug administration. In the third training of the model proposed in this scheme, the key passed to the Multiplication layer is generated from the weights learned by untreated monkeys in the first two training sessions. This is intended to allow the features generated by treated and untreated monkeys to interact, thereby improving the accuracy of recognizing the behavior of treated monkeys in video. Attached Figure Description

[0028] Figure 1 This is a flowchart illustrating an automatic scoring method for primate actions based on a behavioral interaction network, as proposed in an embodiment of the present invention. Detailed Implementation

[0029] To better understand the above-mentioned objectives, features, and advantages of this application, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of this application can be combined with each other.

[0030] In the following description, many specific details are set forth in order to provide a full understanding of this application. However, this application may also be implemented in other ways different from those described herein. Therefore, the scope of protection of this application is not limited to the specific embodiments disclosed below.

[0031] This invention presents a method for automatic scoring of primate actions based on behavioral interaction networks, the method comprising the following steps:

[0032] Step 1: Data collection and behavioral scoring labeling;

[0033] Step 11: The monkeys living in individual cages are filmed using a regular camera. This invention uses 10 cynomolgus monkeys. The first step involves filming the normal behavior of the 10 monkeys for one week. Then, the 10 monkeys are divided into two groups and filmed simultaneously: the first group consists of monkeys after medication, and the second group consists of monkeys undergoing sham surgery, for another week. Finally, three sets of video data containing monkey behavior are obtained. The camera used is at 1080P resolution and 30FPS; the subjects are cynomolgus monkeys.

[0034] Step 12 involves filtering the video data and cropping out video segments that involve the same action. Here, because monkeys exhibit a long-tailed distribution of their specific behaviors when performing the same action, manual filtering is necessary to balance the amount of data of various types; for example, a monkey's eating action may have multiple forms, such as sitting, standing, and eating while suspended by one hand. The filtered video segments are then cropped to a certain length as the shortest input duration to improve model accuracy; here, 2 seconds is used.

[0035] Step 13: Have a professional review the edited video clips and score the monkey's motor abilities. More intensive scoring criteria will help improve the model's behavioral scoring ability. Here, the scoring levels are 0, 0.5, 1...10 points.

[0036] Step 2, Building the Behavioral Interaction Network;

[0037] The network mainly consists of four parts: video segmentation module, spatiotemporal feature extraction module, behavior interaction module, and quantitative scoring module. For example... Figure 1 As shown:

[0038] The video segmentation module uses basic ffmpeg operations to segment the input original video of arbitrary length into video segments of fixed length of 2 seconds, which are then input into the network.

[0039] The spatiotemporal feature extraction module employs a classic 3D convolutional neural network in the video understanding field, such as SlowFast. This module can extract spatiotemporal features of monkey behavior from a 2-second input video. These features are represented by two four-dimensional tensors, denoted as... Where T, C, H, and W represent the dimensions in the four dimensions of time, number of channels, height of the feature map, and width of the feature map, respectively. Then, a global average pooling (GAP) operation is performed on the dimension corresponding to T, and the output tensors of the two operations are added together to obtain the monkey feature M∈R. CxHxW .

[0040] The behavioral interaction module changes according to the training phase. In the first and second training phases, the network input data are data from normal monkeys and monkeys that have undergone sham surgery, respectively. At this time, the monkey features M are fed into three different 2D convolutional layers, resulting in three tensor queries Q, keys K, and values ​​V. In the third training phase, the input is data from monkeys after drug intervention. In this phase, the monkey features M are used to generate queries Q and values ​​V through the corresponding 2D convolutional layers conv2D_1 and conv2D_3 from the previous two training phases. The tensor key K is initialized with the weights learned in the previous two training phases and remains unchanged.

[0041] Finally, the multiplication layer is used to interact the behavioral characteristics after drug intervention with the behavioral characteristics of the two control groups. This layer can be represented by the following formula:

[0042]

[0043] Z = sum(Att·V)

[0044] Where Att represents attention weight, Q represents query, K represents key, V represents value, d represents the number of feature channels, and Z represents the behavioral interaction feature matrix used for final scoring;

[0045] The multiplication layer uses matrix dot product instead of matrix cross product in the attention mechanism, which allows features from different monkeys to interact rather than local features from the same monkey to interact.

[0046] The quantization scoring module inputs the behavioral interaction features Z generated by the Multiplication layer into the GAP layer and the Full Connection (FC) layer, and outputs a value O between 0 and 1. pMultiplying the value by 10 gives the behavioral evaluation score given by the network; correspondingly, the predicted value input into the binary cross-entropy loss function is multiplied by 10. p .

[0047] In the third training of the model proposed in this scheme, the key passed to the Multiplication layer is generated from the weights learned by the untreated monkeys in the first two training sessions. This is intended to allow the features generated by the treated monkeys and the untreated monkeys to interact, thereby improving the accuracy of the video of the treated monkeys' behavior.

[0048] Although this application has been disclosed in detail with reference to the accompanying drawings, it should be understood that these descriptions are merely exemplary and not intended to limit the application of this application. The scope of protection of this application is defined by the appended claims and may include various variations, modifications, and equivalents of the invention without departing from the scope and spirit of this application.

Claims

1. A method for automatic scoring of primate motion based on behavioral interaction networks, characterized in that, The method for automatic scoring of primate actions based on behavioral interaction networks includes the following steps: Step 1: Collect video images of primates, and score and label them; Step 2: Train the neural network using labeled video images; In step 1, the primate video images include normal behavior, behavior after medication, and behavior during sham surgery. In step 2, the neural network extracts the spatiotemporal features of primate video images. Where T, C, H, and W represent the dimensions in four dimensions: time, number of channels, height of the feature map, and width of the feature map, respectively; global average pooling is performed on the dimension corresponding to T, and the output tensors of the two are added together to obtain the monkey features. ; The neural network was trained three times using monkey features. The first training input was monkey features of normal behavior, the second training input was monkey features of sham surgery behavior, and the third training input was monkey features of behavior after medication. The primate action score is output through the GAP layer and the fully connected layer; In step 2, the monkey features of normal behavior and the monkey features of sham surgery behavior are fed into three different 2D convolutional layers during training, resulting in three tensors: query Q, key K, and value V; the monkey features of post-drug behavior are only input into the convolutional layer that generates the query Q and value V tensors, and the key K tensor is initialized by the weights learned in the previous two training sessions and remains unchanged.

2. The method for automatic scoring of primate actions based on behavioral interaction networks according to claim 1, characterized in that, Step 1 also includes the following steps: Step 11: Capture video images of the normal behavior of the group of primates. Divide the group of primates into two groups and administer medication and sham surgery to the other group respectively. At the same time, acquire video images of their behavior after medication and during sham surgery. Step 12: Filter the video data and obtain video segments involving the same action; Step 13: The primate’s movements are scored manually and the scores are labeled on the video segments as training annotation data.

3. The method for automatic scoring of primate actions based on behavioral interaction networks according to claim 2, characterized in that, In step 2, the video segment is fixed at a length of 2 seconds and input into the neural network.

4. The method for automatic scoring of primate actions based on behavioral interaction networks according to claim 1, characterized in that, In step 2, the monkey features of normal behavior and the monkey features of sham surgery behavior are fed into three different 2D convolutional layers during training, resulting in three tensors: query Q, key K, and value V; the monkey features of post-drug behavior are only input into the convolutional layer that generates the query Q and value V tensors, and the key K tensor is initialized by the weights learned in the previous two training sessions and remains unchanged.

5. The method for automatic scoring of primate actions based on behavioral interaction networks according to claim 4, characterized in that, In step 2, the query Q, key K, and value V interact with the behavioral characteristics of the post-drug intervention and the behavioral characteristics of the two control groups through a multiplication layer. This layer can be represented by the following formula: ； ； in, Indicates attention weights, Indicates a query. Indicates key. Represents the value. The number of channels representing the feature. This represents the behavioral interaction feature matrix used for the final scoring; The Multiplication layer uses matrix dot product to enable the interaction of features from different monkeys, rather than the interaction of local features from the same monkey.

6. The method for automatic scoring of primate actions based on behavioral interaction networks according to claim 2, characterized in that, In step 12, the number of video segments recording different manifestations of the same action in primates is balanced.

Images